Synthetic code dataset improves LLMs on Python | Keryc
NVIDIA presents an approach to generate synthetic data targeted at programming concepts and validates it by showing clear gains in coding tasks. What does this mean for models that already handle a lot of text but lack rigor in specific skills like execution reasoning or algorithms? I'll walk you through it step by step.
What they did
They built a scalable pipeline to generate synthetic data oriented to concepts in programming. The main idea is not just more tokens, but data that target concrete skills. As a first use case, they produced a subset called Nemotron-Pretraining-Code-Concepts with roughly 15 million Python problems.
These problems were created from a taxonomy of programming concepts built by massive annotation of previous datasets (Nemotron-Pretraining-Code-{v1,v2}). Generation used GPT-OSS 120B and each problem was validated to be executable Python code using ast.parse.
How the concept-driven generation workflow works
The key piece is a hierarchical taxonomy that encodes thousands of concepts, from basic constructs like strings and recursion to advanced patterns in algorithms and data structures.
They extract relevant concepts (for example from HumanEval prompts).
They combine and distill those concepts to build conceptual seeds. This lets you control difficulty, diversity, and conceptual balance.
They use a generative model to produce open-ended problems that follow defined instructions and constraints.
They filter and validate using parsing and quality rules.
The original paper includes figures that summarize the process. Figure 1 shows taxonomy-guided generation and Figure 2 illustrates a seed with concepts like data-structures.sets.operation and algorithms.geometry.computational that leads to a problem about convex hull areas.
Technical results and metrics
They integrated about 10 billion tokens from the Code Concepts dataset into the last 100 billion tokens of Nemotron-Nano-v3 pretraining. The result was a 6-point improvement on the HumanEval accuracy benchmark, rising from 73 to 79.
Beyond the numeric gain, qualitative analysis showed improvements in specific concepts like graph algorithms and set operations, plus better handling of edge cases and execution-style reasoning. Figure 3 in the report compares base evaluations for the model with and without the synthetic data.
Why this matters (practical view)
Do more data always help? Not necessarily. This work shows that data designed to cover specific conceptual gaps can amplify performance on tasks where models tend to fail.
For research and product teams this means you can target pretraining or later stages with datasets that reinforce critical skills without collecting massive amounts of human-labeled code. For entrepreneurs and educators, these sets can be a source of better-balanced exercises or benchmarks.
Useful technical details if you want to replicate it
Taxonomy: You need a clear concept hierarchy. They used 91 core concepts for HumanEval, but the original taxonomy contains thousands.
Generator: A large controllable generation model (here GPT-OSS 120B) to produce prompts and solutions.
Validation: Parsing with ast.parse, quality filtering, and automated tests when possible. This reduces noise and ensures executable code.
Token balance: In the experiment, ~10B synthetic tokens within 100B total were enough to see impact. You don't have to copy that exact ratio, but plan your token budget.
Limitations and open questions
Risk of overfitting to benchmarks: improving HumanEval doesn't guarantee uniform gains across all real-world programming scenarios.
Concept coverage: even with a broad taxonomy, there will be conceptual gaps or socio-technical issues (licenses, solution biases, coding style).
Generator quality: effectiveness depends on the generator's ability to produce non-trivial problems and correct solutions.
What you can do now
If you work on LLMs or programming education, download the taxonomy and dataset, check licenses, and test how these data affect your model on specific tasks. If you're curious, think about adapting the workflow to other domains: math, legal reasoning, or science questions.
The core contribution isn't just the 15 million problem dataset, it's showing that concept-driven generation at scale can be a practical tool to steer capabilities in large models. It's not magic — it's intentional data design.
