One-sentence Summary

NVIDIA researchers introduce OpenMathInstruct-1, a $1.8 \times 10^6$1.8×106 problem-solution math instruction tuning dataset synthesized from GSM8K and MATH via code-interpreter prompting with the permissively licensed Mixtral model, overcoming the performance gap that previously limited open-source data generation, and their OpenMath-CodeLlama-70B model achieves $84.6\%$84.6% on GSM8K and $50.7\%$50.7% on MATH, competitive with GPT-4 distilled counterparts, with all code, models, and data released commercially.

Key Contributions

• The paper introduces OpenMathInstruct-1, a math instruction tuning dataset containing 1.8M problem-solution pairs synthesized with the permissively licensed Mixtral model, which is at least four times larger than prior math reasoning datasets and reaches a training coverage of 99.9% for GSM8K and 93% for MATH.

• The work proposes a prompting novelty that uses masked reference text solutions to guide code-interpreter style solution synthesis, which substantially improves coverage and downstream fine-tuning performance compared to subject-specific or raw-reference prompts.

• Results show that the fine-tuned OpenMath-CodeLlama-70B model achieves 84.6% on GSM8K and 50.7% on MATH, competitive with the best GPT-distilled models. The dataset, code, and all fine-tuned models are publicly released under a commercially permissive license, along with 6.6M additional incorrect sampled solutions.

Introduction

The authors address the growing practice of creating task-specific small language models by distilling synthetic data from powerful closed-source models like GPT-4, particularly for mathematical reasoning. They note that current state-of-the-art open-source math models are all GPT-distilled, which introduces serious limitations: legal restrictions on how fine-tuned models can be used, higher generation costs compared to open-source alternatives, and reproducibility issues since closed-source model behavior shifts over time. Comparing GPT-4 with Mixtral 8x7B, a leading open-source model, they find that Mixtral's training set coverage (TSC) only catches up to GPT-4 on the easier GSM8K benchmark with roughly 8 times more sampled solutions, and still falls short on the harder MATH benchmark even with 12 times the samples. This coverage gap directly degrades distillation data quality and downstream model performance.

The authors first attempt brute-force solution sampling, which scales only logarithmically, and subject-specific few-shot prompting, which yields marginal gains. Their key contribution is leveraging the reference text solutions that accompany benchmarks like MATH and GSM8K, incorporating them into few-shot prompts with a slight modification to synthesize code-interpreter style solutions. This approach substantially boosts coverage and produces OpenMathInstruct-1, a 1.8M problem-solution pair dataset with 93% TSC on MATH and 99.9% on GSM8K. Fine-tuned models from this dataset, such as OpenMath-CodeLlama-70B, reach 84.6% on GSM8K and 50.7% on MATH, competitive with GPT-distilled counterparts. The authors publicly release the dataset, all fine-tuned models, code, and 6.6M incorrect sampled solutions under a commercially permissive license to support open-source efforts.

Dataset

The authors construct OpenMathInstruct-1, a mathematical reasoning dataset, by synthesizing solutions for existing problems from the GSM8K and MATH training sets using a base LLM. The goal is to generate diverse, high-quality code-interpreter style solutions that interleave natural language reasoning with Python code blocks, using special tokens to demarcate code and execution blocks.

Sources and composition

• The dataset is built solely from the training sets of GSM8K and MATH, with no new problem augmentation.
• Solutions are generated via few-shot prompting with 5 representative problems per prompt, sampled at temperature 1.0 and top_p 0.95, with a maximum of 4096 input-output tokens, 512 new tokens per code block, at most 3 code blocks, and halting on any execution error.
• Only problem-solution pairs that yield the correct final answer are kept in the fine-tuning set.

Per-subset details

• GSM8K: 128 solutions sampled per problem, reaching 99.1% training coverage with the default prompt. Using masked text solutions raises this to 99.9%. Final subset contains 7,469 problems (out of 7,473) with 1.04M unique solutions.
• MATH: 224 solutions sampled per problem with the default prompt, reaching only 80.1% coverage. Subject-specific prompting, using one example per difficulty level for each of the seven subjects, adds 14K correct solutions but barely improves coverage. Masked text prompting lifts coverage to 85.9% (default) and 87.5% (subject-specific). Combining default and subject-specific prompts with masked text reaches 85.1% coverage. Final subset contains 6,978 problems (out of 7,500) with 787K unique solutions.

Prompting strategies

• Default: a single 5-shot prompt per dataset, mixing vanilla Python and sympy-heavy examples for GSM8K and code, text, and mixed reasoning examples for MATH.
• Subject-specific (Subj): seven separate 5-shot prompts, one per MATH subject (algebra, geometry, intermediate algebra, number theory, prealgebra, precalculus, probability), each sampling 32 solutions per problem, for a total of 224 per problem.
• Masked text (Mask-Text): reference text solutions are used in the prompt but with all numbers in intermediate computations replaced by symbols, preventing shortcut solutions that copy answers directly from the text.

Post-processing

• Solutions with multiple \boxed{} blocks are removed.
• Solutions containing the opening code tag without the closing tag are removed.
• Text beyond the line containing the answer is trimmed.

Data selection

• Fair downsampling iterates over problems round-robin, sampling solutions without replacement, to balance representation across problems, unlike naive sampling which perpetuates the imbalance where easier problems have many more solutions.
• Code-preferential selection removes text-based solutions when code-based ones exist: Majority-Code keeps code if code outweighs text, and Any-Code keeps code if any code solution exists. This is motivated by the fact that 35.1% of MATH solutions are text-only, while GSM8K has only 2%.

Final dataset

• OpenMathInstruct-1 merges and deduplicates all generated problem-solution pairs, yielding 787K unique solutions for MATH and 1.04M for GSM8K. The authors use this data for supervised fine-tuning of a base LLM, selecting subsets via fair downsampling and code-preferential strategies to reduce training time and improve performance.

Method

The authors propose a comprehensive pipeline for synthesizing high-quality, diverse training data for mathematical reasoning tasks. The process begins with a foundation LLM, such as Mixtral-base, tasked with generating solutions for a dataset $\mathcal{X} = \{ (q_i, a_i) \}$X={(qi​,ai​)}. To ensure precision and expressiveness, the synthesized solutions adopt a code-interpreter format. This format interweaves natural language reasoning with Python code blocks, demarcated by specific tokens (<llm-code> and <llm-code-output>). During inference, the model generates the code, invokes a Python interpreter to execute it, appends the output, and continues autoregressive generation.

The core of the synthesis pipeline relies on few-shot prompting. The prompt structure includes a text-based instruction $\mathcal{I}$I, a set of $K$K representative problem-solution pairs $\{ (q_1, c_1), \dots, (q_K, c_K) \}${(q1​,c1​),…,(qK​,cK​)}, and the target question $q'$q′. The base LLM generates a candidate solution $c'$c′, which is added to the fine-tuning set if it yields the correct answer. The authors explore several prompting strategies to maximize coverage and diversity. Initially, a default 5-shot prompt is used. For more complex benchmarks like MATH, subject-specific prompts are constructed by selecting examples from different difficulty levels and subjects.

A significant challenge in using reference text solutions is the tendency of the model to generate trivial shortcuts, such as directly copying the answer from the prompt. To mitigate this, the authors introduce Masked Text Solution Prompting. In this approach, the reference text solutions provided in the prompt have all numbers in intermediate computations replaced with symbols, forcing the model to perform the reasoning steps rather than copying results.

As shown in the figure below:

Following generation, the pipeline applies post-processing to remove syntactically noisy solutions, such as those with multiple answer blocks or unclosed code tags. The resulting corpus, OpenMathInstruct-1, contains hundreds of solutions per problem, leading to a distribution imbalance where easier problems have significantly more solutions than harder ones.

To address this imbalance, the authors analyze the distribution of solution counts.

As shown in the figure below:

They propose a fair downsampling strategy that iterates over problems round-robin to sample solutions without replacement, ensuring a balanced representation compared to naive instance-level sampling. Finally, the pipeline implements code-preferred data selection strategies. For problems with both text-based and code-based solutions, the system prioritizes code-based solutions using two strategies: Majority-Code (removing text solutions if code solutions are the majority) and Any-Code (removing text solutions if any code solution exists). This selection process helps reduce training time and improves model performance by focusing on the precise nature of code-based reasoning.

Experiment

The experiments fine-tuned models on a math reasoning corpus, evaluating on benchmarks like GSM8K and MATH with greedy and self-consistency decoding. The OpenMath-finetuned models proved competitive with GPT-distilled counterparts, and ablations showed that fair downsampling, masked prompting, larger datasets, and code-preferential solution selection improve performance, though the latter reduces solution diversity. Analysis revealed that code-based answers outperform text-based ones and that reasoning errors dominate failure modes.

OpenMathInstruct-1 is significantly larger than existing mathematical reasoning fine-tuning datasets, being four times bigger than the largest prior dataset, and it is one of only two datasets with a permissive license, enabling unrestricted reuse. The dataset was generated using Mixtral, an open-source model, in contrast to many datasets that rely on proprietary GPT models. OpenMathInstruct-1 is four times larger than the previous largest dataset, MetaMathQA. It is the only dataset besides Lila that carries a permissive license, allowing unrestricted usage. The dataset was generated with Mixtral, an open-source model, whereas most other datasets depend on closed-source generators like GPT-4 or GPT-3.5.

Subject-specific prompts did not improve MATH coverage over the default prompt, but masked text prompting substantially increased coverage for both benchmarks. Combining all prompting strategies achieved the highest overall coverage, though it required sampling many solutions per problem. Subject-specific prompting alone left MATH coverage unchanged, even though it increased the number of unique solutions. Masked text prompting raised MATH coverage significantly and also improved GSM8K coverage to near-perfect levels. Combining all prompt types yielded the best overall coverage, but required nearly four times more samples per problem than the default setup.

The OpenMath-finetuned models are competitive with GPT-distilled counterparts, especially at the 7B scale. They outperform MetaMath and MAmmoTH, and with self-consistency decoding they surpass ToRA on both GSM8K and MATH. OpenMath-Mistral-7B is second best to WizardMath on GSM8K and trails ToRA by only a tiny margin on MATH. OpenMath models consistently outperform MetaMath and MAmmoTH across all evaluated reasoning tasks. With self-consistency decoding, OpenMath models achieve better results than ToRA on both GSM8K and MATH. OpenMath-CodeLlama shows strong performance on multiple benchmarks, including GSM-Hard, SVAMP, and MAWPS.

Fair downsampling produces a model that outperforms naive downsampling on both GSM8K and MATH validation sets. The improvement is more pronounced for MATH, which has a greater data imbalance in the corpus. Fair downsampling improves GSM8K accuracy from 74.3 to 75.3. Fair downsampling improves MATH accuracy from 35.0 to 37.0, a larger relative gain than on GSM8K.

Fine-tuning on data generated with masked prompting yields better validation performance than default prompting on both GSM8K and MATH. The improvements align with earlier observed gains in training set coverage, indicating that masked prompting produces more effective training data. Masked prompting outperforms default prompting on both GSM8K and MATH benchmarks. The performance gains from masked prompting translate to improved fine-tuning outcomes. The advantage of masked prompting is consistent across both evaluated datasets.

OpenMathInstruct-1 stands out as the largest mathematical reasoning dataset with a permissive license, generated using the open-source Mixtral model. Prompting experiments show that masked text prompting substantially boosts benchmark coverage, while fine-tuning on such data yields better model performance, and combining all prompt strategies maximizes coverage at a higher sampling cost. Models fine-tuned on OpenMathInstruct-1 rival or surpass GPT-distilled baselines, particularly with self-consistency decoding. Finally, fair downsampling of training data outperforms naive downsampling, especially for datasets with greater imbalance.