One-sentence Summary

Researchers at Microsoft Research propose Orca-Math, a $7\text{-billion-parameter}$7-billion-parameter Mistral-7B-based SLM that achieves $86.81\%$86.81% $\text{pass@1}$pass@1 on GSM8K without verifiers, code execution, or ensembling, by leveraging a multiagent-generated $200\text{K}$200K-problem dataset and iterative preference learning, surpassing larger models such as LLAMA-2-70B with substantially less data.

Key Contributions

• Orca-Math, a 7-billion-parameter model built on Mistral-7B, achieves 86.81% pass@1 on GSM8K without ensembling, verifiers, code execution, or other external tools, surpassing larger models such as LLAMA-2-70B (56.8%) and Gemini-Pro (86.5% with 32 trials).

• Orca-Math-dataset is a synthetic dataset of 200K math problems generated via a multi-agent setup (Agent-Instruct) that paraphrases existing problems while expanding both diversity and difficulty, using orders of magnitude fewer examples than prior datasets.

• The iterative learning procedure combines supervised fine-tuning with student practice and teacher feedback, training on preference pairs to improve performance from 81.50% (SFT alone) to 86.81% pass@1, demonstrating the effectiveness of model-generated positives and the robustness of KTO over DPO.

Introduction

Frontier models such as GPT-4 exhibit strong reasoning skills that are largely attributed to massive scale, and recent work asks whether small language models (SLMs) can acquire similar abilities through teacher-student training with high-quality synthetic data. Prior approaches on the GSM8K benchmark, such as Phi-GSM and OVM, often rely on ensembling up to 100 model outputs with consensus voting or separate verifiers, which incurs high computational cost (up to 200 model calls), requires very large datasets (e.g., 12M examples), or uses external tools such as code execution to avoid arithmetic errors. The authors extend the teacher-student paradigm to an iterative learning setting. They generate Orca-Math, a 200K synthetic dataset using an agent-based pipeline, Agent-Instruct, that paraphrases and expands problems in diversity and difficulty. Their procedure fine-tunes the student on demonstrations, has it practice generating multiple solutions, and uses teacher feedback for evaluation or to provide reference solutions. With supervised fine-tuning alone they reach 81.50% on GSM8K pass@1, and the iterative loop raises this to 86.81% without ensembling, verifiers, code execution, or external tools, surpassing much larger models such as LLAMA-2-70B, WizardMath-70B, and GPT-3.5 while using orders of magnitude less training data.

Dataset

Orca-Math Dataset: Composition, Construction, and Use

The authors build a synthetic dataset called Orca-Math-dataset, containing 200K grade school math word problems paired with GPT-4-Turbo solutions. It is constructed through a multi-stage agent-based pipeline called Agent-Instruct, which expands a small seed set in both diversity and difficulty.

1. Dataset composition and sources

• The seed set is built by collecting 36,217 word problems from ten existing open-source datasets: NumGLUE, AddSub, ALGES, ASDiv, DRAW, GSM8k, MATHQA, MultiArith, SingeOP, SingleEQ, and SVAMP. Problems are drawn from the train and validation splits of the Lila benchmark.
• Three agent-based expansion paths are used to grow the seed set into the final 200K examples.

2. Key details for each subset

• Ask Me Anything agent: This agent takes each seed problem, converts the question into a statement, and then generates multiple new word problems by replacing each number in turn with an unknown. It produces 120,445 new problems. All generated problems retain a similar narrative to the original seed. Solutions are generated with GPT-4-Turbo.
• Suggester and Editor agents: This team works together to create harder problems. The Suggester proposes ways to increase complexity (more variables, additional transactions, longer duration), and the Editor rewrites the problem accordingly. The process is implemented with the AutoGen framework and runs for up to two rounds per problem. Problems where the GPT-4-Turbo generated answer exceeds 1,800 characters are filtered out. This path yields 37,157 problems.
• DMath subset: The authors include 6,216 problems from DMath, specifically a filtered subset of the 7,943 DMath training examples where the GPT-4-Turbo solution matches the gold-standard answer exactly.

3. How the dataset is used in the model

• The full 200K problem-solution pairs are first used for supervised fine-tuning (SFT) of the small language model on the demonstrations. This yields 81.50% pass@1 accuracy on GSM8k without ensembling or external tools.
• The authors then run an iterative learning loop. In iteration #2, they sample four responses per problem from the SFT-tuned model (using top_p = 0.95 and temperature = 0.7). A prompt-based exact-match check against the teacher (GPT-4-Turbo) answer labels each student response as positive or negative. For the roughly 80K problems where all four student responses are correct (so no negative examples exist), they randomly sample one negative response from another problem that has negatives. This creates a preference dataset of positive and negative pairs for each question.
• In iteration #3, the process is repeated, but the four responses are generated by the KTO-trained model from iteration #2 (called M2) instead of the SFT model.
• The authors evaluate both Direct Preference Optimization (DPO) and Kahneman-Tversky Optimization (KTO) for learning from these preference pairs. With the iterative loop, pass@1 improves to 86.81% on GSM8k.

4. Processing details

• The Agent-Instruct pipeline paraphrases existing problems and introduces new complexity through the Suggester and Editor agents.
• Filtering rules include removing generated problems whose GPT-4-Turbo answer exceeds 1,800 characters, and retaining only DMath problems where the teacher solution matches the gold answer.
• No cropping strategy is mentioned; the main processing steps are answer-length filtering, solution alignment checking, and the construction of preference pairs (positive and negative) for preference-based training.

Method

The authors construct a training corpus of grade school math word problems through a multi-agent pipeline, then train an open-source 7B model with a two-stage recipe that combines supervised fine-tuning with iterative preference learning from both positive and negative signals.

Dataset Construction: Agent-Instruct

The data construction process begins with a seed set of 36,217 problems collected from existing open-source datasets, including NumGLUE, AddSub, ALGES, ASDiv, DRAW, GSM8k, MATHQA, MultiArith, SingeOP, SingleEQ, and SVAMP, assembled via the Lila benchmark from its train and validation splits. The authors then deploy specialized agents to expand and diversify this seed set.

The first agent, "Ask Me Anything," generates multiple new word problems from each seed problem. It first converts the question of a word problem into a declarative statement, then systematically replaces each number in the statement to create new questions that target different unknown quantities. This agent produces 120,445 new problems. While these generated problems maintain a similar narrative to the original seed, they substantially increase the breadth of the dataset. Solutions for all generated problems are produced by GPT4-Turbo.

To introduce genuinely challenging problems, the authors design a collaborative two-agent workflow consisting of a Suggester and an Editor. The Suggester examines a given problem and proposes strategies to increase its difficulty without writing the actual problem, such as increasing the number of entities, adding more variables, or incorporating additional transactions. The Editor then takes the original problem along with these suggestions to generate an updated, more complex version. This iterative process can run over multiple rounds, with each round further escalating the difficulty. The multi-agent workflow is implemented using the AutoGen framework, and the authors allow two rounds of iteration per problem. Problems for which the GPT4-Turbo generated answer exceeds 1,800 characters are filtered out, yielding 37,157 problems. Additionally, 6,216 problems from the DMath dataset are included, a subset whose solutions computed by GPT4-Turbo exactly match the gold-standard answers. Combined with the initial seed set, the final dataset, Orca-Math-200K, contains roughly 200,000 problems with GPT4-Turbo generated solutions.

Supervised Fine-Tuning (Iteration #1)

The base model, Mistral-7B, is fine-tuned on the Orca-Math-200K dataset. Each training instance is formatted in the instruction style:

The loss is computed only on the answer tokens. Training uses a constant learning rate of $1 \times 10^{-6}$1×10−6, a per-device batch size of 3, and runs for one epoch across eight A100 nodes, each with eight GPUs.

Iterative Learning from Positive and Negative Signals

The authors extend the training beyond supervised fine-tuning through an iterative preference learning scheme that leverages both correct and incorrect student outputs.

For iteration #2, they sample four responses per problem from the SFT-tuned model of iteration #1, using top-$p = 0.95$p=0.95 and temperature $= 0.7$=0.7. Each of the 200,000 problems now has one GPT4-Turbo generated solution and four student-generated solutions. The authors employ a GPT4-based exact-match evaluator to check whether each student answer aligns with the teacher solution. Student answers that do not match the teacher answer are labeled negative; matching answers are labeled positive. The preference dataset is then constructed per question $q_i$qi​: $q_i^+$qi+​ contains all positive solutions (by construction, the teacher solution guarantees at least one positive), and $q_i^-$qi−​ contains all negative solutions. For roughly 80,000 questions where all four responses are aligned with the teacher solution and $q_i^-$qi−​ is empty, negative examples are borrowed from four other questions $q_j$qj​ that do contain negatives, one from each. In this special case, $|q_i^+| = 4$∣qi+​∣=4. The final preference dataset is the union over all questions of the Cartesian product set $Q_i = \{(q_i, a_i^+, a_i^-) | (a_i^+, a_i^-) \in q_i^+ \times q_i^-\}$Qi​={(qi​,ai+​,ai−​)∣(ai+​,ai−​)∈qi+​×qi−​}.

For iteration #3, the same procedure is repeated, but the four responses are generated by the model trained with the preference optimization algorithm on the iteration #2 dataset rather than the SFT model from iteration #1.

To learn from the positive and negative signals, the authors evaluate two alignment algorithms: Direct Preference Optimization (DPO), a popular approach for efficiently fine-tuning language models to match preferences, and Kahneman-Tversky Optimization (KTO), which differs in that it only requires a binary yes/no judgment on the quality of each output rather than paired comparisons. This two-stage pipeline, combining supervised fine-tuning with iterative preference learning from both teacher-aligned and misaligned student solutions, forms the core training methodology for the Orca-Math model.

Experiment

The authors evaluate Mistral-7B models on GSM8k and other math word problem datasets using a GPT4-based exact match metric. They compare supervised fine-tuning, DPO, and KTO across iterative training rounds, finding KTO to perform best in the second iteration and that removing model-generated positives or synthetic negatives causes significant performance drops. Additional benchmarks beyond GSM8k are tested, and a contamination check reveals negligible overlap between training and test sets.

The table compares GSM8K accuracy across several Llama-2 based models, showing that larger models and specialized math fine-tuning both lead to higher performance. Fine-tuned models such as MetaMath and WizardMath outperform the base Llama-2 at every size, and the largest fine-tuned variants exceed 80% accuracy. Scaling model size yields consistent accuracy gains across all model families. Domain-specific fine-tuning on math problems provides large improvements over the base model at equal sizes. The highest accuracies are achieved by large fine-tuned models, surpassing the 80% threshold.

Iterative training with preference optimization methods improves GSM8k accuracy over supervised fine-tuning alone. KTO consistently outperforms DPO in the second iteration, and a final KTO round on Orca-Math data yields the highest accuracy. KTO after one SFT epoch reaches 85.06% accuracy, surpassing DPO (84.23%) and additional SFT (81.50%). A third iteration using KTO on Orca-Math data achieves the best result at 86.87%, while SFT repeated three times only reaches 80.44%. DPO after a KTO-trained model (84.91%) also beats DPO after initial SFT (84.23%), indicating iterative learning gains.

Removing model-generated positives from the preference dataset leads to a significant performance drop for both DPO and KTO training, indicating that these positives provide important supervision. The drop is consistent regardless of the training algorithm used. Excluding model-generated positives from the preference dataset reduces accuracy by about 2.27 points for both DPO and KTO. The performance decline occurs for both training methods, showing that model-generated positives are crucial for effective learning.

Removing problems where all sampled responses are positive (synthetic negative cases) hurts performance overall, but the effect varies by training algorithm. The drop is substantial for DPO, while KTO-based training remains largely unchanged, suggesting these cases are more critical for DPO. Excluding all-positive-response problems causes a large performance decline for DPO training. KTO training is minimally affected, with a slight improvement in one configuration and a small drop in another. Eliminating these questions reduces the dataset by roughly 80k questions in iteration 2 and 104k in iteration 3.

Across several single-answer word problem benchmarks, Orca-Math matches or exceeds the accuracy of its earlier SFT iteration, with equal performance on two datasets and clear improvements on the others. The gains are most pronounced on structured and multi-step problems. Orca-Math outperforms the SFT model on AddSub, SingleOp, SinglEq, and Svamp structured. Accuracy is identical for ASDiv and MultiArith between the two models. The largest relative gains occur on Svamp structured and AddSub.

Across a series of experiments, the work validates that larger model scales and domain-specific fine-tuning boost math reasoning, while iterative preference optimization further improves accuracy over standard supervised fine-tuning. KTO consistently outperforms DPO across iterations, and removing model-generated positives from the preference data hurts both methods, whereas eliminating synthetic negative cases degrades DPO more than KTO. On additional word problem benchmarks, the final Orca-Math model matches or improves on its supervised fine-tuning predecessor, especially on structured and multi-step tasks.