One-sentence Summary

Researchers from NVIDIA and the University of Hong Kong introduce ToolOrchestra, a reinforcement learning method that trains small orchestrators to coordinate intelligent tools using outcome-, efficiency-, and user-preference-aware rewards, yielding an $8\text{B}$8B model that outperforms GPT-5 on HLE ($37.1\%$37.1% vs. $35.1\%$35.1%) while being $2.5\times$2.5× more efficient and using roughly $30\%$30% of the cost on $\tau ^ { 2 } .$τ2. -Bench and FRAMES, demonstrating that lightweight orchestration of diverse tools achieves a better performance-cost trade-off than existing tool-use agents.

Key Contributions

• The paper introduces ToolOrchestra, a reinforcement learning method that trains small language models to act as orchestrators coordinating diverse tools and specialized models. Trained end-to-end with outcome-, efficiency-, and user-preference-aware rewards, the resulting Orchestrator-8B achieves 37.1% on Humanity's Last Exam, outperforming GPT-5 (35.1%) while using 2.5x less compute.

• The method contributes a novel reward training design that goes beyond accuracy, jointly optimizing task correctness, cost and latency efficiency, and alignment with user tool and cost preferences. To support this training, the work also releases ToolScale, a synthetic user-agent-tool dataset built to aid the reinforcement learning process.

• Results demonstrate that the Orchestrator reaches state-of-the-art accuracy on challenging reasoning benchmarks, surpassing GPT-5 on tau^2-Bench and FRAMES by wide margins while costing roughly 30% as much. Further analysis shows robust generalization to unseen tools and tasks, indicating the learned tool-use policy transfers well beyond the training distribution.

Introduction

Large language models have shown impressive reasoning but remain limited in complex agentic tasks such as those in Humanity's Last Exam, where tool use through search engines and code interpreters can boost accuracy and reduce hallucinations. Prior research typically equips a single powerful model with such tools, which underutilizes the potential of diverse resources and ignores cost and user preferences. The authors instead propose an orchestration paradigm, where a small orchestrator model dynamically selects and sequences both deterministic tools and other language models of varying capabilities, treating them as intelligent tools. They note that naive prompting of an off-the-shelf model as orchestrator suffers from biases like overusing similar or strongest models, and that controllability along cost efficiency and user preference axes is underexplored. Their main contribution is ToolOrchestra, a reinforcement-learning method that trains an 8B parameter model end to end using rewards balancing correctness, resource efficiency, and user alignment, supported by an automatically generated dataset (ToolScale). Experiments show this orchestrator outperforms frontier models on reasoning benchmarks while using a fraction of the compute.

Dataset

The authors construct a comprehensive dataset to train and evaluate the Orchestrator, focusing on synthetic task generation, user preference modeling, and diverse tool configurations.

1. Synthetic Agentic Tool-Call Dataset (ToolScale)

• To address the scarcity of verifiable agentic tool-call data, the authors devise a two-step synthesis pipeline.
• First, they simulate user-agent-tool environments by asking an LLM to generate a database (including schema and entries) and propose frequently used tools for a given domain.
• Second, the LLM generates diverse user tasks based on the database information. Each task includes an instruction, a sequence of golden function calls, and short information that must be mentioned to solve the task.
• To increase difficulty, an additional LLM complicates the generated tasks by adding more constraints.
• The data is filtered to remove tasks if the golden function calls produce an error, if LLMs cannot solve it in pass@8, or if the task can be solved without any actions.
• A trajectory is considered successful if it meets three criteria: execution correctness (database content matches), process fidelity (predefined information is mentioned), and operation completeness (the same database entries are operated on).

2. User Preference Data

• The authors construct pairs of preference instructions and preference vectors to train the model to account for user preferences in tool selection.
• Given a set of tools and their configuration metadata (e.g., price, latency), an LLM proposes diverse pairs, which are validated by another LLM for consistency.
• The pairs are split into training and evaluation sets.
• The preference instruction is concatenated to the original task instruction, and the preference vector is used to calculate the reward during training.

3. General Tool Configuration

• To improve generalization, the authors randomize the subset of tools available in each training instance.
• They also vary pricing schedules across training instances to expose the model to different cost structures.
• This approach models the variability in tool availability and cost, providing a richer supervisory signal.

4. Tools Used in Training and Evaluation

• A large and comprehensive tool set is prepared for training, but only a subset is sampled for each instance to create diverse configurations.
• For evaluation, a fixed tool set is used, including:
  • Basic tools: Tavily search API for web search, a Python sandbox for a code interpreter, and a Faiss index built with Qwen3-Embedding-8B for local search. Domain-specific functions are also included.
  • Specialized LLMs: GPT-5 and GPT-5-mini as code writers, Qwen2.5-Coder-32B-Instruct as another code writer, and Qwen2.5-Math-72B and Qwen2.5-Math-7B as specialized math models.
  • Generalist LLMs: GPT-5, GPT-5-mini, Llama-3.3-70B-Instruct, and Qwen3-32B.

Method

The authors propose ToolOrchestra, a framework designed to train a small language model, termed the Orchestrator, to function as an intelligent agentic system capable of solving complex tasks by dynamically selecting and coordinating a diverse array of external tools. The core architecture relies on a multi-turn reasoning loop where the Orchestrator iteratively plans and executes actions.

Refer to the framework diagram

The system operates through a unified tool calling interface that standardizes interactions with various resources. The toolset is categorized into basic tools (e.g., local search, web search, code interpreters), specialized LLMs (e.g., for math or coding), and generalist LLMs (e.g., GPT-5, Claude). Each tool is defined via a JSON schema specifying its name, description, and parameter types. For LLM-based tools, descriptions are automatically generated by analyzing model trajectories on sampled tasks. During inference, the Orchestrator follows a reasoning–action–observation loop: it first generates a chain-of-thought to analyze the current state, then selects a tool and specifies parameters, and finally processes the tool's response to inform the next step. This cycle continues until a termination signal is received or a maximum turn limit is reached.

The training of the Orchestrator is conducted via end-to-end agentic reinforcement learning, specifically utilizing Group Relative Policy Optimization (GRPO). The objective is to maximize a composite reward function that balances three distinct objectives: correctness, efficiency, and user preference alignment.

The reward structure is defined as follows:

1. Outcome Reward: A binary reward indicating whether the final answer is correct.
2. Efficiency Rewards: Penalties applied for high monetary costs (converting token counts to API pricing equivalents) and excessive wall-clock latency.
3. Preference Reward: A vector-based reward that aligns the model's tool selection and resource usage with specific user preferences (e.g., prioritizing privacy by preferring local search over web search).

The total reward $R(\tau)$R(τ) for a trajectory $\tau$τ is calculated by normalizing the metrics across a batch and dotting them with a preference vector $P$P. The policy is updated to maximize a clipped surrogate objective, while training stability is maintained through filtering techniques such as homogeneity filtering and format consistency checks.

To support this reinforcement learning process, the authors introduce a data synthesis pipeline named ToolScale.

As shown in the figure below:

This pipeline automates the generation of verifiable multi-turn tool-use tasks. The process begins by selecting a domain and using an LLM to generate a simulated environment, including database schemas, entries, and relevant tool APIs. Subsequently, the system generates diverse user intents and converts them into specific task instructions paired with ground-truth action sequences. To ensure data quality, the pipeline filters out tasks that result in execution errors, cannot be solved by baseline models, or do not require any tool usage. Furthermore, the training data is augmented with heterogeneous tool configurations and pricing schedules to enhance the model's generalization capabilities and robustness to varying constraints.

Experiment

The evaluation used a fixed toolkit of web search, code interpreter, local search, specialized math/code models, and generalist LLMs, comparing the trained Orchestrator-8B against prompted LLM orchestrators and monolithic systems on HLE, FRAMES, and Z-Bench. Orchestrator-8B outperformed all baselines, including GPT-5 and Qwen3-235B-A22B with tools, achieving higher scores on HLE and FRAMES and better Z-Bench function calling at a fraction of the cost. Analyses show it strategically balances tool and model usage, consistently delivers superior performance at any given budget, adapts to tool configurations with unseen models, and adheres to user preferences far better than strong monolithic systems.

Orchestrator-8B, despite having only 8B parameters, outperforms all tested baselines on HLE and FRAMES, including much larger models and those equipped with tool access. It also surpasses GPT-5 on tau^2-Bench when both use basic tools, while incurring significantly lower cost and latency. The results highlight the effectiveness of orchestrating multiple models and tools strategically over relying on a single monolithic system. Orchestrator-8B achieves higher HLE and FRAMES scores than all baselines, including larger models with tool use, with only a small fraction of cost and latency. On tau^2-Bench, Orchestrator-8B outperforms GPT-5 using basic tools by 2.5%. Relative to GPT-5 and Qwen3-235B with tools, Orchestrator-8B gains 2 to 4.3 absolute points on HLE while using far fewer resources.

Orchestrator-8B achieves the highest scores across HLE, Frames, and τ²-Bench while also having the lowest cost and latency among all compared models, including much larger models like GPT-5 and Claude Opus 4.1. The results indicate that its orchestration strategy effectively coordinates diverse tools, leading to superior performance at reduced expense. Orchestrator-8B outperforms all baselines on all three benchmarks, despite having only 8B parameters. It attains top performance with the lowest computational cost and latency, surpassing even larger and more capable models. The model adapts to unseen tool configurations during training, showing strong generalization in using models as tools.

Orchestrator-8B achieves the highest preference adherence scores across all evaluated benchmarks, outperforming much larger monolithic models. The results indicate that orchestration, rather than model scale alone, is key to following diverse user instructions at test time. Orchestrator-8B exceeds all baselines on preference-aware tasks, including state-of-the-art monolithic systems like GPT-5 and Claude Opus 4.1. Despite having only 8B parameters, Orchestrator-8B coordinates stronger models to better satisfy user preferences, demonstrating the effectiveness of strategic tool use. Strong monolithic models such as GPT-5 show notably lower preference adherence, highlighting a key advantage of the orchestration approach.

The evaluation shows that Orchestrator-8B, despite its small size, consistently outperforms much larger baselines, including GPT-5 and Claude Opus 4.1, across HLE, FRAMES, tau^2-Bench, and preference-aware tasks while incurring significantly lower cost and latency. It also generalizes effectively to unseen tool configurations during training. The findings indicate that strategic orchestration of multiple models and tools is more effective than relying on a single large monolithic system, and that preference adherence is better achieved through orchestration than by scaling model size alone.