One-sentence Summary

Researchers from Beijing Academy of Artificial Intelligence et al. propose DisCo, a skill-powered research agent that distills GitHub repositories into operational skills to create the AREX-Skill Library of over 5,000 verified skills from 1,000 ML repositories, boosting the agent's performance by 134.3% on MLE-bench, 34.4% on PaperBench, 9.2% on FrontierCS, and 14.0% on PassNet relative to a skill-less agent.

Key Contributions

• The paper introduces operational knowledge as a missing layer for autonomous ML research agents and presents DisCo, a method that distills version-specific know-how from static artifacts (papers and repositories) into reusable, verified skill graphs, which agents can load as operating context without altering the model backbone or research harness.
• DisCo's distillation runs in two complementary modes: a task-agnostic process that condenses widely used ML repositories into skills, and a task-oriented process that produces skills for specific research tasks; scaling the former yields the AREX-Skill Library with over 5,000 verified skills from 1,000 repositories, organized into 20 areas and 178 capability families.
• Under a fixed GPT-5.5 backbone and matched downstream execution budgets, the skill-equipped agent scores 134.3% higher on MLE-bench, 34.4% higher on PaperBench, 9.2% higher on FrontierCS, and 14.0% higher on PassNet compared to the same agent without skills, demonstrating that adding distilled operational knowledge substantially improves autonomous research performance.

Introduction

Autonomous agents for ML research are typically built from a large language model and a harness that orchestrates reasoning and execution, but both components lack the domain-specific operational knowledge needed to select appropriate methods, configure tools correctly, and avoid common pitfalls. This missing layer forces agents to waste execution budget on trial and error and prevents them from reusing insights across tasks. The authors identify operational knowledge as the critical missing ingredient and introduce DisCo, a skill-powered agent that distills such knowledge from existing repositories and papers into compact, verified skills. By equipping a fixed model and harness with these skills, they achieve substantial performance gains on benchmarks including MLE-bench and PaperBench, demonstrating that operational knowledge alone can significantly boost autonomous research capabilities.

Dataset

The authors introduce the AREX-Skill Library, a persistent store of operational knowledge graphs used by the DisCo system. The library is organized around three source anchors: a curated snapshot of 1,000 ML repositories, paper-derived skill graphs, and task-oriented skill graphs. Below is a breakdown of its composition, processing, and usage.

• Repository-derived skills

  • Source and scope: 1,000 open-source ML repositories selected for visibility and practical utility, using signals like GitHub stars. The collection covers model implementations, training and deployment systems, data/evaluation tools, and scientific software.
  • Graph construction: For each repository, DisCo builds a verified skill graph using GPT-5.5 and GPT-5.6-sol with high reasoning effort (average cost ~$40 per repo). The evidence boundary includes source code, documentation, examples, tests, scripts, and configuration files. The graph decomposes supported workflows into skills for data preparation, training, inference, evaluation, serving, troubleshooting, or maintenance, with references and scripts that retain execution details.
  • Verification and filtering: Before inclusion, each graph is checked with assertion-backed cases, repository-native examples, tests, CLI checks, tiny-fixture checks, or smoke scripts. Failures attributed to the graph trigger local repair and reruns. This yields 5,353 skills across the 1,000 graphs.
  • Taxonomy and routing: A two-level capability taxonomy (20 areas and 178 families) is induced from short repository summaries, excluding stars, URLs, and pre-existing categories. The taxonomy is fixed before final assignment. Each verified graph is then classified against exact area-to-family paths, with assignments requiring a rationale, repository evidence, and confidence. Keyword-only, dependency-only, optional-integration, and example-only matches are rejected. A repository can belong to multiple families; 2,209 exact assignments are recorded, with 700 repositories appearing in more than one family. A router generated from these assignments implements progressive disclosure during use.

• Paper-derived skills

  • Source: 153 prior papers selected as related work for 20 PaperBench targets. Each paper is decomposed into module-level skills covering method components, data/evaluation procedures, and implementation workflows.
  • Construction and filtering: Every module is checked in isolation, followed by a bounded recovery experiment that excludes the original implementation repository. This produces 636 paper-derived skills. The target paper and its released artifacts are excluded as skill sources; related-work context determines which source-paper skills are available to each target. The resulting skills remain reusable beyond the target tasks.

• Task-oriented skills

  • MLE-bench: One descriptive graph per competition for 75 tasks, built through task decomposition, source discovery, and bounded diagnostic trials. Competition-specific content is excluded.
  • FrontierCS: A single recovery-oriented graph shared across all 188 Agent Track tasks.
  • PassNet: One benchmark-level graph covering FX-graph inspection, pattern matching, semantics-preserving rewrites, Triton implementation, and performance diagnosis.
  • All task-oriented graphs are constructed via task decomposition and diagnostic trials, with details in the appendices.

• How the library is used

  • The library serves as the operational knowledge store for DisCo. Creator mode writes accepted skill graphs into the library; researcher mode retrieves a task-relevant branch as operating context.
  • During research, the model starts with the router description, follows the relevant area-to-family path, opens the selected repository graph, and loads only the skills, references, or scripts needed for the current step. It can open multiple graphs when they provide distinct capabilities, but it does not force a route when no family is a close fit. This progressive disclosure ensures only the selected branch enters the operating context.

Method

The authors propose DisCo, a framework that instantiates operational knowledge as a layer of skills to bridge the gap between a model's broad prior and the specific procedural requirements of a research task. DisCo operates as a single research agent that functions in two distinct modes: Creator mode and Researcher mode, sharing the same backbone model and harness.

Refer to the framework diagram:

In Creator mode, the agent performs skill distillation to convert declarative source knowledge into executable operational knowledge. This process follows a four-stage pipeline regardless of the anchor that initiates it. First, the agent scopes the capability by identifying which skills are worth exposing from the source material. Second, it grounds the evidence by gathering and filtering the relevant documentation, code, or papers. Third, it constructs a candidate skill graph by encapsulating tools behind stable interfaces and assembling them into a structured format. Finally, it verifies and refines the graph through assertion-backed checks and task trials, recording any unresolved gaps. The distillation can be task-agnostic, where the anchor is a source like a repository or paper, or task-oriented, where the anchor is a specific problem and the agent actively seeks out missing capabilities.

The operational knowledge is structured as a skill graph $\mathcal{G} = (\mathcal{S}, \mathcal{L})$G=(S,L), where each skill $S$S is organized into three layers: a knowledge interface (SKILL.md) that provides the standard operating procedure and routing logic, a knowledge substrate (references/) containing deeper API and algorithmic details, and an execution interface (scripts/) consisting of executable wrappers. This separation ensures that the agent only loads the necessary information.

In Researcher mode, the agent solves a task $\tau$τ by drawing operational knowledge $\mathcal{K}$K from the AREX-Skill Library. The authors employ a progressive disclosure principle to manage context efficiently. Instead of loading entire graphs, the agent first consults a router or entry skill to identify relevant branches. It then opens only the specific skills required for the current step, following links for setup, evaluation, or repair as needed. This allows the harness to maintain control over planning and execution while the skill graph provides selective, task-specific operating context.

The AREX-Skill Library serves as the persistent store connecting the two modes. It organizes distilled skill graphs by source anchor, covering thousands of repositories, papers, and benchmark tasks. To facilitate efficient retrieval, the library utilizes a two-level capability taxonomy and a generated router.

As shown in the figure below:

The library router narrows a request from a broad area to specific package families, then to individual repository graphs, and finally to the relevant skills. This structured retrieval path ensures that the agent inherits the distilled operational knowledge without re-deriving it, making the system scalable across diverse machine learning domains.

Experiment

Across four diverse benchmarks, adding DisCo-distilled skills as operating context to a fixed Codex agent consistently improves performance, with the largest gains on harder tasks and on tasks where the baseline agent struggles, while also reducing failures and incorrect outcomes. The improvements are not explained by increased resource usage alone, and the skill-equipped agent surpasses strong baselines, demonstrating that externalized operational knowledge significantly enhances ML research capabilities without altering the agent’s backbone or harness.

Adding distilled operational skills to the Codex agent substantially improves its performance on MLE-bench, raising the overall Any-Medal score from 31.11% to 72.89%. This gain is consistent across all difficulty tiers and is largest on High-difficulty tasks, where the relative improvement reaches 366.8%. With skills, Codex also surpasses the strongest public baseline by 8.45 percentage points, without modifying the agent backbone or execution harness. Codex with skills achieves an overall Any-Medal score of 72.89%, a 41.78 percentage point gain over the no-skill baseline. The improvement is most pronounced on High-difficulty tasks, where the score rises from 13.33% to 62.22%. Skills enable Codex to outperform the best public baseline by 8.45 points overall, with the largest margin (15.55 points) on High tasks. The relative gain grows with task difficulty, suggesting that distilled knowledge is especially valuable for complex ML problems.

Equipping GPT-5.5 Codex with distilled skills raises the average PaperBench replication score by 10.14 points, a 34.4% relative improvement. Gains are broad, appearing on 18 of 20 tasks, and are largest on tasks where the baseline agent struggles, sometimes multiplying the score several-fold. A small number of higher-scoring tasks regress slightly, consistent with occasional retrieval-precision failures that distract from effective unaided strategies. Average replication score rises from 29.45% to 39.59% when skills are added, a 34.4% relative improvement. Skills improve performance on 18 of 20 tasks, with the largest relative gains on low-baseline tasks such as ftrl (from 1.50 to 17.17) and rice (from 7.94 to 48.51). Two tasks with above-average baseline scores regress (samplespecific-masks and stay-on-topic), suggesting that retrieved skill content can occasionally mislead the agent. The skill-equipped agent achieves higher scores while using substantially fewer tokens, steps, and tool calls than Claude Code configurations, Pareto-dominating them on the leaderboard.

Providing Codex with a distilled skill graph raises its FrontierCS Agent Track score from 70.63 to 77.14, an absolute gain of 6.51 points and a 9.22% relative improvement. This skill-equipped Codex achieves the highest overall score while using substantially fewer tokens, steps, and tool calls than the Claude Code entries, and it particularly lifts performance on tasks where unguided exploration struggles. Adding the frozen skill graph to Codex improves performance on 74 of 188 tasks, with an average gain of 22.23 points on improved tasks versus an average loss of 8.76 points on degraded ones. The largest benefit occurs on tasks with a no-skill score below 50, where the mean rises from 19.43 to 45.99 and 30 tasks cross the 50-point threshold. Codex with skills achieves the highest score (77.14) while using only 4.47M tokens per task, compared to over 13.8M tokens for the Claude Code configurations that score lower. Score gains are essentially uncorrelated with additional resource usage (Spearman’s ρ ≤ 0.015 for tokens, steps, and tool calls), indicating that skills improve performance beyond raw scaling.

Equipping Codex with a distilled PassNet skill substantially improves graph-compiler pass generation. The skill raises aggregate score, correctness, and speedup while reducing failed samples, enabling the agent to surpass both its unguided counterpart and the TorchInductor compiler baseline. Adding the distilled skill boosts the AS Score by 14% relative to the no-skill Codex baseline. Failed samples fall from 14 to 5, a 64% reduction, and correctness climbs from 81% to 91%. Skill-equipped Codex achieves a higher aggregate score than TorchInductor (1.5313 vs. 1.419). Geometric-mean speedup increases from 1.5891 to 1.6688 with the skill.

Across four diverse benchmarks, equipping the Codex agent with distilled skills consistently boosts performance, with the largest relative gains on complex tasks where the baseline struggles. The skill-equipped agent surpasses strong public baselines and Claude Code configurations while using substantially fewer tokens and tool calls, indicating that distilled operational knowledge improves capability beyond raw scaling. Gains are broad, though occasional regressions suggest that retrieval imprecision can mislead the agent on a small number of tasks. Overall, these experiments validate that incorporating distilled skill graphs and operational patterns significantly enhances an agent's ability to solve challenging ML engineering and research problems.