In recent years, large language models and intelligent agent technologies have evolved rapidly, and the boundaries of AI's participation in scientific research are constantly expanding. From literature review and code writing to data analysis, more and more standardized scientific research processes can be completed with AI assistance, and some intelligent agent systems have even begun to attempt to independently undertake specific scientific research sub-tasks. However, a more critical question still lacks a systematic answer: when researchers no longer provide clear methods and execution paths, but only present a scientific problem to be solved...To what extent can AI autonomously complete research?

Existing research benchmarks struggle to answer this question. One type of evaluation primarily assesses a model's grasp of existing scientific knowledge, essentially remaining close to "knowledge-based question answering with known answers." Another type, while incorporating code execution and tool usage, often predefines research methods or operational procedures, mainly testing whether the AI can complete tasks according to a predetermined process. These benchmarks can measure knowledge understanding and research execution capabilities, but they struggle to further differentiate between different areas:Is AI truly capable of autonomously designing research solutions, or is it merely replicating solutions already provided by humans?

To this end, more than 40 domain experts from over ten institutions, including Tsinghua University, MIT, Harvard University, Carnegie Mellon University, and Microsoft Research, have collectively invested over 31,000 hours to jointly build the ASI-Bench, a benchmark for independent research capabilities. Its core idea is not simply to increase the difficulty of the problems, but rather to progressively reduce the amount of methodological information provided by humans for the same research project, observing how the AI's performance changes, thereby quantifying the system's dependence on human guidance.

The related research findings, titled "ASI-Bench: At the Dawn of Artificial Superintelligence," have been published on the preprint platform arXiv.

View the paper:
https://hyper.ai/papers/2608.17271

Dataset: A project-level research task repository covering 11 major disciplines

ASI-Bench is based on a set of project-level tasks modeled after real-world research problems. Unlike knowledge-based questions or standardized exercises in traditional benchmarks, each task is designed as a relatively complete micro-research project, requiring the agent to go through multiple stages such as problem understanding, method selection, code implementation, experimental execution, error checking, and result iteration.

The task initially came from more than 1,300 academic papers and cutting-edge research questions, covering multiple fields such as basic science, life science, and engineering technology.The research team first selects representative candidate topics by field experts, then performs engineering modifications and feasibility verifications to ensure that the originally open research questions can be executed and evaluated in a unified environment. Tasks with scientific logical flaws, results that are difficult to reproduce, unstable operating conditions, or significantly unbalanced difficulty are eliminated in this process.

final,ASI-Bench retains 60 research tasks.It covers 11 fields: mathematics, physics, chemistry, biology, astronomy, materials science, earth science, medical biostatistics, computer science, robotics, and electrical engineering. Different tasks involve different data formats, research methods, and validation standards, thus preventing benchmarks from being dominated by a specific discipline or fixed problem-solving methods to some extent.

Each task includes complete research objectives, specific input data, an executable environment, clear output requirements, and quantitative scoring rules. After entering a task, the agent needs to plan the research process independently based on the information obtained and complete the experiment using code and tools. The 60 tasks involved a total of over 2,600 rounds of agent interaction and 2,400 code executions.The cumulative time for a single agent to complete all tasks exceeds 35 hours.

Given the inherently open nature of scientific research tasks, ensuring that the benchmarks are "truly feasible" and that the scoring is "effective" is paramount. To this end, the research team conducted five rounds of cross-expert review, accumulating over 1,100 review assignments and over 2,000 task revisions. Building upon this foundation, all tasks underwent over 1,500 end-to-end trials in an isolated sandbox to confirm the reproducibility of the reference solutions and the stable operation of the scoring program. Ultimately, this resulted in a project-level research task library that balances scientific rationality and engineering feasibility.

Step-by-step removal of methodological guidance, quantifying research autonomy

ASI-Bench's most recognizable design.The key is that it doesn't design different problems for different abilities, but instead puts the intelligent agents in the same research project, only changing the methodological information provided by humans.The research objectives, input data, operating environment, final output, and scoring criteria remain unchanged; the only difference is the amount of research guidance the system receives. By observing the changes in the model score as information is gradually reduced, we can further determine whether the difficulties in AI stem from execution, method implementation, or the autonomous selection of research paths.

ASI-Bench sets each task into four levels: B1, B2, B3, and B4.

B1:Complete methodological guidance. The system can obtain a relatively complete research plan, including governing equations, numerical methods, key implementation steps, and parameter settings. The intelligent agent mainly needs to convert the existing plan into code and complete its execution, thus more closely resembling an assessment of scientific research execution capabilities.

B2:Only the method category is provided. Detailed implementation steps are removed; the system can only obtain the method category and core principles. For example, the system may know that a certain problem is suitable for solving using spectral methods, but how to discretize, what integration strategy to use, and how to set the parameters all need to be determined manually. This level mainly examines whether the AI can transform abstract methods into a truly workable research process.

B3:Only research objectives and data are provided. All methodological hints are removed; the agent can only see the raw data, research objectives, and output requirements. It needs to independently determine the nature of the problem, select research methods, design computational processes, and complete experiments. This is also the core level of ASI-Bench's measurement of independent research capabilities.

B4:Add irrelevant and distracting information. Building upon B3, add factually correct but task-irrelevant background material, unverified methodological guesses, and other scenario information to test the system's ability to identify truly research-related content and maintain normal research processes in complex information environments.

This setup makes "independent research" no longer just a general concept, but can be broken down into multiple consecutive stages. For example, if the model performs well in B1 but drops significantly in B2, it indicates that the problem may not be with executing the code, but with whether it can complete the research plan on its own; if it drops sharply again from B2 to B3, it indicates that the choice of methodology and research design are the new major obstacles.

To minimize interference from external factors, ASI-Bench executes all tasks in an isolated sandbox. The agent can only access the data and tools provided beforehand for the task; it cannot search online or access external knowledge bases. Therefore, the final results reflect the model's own reasoning, planning, and execution capabilities more than its retrieval abilities.

Based on the four-level information gradientASI-Bench can further analyze the differences between different capabilities:The changes in B1 and B2 reflect the ability to translate methods into specific processes, the changes in B2 and B3 reflect the ability to autonomously select methods and design research schemes, and the changes in B3 and B4 reflect the robustness of the system in the face of redundant information.

Experimental Verification: Performance and Capability Boundaries of 18 Cutting-Edge Systems

Based on ASI-Bench,The research team evaluated 18 cutting-edge systems composed of an "intelligent agent framework + large-scale base model".It covers intelligent agent frameworks such as Codex, Claude Code, Kimi Code, MiMo Code, and OpenHands, as well as models such as GPT-5.5, GPT-5.6 Sol, Claude Opus, GLM, DeepSeek, Kimi, MiMo, and MiniMax. All systems run in isolated environments, with each task executed independently three times and the average result taken.

The experimental results show thatWith reduced human guidance, AI's research performance declined rapidly. The 18 systems achieved an average score of 50.91 under the complete method guidance in B1. Upon entering B2, which only provided method categories, the average score dropped to 29.10. In B3, where method hints were completely removed, the average score further decreased to 26.62. The best-performing system, Codex + GPT-5.6 Sol (ultra), scored 51.60 in B3, the only configuration exceeding 50. Increasing the inference strength improved the results; for example, upgrading GPT-5.6 Sol from xhigh to ultra increased the B3 score by 10.74 points. However, overall, the current systems still have a significant gap to bridge before achieving stable open-ended autonomous research.

The score variations between different information levels revealed more specific skill gaps. From B1 to B2, the system's average score decreased by 21.82 points; while from B2 to B3, after further removing method category hints, the average score only decreased by 2.48 points. This means that...The main difficulty in AI today is not necessarily deciding "which method to use", but rather how to further transform existing methodological ideas into a complete process consisting of parameter selection, code implementation, experimental operation, and result verification.Meanwhile, B4 scored an average of 26.99 after adding irrelevant and interfering information, which is basically the same as B3's 26.62, indicating that, at least under the current task settings, information noise is not a major limiting factor.

The experiment also revealed that autonomous research capability is not entirely determined by the underlying model. The same model paired with different agent frameworks can yield significantly different results. For example, MiMo V2.5 Pro, when paired with Claude Code, achieved an overall score improvement of over 40% compared to MiMo Code; Kimi K2.7 also performed better under the Claude Code framework than Kimi Code. This indicates that agent mechanisms such as task planning, tool invocation, error correction, and process scheduling directly impact the quality of research task completion.

Finally, there is no simple linear relationship between cost and performance across different systems. For example, the cost per run of Codex + GPT-5.6 Sol (xhigh) is about a quarter that of Claude Opus 5, yet their scores in B3 are similar; the best-performing GPT-5.6 Sol (ultra) costs approximately $1550 per run. Therefore, in practical scientific applications, computational efficiency must be considered in addition to model capabilities. Higher inference investment can increase the upper limit of capabilities, but whether research tasks can be completed stably at a reasonable cost is also a problem that autonomous research systems must face in order to move towards practical application.

Final Thoughts

For a long time, the evaluation of AI's research capabilities has focused on answering scientific questions, reproducing paper results, or completing assigned experiments. ASI-Bench, however, goes further: when humans no longer provide specific methods and execution paths, how much research work can AI independently complete? The evaluation results show that AI can already complete tasks with clearly defined paths quite well, but there is still a significant gap between its current capabilities and its ability to stably conduct open-ended autonomous research. The future improvement of research agents cannot rely solely on stronger models and more computing power; it also requires enhanced capabilities in task planning, utilizing experimental feedback, and error correction. The measurement of AI's research capabilities will gradually shift from "how many questions it answers correctly" to "how many real-world research tasks it can independently complete."