Command Palette
Search for a command to run...
Weekly Paper Digest | 8 Papers Focus on New Advances in Agent Harness, Covering Coding Agents, Embodied Control, and Image/Video Generation
As AI Agents evolve from simple Q&A to complex task execution, the Agent Harness is gradually becoming a crucial direction in agent research. It serves as the runtime framework connecting large language models, tools, and execution environments, responsible for organizing task workflows, managing context and memory, coordinating tool calls, and handling errors and result validation during execution. Simply put, the model determines what an Agent can think, while the Harness largely determines whether these capabilities can be translated into stable and reliable actions.
Why is the Harness gaining increasing attention? As model reasoning and generation capabilities continue to improve, the challenges facing agents are no longer just about generating correct answers, but also about how to continuously complete multi-step tasks, reduce erroneous operations, adapt to different execution environments, and control risks during autonomous action. The Harness is gradually evolving from a basic framework supporting model operation into key infrastructure that impacts Agent execution effectiveness, application boundaries, and security.
Around these issues, an increasing number of recent studies have begun to explore the design, evaluation, and optimization of Harnesses from various perspectives: some studies analyze the roles of planning, tools, and context management through controlled experiments; others explore more reliable execution mechanisms in terminal operations, robot control, and visual generation; still others focus on safety constraints, automatic construction, and recursive self-improvement, attempting to enable the Harness not only to run but also to continuously optimize based on task feedback.
This issue, we have selected 8 studies related to Agent Harnesses, bringing together relevant explorations from institutions such as NVIDIA, Google Cloud AI Research, Northwestern University, and the University of Science and Technology of China. These cover directions including framework design, execution control, embodied intelligence, visual generation, safety governance, engineering capability evaluation, recursive optimization, and self-evolution. Let’s take a look!
Additionally, the "Papers" section on the HyperAI official website will continue to track cutting-edge achievements in the field of artificial intelligence, helping you stay informed about relevant research developments ⬇️
View the latest papers:
Papers Recommended This Week
1. Harness Design for Coding Agents
Paper Title:
An Empirical Study of Harness Design for Coding Agents
Paper and Detailed Analysis:
This study conducts a systematic empirical analysis of Harness design for coding agents, focusing on how different operational mechanisms affect task performance while keeping the underlying model constant. Researchers conducted controlled experiments around three core components: planning mechanisms, action spaces, and context management, aiming to clarify the specific impact of each Harness component on task success rates, execution efficiency, and resource costs, thereby avoiding judging the effectiveness of a design solely based on the final performance of the complete system.
The research team conducted multiple matched experiments on benchmarks such as SWE-Bench Verified and Terminal-Bench 2.1. The results showed that context management is particularly important for long-horizon tasks, where reasonable handling of historical information can reduce failures caused by context overflow; the benefits of planning mechanisms vary depending on the model's capabilities, and the choice of tool interfaces also needs to be adjusted according to model characteristics. The study indicates that more components in the Harness do not necessarily mean better performance; instead, it should be designed according to task requirements, model capabilities, and resource budgets, providing an empirical basis for subsequent quantifiable Harness optimization research.
2. Mid-Harness
Paper Title:
Mid-Harness: Scaling Actions Between Model and Harness for Terminal Agents
Paper and Detailed Analysis:
Researchers from institutions such as NVIDIA and the Korea Advanced Institute of Science and Technology (KAIST) proposed Mid-Harness, exploring how to introduce action sampling and validation mechanisms between large language models and execution frameworks to improve the execution reliability of terminal agents. Traditional terminal Agents typically execute commands generated by the model directly, but erroneous operations may alter the runtime environment and affect subsequent tasks. Even if the model itself has the ability to generate better commands, it may not provide the optimal choice on the first attempt. Mid-Harness generates multiple candidate actions before actual execution, then compares and filters them through a validator to reduce the impact of improper operations on task progress.
The research team conducted experiments focusing on the sampling methods of candidate actions, the selection of validators, and different execution strategies, analyzing the practical value of allocating additional computational resources between the model and the Harness. The study shows that increasing candidate actions does not necessarily bring benefits; its effectiveness depends on whether the validator can identify better operations. Appropriate comparison and filtering mechanisms can improve action quality and enhance the success rate of the entire task trajectory. This work shifts the optimization focus from solely improving the model's generation capabilities to pre-execution action selection, providing new insights for building more reliable terminal agents.
3. Show-Harness
Paper Title:
Show-Harness: Just a VLM Agent Can Play Robots
Paper and Detailed Interpretation:
Researchers propose Show-Harness to address how Vision-Language Models (VLMs) can translate visual understanding and language reasoning capabilities into actual robotic manipulation. Traditional robotic systems typically require specifically trained Vision-Language-Action models or rely on independent controllers to execute instructions generated by high-level models. Show-Harness designs a semantic action interface, representing robotic manipulation as discrete action units that models can understand and reason about. An interpreter designed for specific robots then converts these actions into actual control commands. Additionally, the research team developed GUMI (GUI Manipulation Interface), which supports collecting robotic manipulation demonstrations via graphical interfaces, reducing reliance on specialized teleoperation hardware.
The research team conducted evaluations across multiple tasks, robot morphologies, and environments to explore the framework's performance in zero-shot control and lightweight fine-tuning scenarios. Results show that Show-Harness enables state-of-the-art VLMs to perform various robotic manipulation tasks without specialized fine-tuning, and can also adapt small open-source models through relatively lightweight fine-tuning. The study also indicates that mechanisms such as situational planning, action history recording, and failure recovery improve execution stability. This work demonstrates that well-designed interaction interfaces between models and environments can unlock the embodied capabilities inherent in foundation VLMs, advancing Harness from a software execution framework to a physical world action control layer.
4. MaLiang-Harness
Paper Title:
MaLiang-Harness: A Programmable Path to Image and Video Generation
Paper and Detailed Interpretation:
Researchers propose MaLiang-Harness to explore extending Harness design principles to the domain of image and video generation. While traditional procedural generation methods can provide explicit composition and animation control, successful program execution does not guarantee that the final visual results meet user expectations. To address this, the framework organizes visual generation driven by multimodal large language models into a continuous process of construction, inspection, and revision. By saving executable programs and task contexts, tracking editing history and rendering evidence, and validating against the current version, the visual generation process becomes more traceable, recoverable, and iterable.
The research team constructed MaLiang-IBench and MaLiang-VBench to evaluate image and video generation capabilities respectively, testing multiple state-of-the-art multimodal large language models. Results show that GPT-6-Astra achieved a 100% generation success rate on both benchmarks, but only 96.0% of image tasks and 76.9% of video tasks met all visual quality thresholds. This result indicates that successful program execution and final visual quality compliance remain distinct evaluation dimensions; similar general model capability scores do not imply equivalent performance in visual generation tasks. This study provides a systematic execution framework for programmable visual generation and expands the application scope of Harness in generative AI scenarios.
5. EvoSafeHarness
Paper Title:
EvoSafeHarness: Evolving Model- and Domain-Specific Harnesses for Securing Agents
Paper and Detailed Interpretation:
Researchers propose EvoSafeHarness to address security protection for agents against direct harmful requests and indirect prompt injections. Traditional security constraints are typically designed by experts in advance and applied across different models and task domains. However, behavioral characteristics of different models and risk requirements across different domains vary, meaning fixed rules may over-restrict normal tasks or miss security risks in specific scenarios. EvoSafeHarness transforms the design of security constraints into an optimizable process, jointly adjusting natural language security policies and executable code logic. By incorporating behavioral feedback from target models, domain norms, and adversarial reviews, it generates constraint suites better suited for specific deployment environments.
The research team evaluated the method on multiple agent safety benchmarks. In the relevant test settings of DecodingTrust-Agent, the method reduced the average attack success rate from 45.6% to 10.0%, with a utility loss of approximately 3.3 percentage points. Under specific evaluation settings in AgentDojo, the attack success rate dropped to 0.0%, while normal task utility reached 82.8%. These results suggest that customizing safety mechanisms for specific models and domains can help improve the balance between safety and task utility. However, safety performance still depends on the type of attack, the task environment, and the scope of evaluation. This work extends the role of Harnesses from task execution to risk control, exploring how to enable agents to adhere to corresponding safety boundaries while completing tasks.
6. Doc 2 Harness
Paper Title:
Doc2Harness: Benchmarking LLMs on Long-Horizon Harness Engineering of Coding Agents
Paper and Detailed Analysis:
Researchers proposed Doc2Harness, aiming to directly evaluate the engineering capabilities of large language models in building complete Agent Harnesses. Previous studies typically tested agent task performance within pre-designed execution frameworks, rarely examining whether models could independently construct truly runnable execution frameworks based on development documentation and interface contracts. Doc2Harness designs the evaluation in two stages: first, the model generates candidate Harnesses based on interface contracts and development documentation; subsequently, the research team runs the framework in an isolated environment, checking whether its actual behavior meets requirements through observable information such as model requests, disk files, process exit codes, and command outputs.
The research team used this benchmark to evaluate 19 state-of-the-art models, with each model performing three independent constructions for 328 test cases. The results showed that the best-performing model, GPT-6 Astra, achieved a pass rate of 53.05%, while the weakest model scored 27.44%; even the best-performing model failed nearly half of the test cases. Failure issues were primarily concentrated in features requiring implementation across multiple modules and the division of responsibilities between different modules. The study indicates that a model's ability to generate code does not imply it possesses the complete engineering capability to build complex execution infrastructure. Doc2Harness provides a deterministic evaluation benchmark for measuring long-horizon Harness engineering capabilities and offers reference for future improvements in the automatic construction of agent frameworks.
7. Harness²
Paper Title:
Harness²:Recursive Agent Harnessing for an Open World
Paper and Detailed Analysis:
When agents face real-world tasks that are constantly changing and difficult to predefine, traditional Harnesses relying on manual design and fixed configurations often struggle to adapt in a timely manner. To address this issue, the study proposes Harness², attempting to enable agents not only to complete tasks but also to continuously improve their own execution frameworks during execution. The method constructs two interrelated recursive levels: task-level recursion uses a Propose–Probe–Compose process to propose candidate framework modifications for the current task, conduct comparative tests, and decide whether to adopt them based on actual execution trajectories and task outputs; framework-level recursion extracts reusable experiences from these modifications, gradually updating the approach for subsequent framework optimization. Even without ground-truth answers that directly determine task correctness, the system can provide a basis for framework adjustments by comparing execution performance under different configurations.
The research team conducted evaluations on three specialized benchmarks and four model-framework configurations. Experiments showed that single-step framework improvements enhanced the performance of GLM-5 and Gemini 3.5 Flash on multiple benchmarks: up to an 11.5-point increase on WorkBuddy-Bench, a 4.5-point increase on JobBench, and an 11.8 percentage point increase on LAB. As the task-level recursive budget increased, the magnitude of performance improvement could further expand. The significance of Harness² lies in its attempt to bridge the connection between "completing the current task" and "accumulating experience to improve subsequent tasks," transforming Harnesses from one-time configurations into execution systems capable of continuously adapting to new tasks, and providing a new approach for agent self-improvement in open environments.
8. Self-Evolving Agent Harness
Paper Title:
Self-Evolving Agent Harness
Paper and Detailed Analysis:
If previous research focused on how to design Harnesses, control actions, or recursively optimize task performance, this study raises a more holistic question: when agents require long-term operation, can a Harness develop a mechanism for continuous self-updating? The study defines "self-evolution" as a feedback-driven process of continuous revision: without altering the underlying model weights, the system can adjust prompts, memory, workflows, and executable components based on execution experience, retaining accepted changes for subsequent runs. Adjustable objects include not only individual modules but also tools and code, repository-level control logic, and the compositional relationships among different plugins and components. Meanwhile, the study emphasizes that proposing modifications, judging their effectiveness, and deciding on formal adoption are three distinct processes that must be handled separately; an improvement in a single task’s performance is not sufficient to establish the long-term value of a change.
At the theoretical level, the paper combines evolutionary optimization with continual PAC learning to analyze the generalization and performance regression issues that may arise during continuous Harness updates. Under clearly defined assumptions, it discusses expected generalization bounds, retention bounds related to historical regression budgets, and admission rules for candidate changes based on fresh, independent validation data. Unlike simply comparing scores on a specific benchmark, this study focuses more on how the Harness retains effective experience through repeated iterations while reducing overfitting, forgetting, and performance degradation on old tasks. The paper also discusses deployment issues such as shifts in task distributions, evaluator bias, and concerns regarding scientific reliability, physical safety, and organizational accountability, noting that continuous validation and independent safety safeguards remain indispensable. The direction it outlines is that the Harness is no longer merely a static software structure built around a model, but may become a system capable of accumulating feedback, prudently modifying itself, and undergoing continuous verification.
