One-sentence Summary

Princeton University researchers propose SWE-agent, a system that provides language model agents with a custom agent-computer interface to autonomously perform software engineering tasks, achieving state-of-the-art pass@1 rates of 12.5% on SWE-bench and 87.7% on HumanEvalFix by enhancing code editing, repository navigation, and test execution.

Key Contributions

• SWE-agent is introduced, a system that equips language model agents with a custom agent-computer interface (ACI) for autonomous, end-to-end software engineering tasks.
• The ACI significantly enhances the agent's ability to create and edit code, navigate repositories, and execute tests, achieving state-of-the-art pass@1 rates of 12.5% on SWE-bench and 87.7% on HumanEvalFix, far exceeding prior non-interactive LMs.
• Analysis shows that crafting LM-centric interactive components within the ACI meaningfully impacts agent behavior and downstream task performance.

Introduction

Language model agents have recently proven effective for code generation when paired with execution feedback, but extending them to full software engineering tasks remains challenging. Humans rely on sophisticated interfaces like IDEs, yet existing LM agents typically operate through the raw Linux shell, where they struggle with simple edits, receive no error guidance, and lack safeguards against common mistakes. The authors introduce SWE-agent, a system built around an agent-computer interface (ACI) that mediates between the language model and the computer. By providing a small set of well-guarded actions for viewing, searching, and editing files, along with concise per-turn feedback, the ACI dramatically improves reliability. SWE-agent achieves a 12.47% resolve rate on the SWE-bench benchmark using GPT-4 Turbo, far surpassing the prior 3.8% from a non-interactive retrieval-augmented method, and demonstrates that thoughtfully designed interfaces tailored to LM strengths can lift performance without altering model weights.

Method

The authors propose the Agent-Computer Interface (ACI), a specialized framework designed to enable large language models to act as effective software engineering agents. Unlike traditional graphical user interfaces designed for humans, the ACI is tailored to the specific capabilities and limitations of language models. For instance, current models lack the visual understanding required to operate GUI-based applications directly, and they are highly sensitive to distracting context due to fixed memory and computation costs. To address these disparities, the authors shape the actions, documentation, and environment feedback to complement the model's abilities.

Refer to the framework diagram below:

The ACI framework provides language model agents with essential functionality to interface with computers, analogous to how integrated development environments enhance human productivity. The system operates by having the model generate a thought and a command at each step, subsequently incorporating the feedback from the command execution in the environment. This iterative process is built atop the Linux shell, granting access to common utilities while enforcing specific design principles. Actions are kept simple and easy to understand, avoiding complex commands with dozens of options. Furthermore, operations are consolidated into compact and efficient actions to ensure meaningful progress in a single step. Environment feedback is designed to be informative yet concise, providing substantive information about the current state without unnecessary details. Guardrails, such as automated code syntax checkers, are also integrated to mitigate error propagation and hasten recovery from mistakes.

The SWE-agent implementation of the ACI comprises several principal modules including search and navigation, file viewing, file editing, and context management. For search and navigation, the authors introduce specialized commands to locate filenames and strings within files or directories. These commands output summarized results and suppress verbose outputs, returning at most 50 results per query to encourage efficient and specific searches.

The file viewer module allows agents to interact with code files by presenting a window of up to 100 lines at a time. Agents can navigate this window using scrolling commands or jump to specific lines. The interface displays the full file path, total line count, omitted lines, and line numbers to facilitate precise code localization.

For file editing, the system provides an edit command that works in conjunction with the file viewer. Agents can replace a specific range of lines by providing the start line, end line, and replacement text. Upon execution, the file viewer automatically displays the updated content. To prevent format errors, a code linter is integrated into the edit function. If the linter detects mistakes, it alerts the agent with relevant error snippets, discards the invalid edit, and prompts the agent to retry.

Context management is handled through informative prompts, error messages, and history processors. The system instructs the agent to generate both a thought and an action at each step. Malformed generations trigger specific error responses, which are omitted after the first occurrence to save context. To maintain relevance and reduce unnecessary context, observations preceding the last five are collapsed into a single line. This approach preserves essential information about the plan and action history while allowing for more interaction cycles and preventing the display of outdated file information.

Experiment

The evaluation on SWE-bench and HumanEvalFix shows that SWE-agent with GPT-4 Turbo significantly outperforms both retrieval-augmented generation and shell-only baselines, demonstrating the value of a carefully designed agent-computer interface. Ablations reveal that efficient search and compact multi-line editing are essential, while guardrails such as linting help agents recover from erroneous edits. Behavioral analysis indicates that agents follow a reproduction-and-localization step followed by edit-execute loops, succeed quickly when they do, and most failures stem from incorrect implementations or cascading edit errors.

SWE-agent with GPT-4 Turbo achieves the highest resolve rates on SWE-bench, solving 18.00% of Lite instances but at a significantly higher cost than retrieval-augmented generation. Adopting an LM-friendly interface yields a 64% relative improvement over a shell-only agent, while editing remains a key challenge: over half of unresolved instances are caused by incorrect or overly specific implementations. SWE-agent with GPT-4 Turbo solves 18.00% of SWE-bench Lite, a 64% relative increase over the shell-only agent's 11.00%. Compared to RAG on Lite, SWE-agent offers a 6.7-fold higher resolve rate but is 8-13 times more expensive per instance. In failure analysis, 52.0% of unresolved trajectories are due to incorrect or overly specific implementations, and cascading failed edits account for another 23.4%.

SWE-agent paired with GPT-4 Turbo achieves dominating pass@1 rates on the HumanEvalFix benchmark across Python, JavaScript, and Java, averaging around 88.3%. It surpasses the next best model, WaveCoder-DS-6.7B, by roughly 30 percentage points in each language, while standalone GPT-4 only reaches the 47–50% range. SWE-agent with GPT-4 Turbo sets the highest pass@1 rates in all three languages: 87.7% (Python), 89.7% (JavaScript), and 87.9% (Java). The margin over the next best model is over 30 percentage points in Python and Java, and even larger in JavaScript. GPT-4 without an agent framework scores between 47% and 50%, while the strongest 6.7B-parameter model (WaveCoder-DS-6.7B) reaches up to 57.9% on Python.

Ablating the SWE-agent interface shows that the dedicated edit action with linting is the most critical component, with its absence causing the largest drop in resolved tasks. The summarized search method is beneficial, while iterative search surprisingly underperforms having no search tool at all. File viewer window size and context management also influence performance, with 100 lines and only the last five observations being optimal, respectively. Iterative search yielded 12.0% resolved tasks, lower than the no-search condition's 15.7%. Removing the edit action caused the largest performance decline, dropping the solve rate from 18.0% to 10.3%. A file viewer window of 100 lines gave the highest success; reducing it to 30 lines or showing the full file lowered performance to 14.3% and 12.7%. Using only the last five observations performed best; full history decreased success to 15.0%, and omitting demonstrations to 16.3%.

The evaluations cover SWE-agent's performance on SWE-bench Lite for repository-level issue resolution and on HumanEvalFix for cross-language bug fixing, along with interface ablations. The agent's LM-friendly interface with dedicated editing and linting dramatically improves over a shell-only baseline, while editing failures remain the primary obstacle in unresolved instances. The agent-driven GPT-4 Turbo achieves dominating pass@1 scores on HumanEvalFix across languages, far surpassing standalone models, and ablation reveals that the edit action is the most critical component, with optimal context management (a 100-line viewer and limited history) further boosting success.