What’s the best code agent?

Tasks are sourced from SWE-bench Verified, which tests agents on 500 GitHub issues from 12 open-source Python repositories: Astropy, Django, Matplotlib, seaborn, Flask, Requests, PyData, Pylint, pytest, scikit-learn, Sphinx, and SymPy.

• No lab’s agentic programming tool is able to complete long-horizon (>4 hour) instances.
• Cursor CLI with Grok Code Fast 1 and Gemini CLI with Gemini 2.5 Pro spend exponentially more time on long-horizon tasks than short-horizon and medium-horizon tasks.
  • Claude Code with Claude Sonnet 4.5 and Codex CLI with GPT 5 spend just marginally more time.
• Claude Code with Claude Sonnet 4.5 and Codex CLI with GPT 5 cost more solving SWE-bench Verified’s 1-4 hour tasks than its >4 hour tasks.

• Claude Code with Claude Sonnet 4.5 is the most proactive code agent. It has an especially strong preference to run tests with execute tool calls.
  10 Randomly Sampled Claude Code with Claude Sonnet 4.5 Execute Tool Calls

  • python -m pytest /testbed/lib/matplotlib/tests/test_pickle.py /testbed/lib/matplotlib/tests/test_legend.py /testbed/lib/matplotlib/tests/test_offsetbox.py -x -q 2>&1 | tail -30
  • grep -n "def test_ordering" tests/invalid_models_tests/test_models.py | head -20
  • grep -l "test_ordering" tests/invalid_models_tests/test_models.py
  • python -m pytest /testbed/lib/matplotlib/tests/test_pickle.py -xvs
  • python /testbed/test_final_verification.py
  • python test_empty_name.py
  • python /testbed/test_autodetector_integration.py
  • python tests/runtests.py template_tests -v 2 2>&1 | grep -A 5 "FAILED\|FAIL:"
  • ./tests/runtests.py migrations.test_optimizer migrations.test_autodetector --parallel 1 -v0 2>&1 | tail -10
  • python -m pytest /testbed/sklearn/feature_selection/tests/ -v -k "cv" 2>&1 | head -100
• Codex CLI with GPT 5 executes far less code than Claude Code with Claude Sonnet 4.5.
• Gemini CLI with Gemini 2.5 Pro pulls in little context via reading the codebase, but it writes about the same amount of code as Claude Code with Claude Sonnet 4.5 and Codex CLI with GPT 5.

• Code agents run in SWE-bench Verified task-specific docker containers with our user prompt. The latest code agent version at the time of publication is used.
  User Prompt

  
  <uploaded_files>
  
  {location}
  
  </uploaded_files>
  
  I've uploaded a python code repository in the directory {location} (not in /tmp/inputs). Consider the following PR description:
  
    
  
  <pr_description>
  
  {pr_description}
  
  </pr_description>
  
    
  
  Can you help me implement the necessary changes to the repository so that the requirements specified in the <pr_description> are met?
  
  I've already taken care of all changes to any of the test files described in the <pr_description>. This means you DON'T have to modify the testing logic or any of the tests in any way!
  
    
  
  Your task is to make the minimal changes to non-tests files in the {location} directory to ensure the <pr_description> is satisfied.
  
    
  
  Follow these steps to resolve the issue:
  
  1. As a first step, it might be a good idea to explore the repo to familiarize yourself with its structure.
  
  2. Create a script to reproduce the error and execute it with `python <filename.py>`, to confirm the error
  
  3. Edit the sourcecode of the repo to resolve the issue
  
  4. Rerun your reproduce script and confirm that the error is fixed!
  
  5. Think about edgecases and make sure your fix handles them as well
  
    
  
  Your thinking should be thorough and so it's fine if it's very long.
  

• Code agents execute in headless mode on a 32 vCPU, 64 GiB memory EC2 instance with a 2 hour timeout. Latency is the runtime of the command which invokes the code agent. Cost is from API pricing.
• Accuracy (whether the task is resolved), latency, and cost is averaged across instances in visualizations. One pass was made on each instance.
• Code agent performance might be inflated by data contamination, since SWE-bench Verified tasks are public GitHub issues.