# SWE-bench - Deep Dive

> **TL;DR**: SWE-bench is THE benchmark for evaluating AI coding agents on real software engineering tasks. Created by Princeton, it tests LLMs on 2,294 real GitHub issues from 12 popular Python repos (Django, scikit-learn, Flask, etc.). If you've seen claims like "GPT-4 solves X% of bugs" or leaderboards ranking coding agents, SWE-bench is the standard they're using. Open source, used by OpenAI, Anthropic, Meta, and everyone else.

---

## What Is SWE-bench?

**SWE-bench** is a benchmark that tests whether AI systems can actually solve real-world software engineering problems. Instead of synthetic coding challenges, it uses actual GitHub issues and pull requests from production codebases.

### The Core Concept

```
┌─────────────────────────────────────────────────────────────────────┐
│                    SWE-BENCH CONCEPT                                 │
├─────────────────────────────────────────────────────────────────────┤
│                                                                     │
│   INPUT:                                                            │
│   ├── Repository snapshot (at issue creation time)                  │
│   ├── Issue description (bug report or feature request)             │
│   └── Codebase context                                              │
│                              ↓                                      │
│   AI SYSTEM:                                                        │
│   ├── Understands the problem                                       │
│   ├── Locates relevant files                                        │
│   ├── Generates a patch                                             │
│   └── Outputs code changes                                          │
│                              ↓                                      │
│   EVALUATION:                                                       │
│   ├── Run repository's test suite                                   │
│   ├── Check if patch passes fail-to-pass tests                     │
│   └── Binary: Resolved or Not Resolved                              │
│                                                                     │
│   KEY: No partial credit - either the fix works or it doesn't      │
│                                                                     │
└─────────────────────────────────────────────────────────────────────┘
```

---

## Quick Stats

| Metric | Value |
|--------|-------|
| **GitHub Stars** | 3,100+ |
| **Total Tasks** | 2,294 |
| **Repositories** | 12 Python repos |
| **Task Source** | Real GitHub issues/PRs |
| **Created By** | Princeton NLP |
| **Published** | ICLR 2024 |

---

## The Repositories

SWE-bench draws from 12 popular, well-maintained Python repositories:

| Repository | Domain | Stars |
|------------|--------|-------|
| **django/django** | Web framework | 80k+ |
| **scikit-learn/scikit-learn** | Machine learning | 60k+ |
| **matplotlib/matplotlib** | Plotting | 20k+ |
| **pallets/flask** | Web microframework | 68k+ |
| **pytest-dev/pytest** | Testing | 12k+ |
| **astropy/astropy** | Astronomy | 4k+ |
| **sympy/sympy** | Symbolic math | 13k+ |
| **sphinx-doc/sphinx** | Documentation | 6k+ |
| **pylint-dev/pylint** | Linting | 5k+ |
| **psf/requests** | HTTP library | 52k+ |
| **pydata/xarray** | N-D arrays | 3k+ |
| **mwaskom/seaborn** | Statistical viz | 12k+ |

---

## Benchmark Variants

### SWE-bench Full

The complete benchmark with all 2,294 tasks.

| Aspect | Detail |
|--------|--------|
| **Size** | 2,294 tasks |
| **Difficulty** | Varies widely |
| **Use** | Comprehensive evaluation |

### SWE-bench Lite

A filtered subset for faster, more focused evaluation.

| Aspect | Detail |
|--------|--------|
| **Size** | 300 tasks |
| **Selection** | Curated for quality |
| **Use** | Quick benchmarking |

### SWE-bench Verified

Human-validated subset with confirmed solvability.

| Aspect | Detail |
|--------|--------|
| **Size** | 500 tasks |
| **Validation** | Human-verified |
| **Use** | High-confidence results |

### SWE-bench Pro

More challenging subset requiring deeper understanding.

| Aspect | Detail |
|--------|--------|
| **Focus** | Harder problems |
| **Use** | Differentiating top systems |

---

## How Evaluation Works

```
┌─────────────────────────────────────────────────────────────────────┐
│                    EVALUATION PIPELINE                               │
├─────────────────────────────────────────────────────────────────────┤
│                                                                     │
│   1. SETUP                                                          │
│      ├── Clone repository at specific commit                        │
│      ├── Install dependencies                                       │
│      └── Prepare test environment                                   │
│                              ↓                                      │
│   2. APPLY PATCH                                                    │
│      ├── Take AI-generated patch                                    │
│      └── Apply to codebase                                          │
│                              ↓                                      │
│   3. RUN TESTS                                                      │
│      ├── Execute "fail-to-pass" tests                              │
│      ├── Tests that should fail before fix                         │
│      └── Tests that should pass after fix                          │
│                              ↓                                      │
│   4. VERIFICATION                                                   │
│      ├── All fail-to-pass tests now pass? → RESOLVED               │
│      └── Any test still fails? → NOT RESOLVED                      │
│                                                                     │
│   METRIC: Percentage of tasks resolved                              │
│                                                                     │
└─────────────────────────────────────────────────────────────────────┘
```

---

## Current Leaderboard (Highlights)

Performance varies significantly by model and approach:

| System | SWE-bench Lite | SWE-bench Verified |
|--------|---------------|-------------------|
| **OpenHands** | ~45% | 77.6% |
| **SWE-agent (Live)** | -- | 77.4% |
| **Claude 3.5 Sonnet** | ~40% | -- |
| **GPT-4** | ~12% | -- |
| **Baseline (no agent)** | ~3-4% | -- |

*Note: Scores vary based on agent scaffolding, prompting, and setup.*

---

## Why SWE-bench Matters

### 1. Real-World Validity

Unlike synthetic benchmarks:
- Uses actual bugs from production code
- Requires understanding large codebases
- Tests practical software engineering skills

### 2. Objective Evaluation

```
┌─────────────────────────────────────────────────────────────────────┐
│                    WHY TESTS ARE THE JUDGE                          │
├─────────────────────────────────────────────────────────────────────┤
│                                                                     │
│   PROBLEMS WITH OTHER EVALUATION METHODS:                           │
│   ├── Human evaluation → Expensive, subjective, doesn't scale      │
│   ├── LLM-as-judge → Can be gamed, may miss edge cases             │
│   ├── Code similarity → Working code can look very different       │
│   └── Compile success → Doesn't mean it's correct                  │
│                                                                     │
│   WHY TESTS WORK:                                                   │
│   ├── Objective → Binary pass/fail, no interpretation              │
│   ├── Comprehensive → Tests cover edge cases                       │
│   ├── Fast → Can run thousands of evaluations                      │
│   └── Real → These are the actual tests developers wrote           │
│                                                                     │
└─────────────────────────────────────────────────────────────────────┘
```

### 3. Industry Adoption

Used by everyone measuring AI coding capabilities:
- **OpenAI** - GPT-4, o1 evaluations
- **Anthropic** - Claude evaluations
- **Meta** - Llama evaluations
- **Google** - Gemini evaluations
- **Every AI startup** - Standard benchmark

---

## Task Examples

### Bug Fix Task

```
Repository: django/django
Issue: QuerySet.values() raises error with expressions

Description: When using QuerySet.values() with an expression
that has a default alias, a KeyError is raised...

Expected: Generate patch that fixes the QuerySet.values()
behavior to handle expressions correctly.
```

### Feature Request Task

```
Repository: scikit-learn
Issue: Add warm_start parameter to GradientBoostingClassifier

Description: It would be useful to have warm_start in
GradientBoostingClassifier for incremental training...

Expected: Generate patch implementing warm_start functionality.
```

---

## Using SWE-bench

### Installation

```bash
pip install swebench
```

### Running Evaluation

```bash
# Evaluate predictions
python -m swebench.harness.run_evaluation \
    --predictions_path predictions.json \
    --swe_bench_tasks swebench/test.json \
    --log_dir logs
```

### Accessing Tasks

```python
from datasets import load_dataset

# Load SWE-bench Lite
dataset = load_dataset("princeton-nlp/SWE-bench_Lite")

# Each task contains:
# - instance_id: Unique identifier
# - repo: Repository name
# - base_commit: Starting point
# - problem_statement: Issue description
# - hints_text: Optional hints
# - test_patch: Tests that verify the fix
```

---

## Challenges for AI Systems

What makes SWE-bench hard:

| Challenge | Description |
|-----------|-------------|
| **Codebase Size** | Repositories have millions of lines of code |
| **Context Limits** | Can't fit entire codebase in context |
| **File Navigation** | Must find relevant files among thousands |
| **Test Understanding** | Need to understand what tests expect |
| **Code Style** | Must match existing codebase patterns |
| **Dependency Knowledge** | Understand framework-specific patterns |

---

## Research Behind SWE-bench

### Paper

**Title**: "SWE-bench: Can Language Models Resolve Real-World GitHub Issues?"

| Venue | ICLR 2024 |
|-------|-----------|
| **arXiv** | 2310.06770 |
| **Authors** | Carlos E. Jimenez, John Yang, Alexander Wettig, et al. |
| **Institution** | Princeton NLP |

### Key Findings

1. Raw LLMs perform poorly (~3-4%) without agent scaffolding
2. Agent design (interface, tools) dramatically improves performance
3. Retrieval and localization are key bottlenecks
4. Test execution enables objective, scalable evaluation

---

## How This Connects to Our Other Deep Dives

| Deep Dive | Connection |
|-----------|------------|
| **SWE-agent** | Agent designed specifically for SWE-bench |
| **OpenHands** | Top performer on SWE-bench |
| **lm-evaluation-harness** | Different type of benchmark (knowledge vs. coding) |
| **Promptfoo** | Testing prompts vs. testing code fixes |

---

## Key Links

| Resource | URL |
|----------|-----|
| GitHub | https://github.com/princeton-nlp/SWE-bench |
| Paper | https://arxiv.org/abs/2310.06770 |
| Leaderboard | https://www.swebench.com |
| Dataset | https://huggingface.co/datasets/princeton-nlp/SWE-bench |

---

## Bottom Line

SWE-bench is the **gold standard for evaluating AI coding agents**.

**Why it matters**:
- Uses real bugs from real codebases
- Objective evaluation via test execution
- Industry-wide adoption
- Drives agent development (SWE-agent, OpenHands)

**The key insight**: The gap between "AI can write code" and "AI can fix real bugs in production codebases" is enormous. SWE-bench measures the latter, which is what actually matters for practical software engineering.

For our repository, SWE-bench represents the ultimate goal - AI that can understand large codebases, locate problems, and fix them correctly. Tools like SWE-agent and OpenHands were built specifically to tackle this benchmark.

---

*Document created: January 2026*
*Sources: [GitHub - SWE-bench](https://github.com/princeton-nlp/SWE-bench), [arXiv 2310.06770](https://arxiv.org/abs/2310.06770), [swebench.com](https://www.swebench.com)*
