# SWE-bench Lite: Harness vs Claude Code Comparison

## Setup
- **Model**: Claude Sonnet 4.6 (both agents)
- **Tasks**: 25 (from SWE-bench Lite, first 25 in dataset order)
- **Repos**: astropy (6 tasks), django (19 tasks)
- **Timeout**: 600s per task
- **Harness**: uses Anthropic API with `uv run harness`
- **Claude Code**: uses Claude subscription with `claude --print`

## Overall Statistics

| Metric | Harness | Claude Code |
|--------|---------|-------------|
| Tasks with patches | 24/25 | 24/25 |
| Empty patches | 1 | 1 |
| Median patch size | 58 lines | 44 lines |
| Mean patch size | 81 lines | 50 lines |
| Min/Max patch size | 28/232 | 12/128 |
| Median files changed | 2 | 2 |
| Total additions | 1022 | 535 |
| Total deletions | 106 | 67 |

## Patch Size Distribution

| Size Bucket | Harness | Claude Code |
|-------------|---------|-------------|
| Empty | 1 | 1 |
| Small (1-30) | 1 | 9 |
| Medium (31-60) | 11 | 7 |
| Large (61-100) | 6 | 6 |
| XL (100+) | 6 | 2 |

## Per-Task Comparison

| Instance ID | H lines | CC lines | H files | CC files | H +/- | CC +/- | File overlap |
|-------------|---------|----------|---------|----------|-------|--------|--------------|
| astropy__astropy-12907 | 48 | 44 | 3 | 2 | +18/-2 | +21/-1 | 2/3 |
| astropy__astropy-14182 | 110 | 64 | 3 | 3 | +66/-3 | +27/-3 | 3/3 |
| astropy__astropy-14365 | 31 | 13 | 1 | 1 | +4/-2 | +1/-1 | 1/1 |
| astropy__astropy-14995 | 52 | 13 | 2 | 1 | +32/-1 | +2/-0 | 1/2 |
| astropy__astropy-6938 | 54 | 43 | 2 | 2 | +31/-1 | +20/-1 | 2/2 |
| astropy__astropy-7746 | 90 | 27 | 2 | 1 | +64/-0 | +9/-0 | 1/2 |
| django__django-10914 | 44 | 56 | 3 | 3 | +3/-4 | +6/-8 | 2/4 |
| django__django-10924 | 92 | 67 | 4 | 3 | +32/-4 | +38/-1 | 2/5 |
| django__django-11001 | 62 | 41 | 2 | 2 | +35/-1 | +18/-1 | 1/3 |
| django__django-11019 | 188 | 128 | 2 | 2 | +102/-42 | +89/-7 | 2/2 |
| django__django-11039 | 42 | 42 | 2 | 2 | +18/-2 | +18/-2 | 2/2 |
| django__django-11049 | 39 | 26 | 3 | 2 | +3/-3 | +2/-2 | 2/3 |
| django__django-11099 | 44 | 22 | 2 | 1 | +4/-4 | +2/-2 | 1/2 |
| django__django-11133 | 42 | 13 | 2 | 1 | +12/-1 | +2/-0 | 1/2 |
| django__django-11179 | 31 | 12 | 2 | 1 | +12/-0 | +1/-0 | 1/2 |
| django__django-11283 | 156 | 90 | 2 | 2 | +131/-0 | +68/-0 | 2/2 |
| django__django-11422 | 50 | 48 | 2 | 2 | +25/-2 | +23/-2 | 2/2 |
| django__django-11564 | 232 | 122 | 5 | 4 | +158/-2 | +65/-2 | 4/5 |
| django__django-11583 | 28 | 13 | 1 | 1 | +10/-0 | +1/-1 | 1/1 |
| django__django-11620 | 69 | 56 | 3 | 2 | +13/-2 | +18/-2 | 2/3 |
| django__django-11630 | 147 | 79 | 2 | 2 | +99/-11 | +32/-10 | 2/2 |
| django__django-11742 | 146 | 77 | 3 | 2 | +106/-0 | +53/-2 | 2/3 |
| django__django-11797 | 0 | 0 | 0 | 0 | +0/-0 | +0/-0 | n/a |
| django__django-11815 | 76 | 76 | 2 | 2 | +14/-15 | +14/-15 | 2/2 |
| django__django-11848 | 62 | 20 | 2 | 1 | +30/-4 | +5/-4 | 1/2 |

## Key Findings

### 1. Checkpoint Bloat (Fixed)
Harness agent creates `.harness/checkpoints/` files as part of its workflow.
These were initially included in the git diff, inflating raw patches by ~92%.
After stripping checkpoints, Harness patch sizes are comparable to Claude Code.
The `get_diff()` function now excludes `.harness/` from diffs.

### 2. Patch Size Comparison (After Cleanup)
- Harness median: 58 lines vs Claude Code median: 44 lines
- Harness patches are ~1.3x larger on average
- Harness tends to write more comprehensive test cases
- Claude Code makes more surgical, minimal changes

### 3. Files Modified
- Harness median: 2 files, Claude Code median: 2 files
- Both agents typically touch source file + test file
- File overlap between agents averages ~50%

### 4. Empty Patches
- Both agents failed on `django__django-11797` (produced 0 lines)
- This suggests the task may be particularly difficult or have setup issues

### 5. Grading Required
Patch size alone does not determine correctness. Smaller patches can be wrong
(missing edge cases) and larger patches can be correct (thorough fix + tests).
Docker-based grading via `swebench.harness.run_evaluation` is needed for pass/fail results.

## Next Steps

1. Run official SWE-bench grading (requires Docker)
2. Scale to 100 or full 300 tasks for statistically significant results
3. Add `.harness/` to `.gitignore` in the Harness agent workflow to prevent checkpoint bloat
