# Validation

## A/B test: Does GitScope actually help?

We ran a controlled test to answer one question: does a reviewer agent produce better reviews with GitScope's structured output than with the raw diff alone?

### Setup

- **Commit:** 7ae6492 — 944 lines changed across 3 files
- **Baseline:** AI reviewer with raw diff only (974 lines of diff text)
- **Assisted:** AI reviewer with GitScope v2 JSON output + the same raw diff

### Results

| Metric | Baseline | Assisted |
|--------|----------|----------|
| Total findings | 19 | **26 (+37%)** |
| Design-level issues | 4 | **6** |
| Specific missing test coverage | 1 (generic) | **6 (named functions)** |
| Bugs found | 5 | 5 |

### What the assisted reviewer caught that the baseline missed

1. **`_is_high_impact` dual-set case matching bug** — a real logic error where `"If_Statement"` silently fails. The standout finding.
2. **`_get_ref_content` can't distinguish "file missing" from "git error"** — error handling gap
3. **`reshape_engine_output` mutates input dict** — callers don't expect this
4. **Specific untested functions** — named 6 functions with no test coverage, vs baseline's generic "no CLI tests"

### What the baseline caught that the assisted missed

1. Type annotation `str` with default `None` (should be `str | None`)
2. `ref~1` doesn't exist for initial commit
3. `_risk_level_value` returns -1 for unknown

The baseline's unique findings were surface-level. The assisted reviewer's unique findings were structural.

## Why it works

The structured output gives the reviewer a **map** of all 20 functions before it reads the diff. Instead of scanning 974 lines linearly, it knows the shape of the change and allocates attention to the important parts.

This is the core value: **structural maps shift AI reviewers from line-level scanning to design-level analysis.**

## Honest assessment

GitScope's value scales with PR size:

| PR size | Value added |
|---------|------------|
| Small (<100 lines, 1-2 files) | **5-10%.** The reviewer can read the whole diff easily. GitScope adds marginal context. |
| Medium (200-500 lines) | **15-20%.** Structural overview saves the reviewer from getting lost. |
| Large (500+ lines, multiple files) | **20-30%.** The map is essential. Linear reading of 1000+ lines of diff misses structural patterns. |

GitScope is not magic. On small, focused PRs, you don't need it. On large, multi-file changes — the kind where reviewers most often miss design issues — it earns its keep.

## Philosophy: accuracy over comprehensiveness

GitScope reports structural facts: what functions changed, what signatures broke, what moved. It does **not** assess risk, rate severity, or guess intent. Those are opinions, and opinions can be wrong.

A tool that says "this change is high risk" and is wrong erodes trust fast. A tool that says "authenticate() was removed and authenticateOAuth() was added with a different signature" is always right — and the reviewer can draw their own conclusions.

This is deliberate. Wrong summaries are worse than no summaries.
