# Research, Validation & Vision

> This document captures the full research journey behind GitScope — what we tested, what failed, what survived, and why this tool exists. It serves as the permanent record of our findings so the vision stays grounded in evidence, not hype.

## The Problem We Validated

**AI coding agents waste tokens and miss structural context when reviewing raw diffs.**

Modern LLMs can read unified diff format — that's not the problem anymore. The problem is that a diff shows *what changed* but not *what it breaks*. When a function signature changes, the diff doesn't show you the 5 callers in other files that now pass the wrong arguments. When a default value changes, the diff doesn't show you the 12 call sites relying on that default.

Human reviewers catch this because they know the codebase. AI agents don't — they see exactly what's in the diff and nothing more (unless they go exploring, which costs tokens and time).

## What We Tested

### Three Iterations, Increasingly Focused

**v1 — Full Semantic Diff Analysis**
GumTree-based AST diffing. Too slow, output too generic. Killed after weeks of building.

**v2 — Compact CLI with Tiered Output**
Tree-sitter symbol extraction, cross-file matching, multiple output formats. A/B tested against 10 real PRs from Flask, FastAPI, Pydantic, httpx. Results: Claude produced equally good reviews with just raw `git diff` for most changes. Killed in 1 day.

**v3 (Selective Trigger) — Only Speak When It Matters**
Same engine, but with a critical filter: **stay silent unless the change is structurally significant.** Only trigger on signature changes, removed symbols, default value changes, and file moves with external callers.

### A/B Test Results (12 Real PRs)

Tested against real commits from production open-source projects:

| Metric | Result |
|--------|--------|
| **Precision** | 100% (5/5) — when it spoke, it was right |
| **Silence rate** | 58% (7/12) — stayed quiet on noise |
| **False positives** | 0 — never cried wolf |

The selective trigger turned a 20% hit rate (v2) into 100% precision by simply not speaking when it had nothing useful to say.

### What Triggers a Finding

GitScope only reports when it detects:

- **Signature changes** — parameters added/removed/reordered, return type changes
- **Default value changes** — existing callers may behave differently without code changes
- **Removed symbols** — functions/classes deleted that other code references
- **File moves with external callers** — imports will break

Each finding includes the specific callers affected, with file paths and line numbers.

### What Stays Silent

- New functions/classes (additions don't break anything)
- Body-only changes (internal refactors)
- Formatting/whitespace changes
- Changes where all callers are already in the diff
- Test-only changes

## Why This Matters for AI Agents

### The Zero-Friction Value Proposition

```bash
pip install gitscope
gitscope review main..feature --format json
```

That's it. No signup. No API keys. No SaaS dashboard. No configuration files. No CI pipeline changes. One install, one command, immediate value.

- **Exit code 0** → nothing noteworthy, proceed normally
- **Exit code 1** → findings in stdout, feed to your agent
- **Exit code 2** → error

Any AI agent that can run a shell command can use GitScope. Add one line to your `CLAUDE.md` or `.cursorrules` and every review gets structural context for free.

### What Agents Get That They Don't Have Today

Without GitScope, an AI reviewing a PR sees:
```
-def redirect(location, code=302, Response=None):
+def redirect(location, code=303, Response=None):
```

With GitScope, the agent knows:
```
DEFAULT VALUE CHANGED: redirect(location, code=302) → redirect(location, code=303)
Impact: 5 callers rely on the default:
  auth.py:25  `return redirect(url_for("auth.login"))`
  auth.py:77  `return redirect(url_for("index"))`
  blog.py:81  `return redirect(url_for("blog.index"))`
Review: Verify callers expect HTTP 303 instead of 302
```

This is the difference between "I see a number changed" and "I see a behavioral change that affects 5 call sites across 3 files." The agent can now give a targeted, actionable review instead of a generic "looks fine."

### Compared to Alternatives

| | GitScope | CodeRabbit | Aider repo-map |
|---|---|---|---|
| Setup time | `pip install` (30 seconds) | Account signup + GitHub app install + config | Comes with Aider (locked in) |
| Cost | Free, forever | $15-30/seat/month | Free (but only inside Aider) |
| Privacy | Code never leaves your machine | Code processed on their servers | Local |
| Works with any agent | Yes — CLI + JSON | No — GitHub PR comments only | No — Aider only |
| Selective output | Silent when nothing matters | Comments on every PR | N/A (context, not review) |
| Open source | Apache 2.0 | Proprietary | Apache 2.0 (Aider-locked) |

## How It Works (Technical)

1. **Parse the diff** — tree-sitter builds ASTs for before/after versions of each changed file
2. **Extract symbols** — functions, classes, methods with full signatures
3. **Detect structural changes** — compare symbol signatures, find removals, default value changes
4. **Scan for callers** — `git grep` + tree-sitter to find references to changed symbols across the repo
5. **Apply selective trigger** — only report if the change is high-signal AND has external callers
6. **Output** — structured JSON or human-readable text with file paths, line numbers, caller context

Languages supported: Python, TypeScript/JavaScript, Go. Built on tree-sitter, so adding languages is mechanical (grammar + query patterns).

## Research That Informed the Design

### Key Insight: Silence Is the Feature

The v2 A/B test showed that 80% of PRs don't benefit from structural analysis. The natural instinct is to force the tool to say *something* every time. We did the opposite — we made silence the default. This single decision turned a marginally useful tool into a precision instrument.

### Key Insight: Lazy > Eager

The director's insight that changed our architecture: pre-computing all possible context eagerly is wasteful. Instead, GitScope acts as a **selective trigger** — it does cheap local analysis to determine IF expensive review attention is needed, and WHERE to focus it.

### Key Insight: Agent-Native Means CLI-Native

Research into how agents discover and use tools showed three integration paths:
1. **Clean CLI** with JSON output and meaningful exit codes
2. **SKILL.md / CLAUDE.md** snippets that teach agents when and how to call the tool
3. **llms.txt** on the docs site for web-based agent discovery

All three are zero-config from the user's perspective. Install the tool, and agents can find it.

### What We Learned from CodeRabbit

CodeRabbit's architecture (persistent dependency graph, model cascade, incremental analysis) represents the infrastructure approach. GitScope takes the opposite approach: stateless, lightweight, local. This is intentional — we optimize for the common case (agent with filesystem access) rather than the enterprise case (CI/CD pipeline integration).

The tradeoff: CodeRabbit handles massive repos better (persistent graph vs. on-the-fly scanning). GitScope handles privacy, cost, and simplicity better. For most teams, the latter matters more.

## The Vision

**GitScope is the open-source code intelligence layer for AI agents.**

Not a SaaS product. Not a platform. A sharp, focused utility that gives any AI agent structural understanding of code changes — for free, locally, with zero setup.

The competitive advantage isn't a feature list. It's the combination of:
- **Zero friction** — `pip install` and one command
- **Zero cost** — free forever, Apache 2.0
- **Zero trust required** — code never leaves your machine
- **Precision over noise** — silent when it has nothing useful to say

GitScope is designed from the ground up for the agent era — local, stateless, and CLI-native.

## Validated Roadmap

Validated against real practitioner workflows (steipete's CLI-first methodology, Thomas Dohmke's developer evolution research) and grounded in our A/B test results.

### Design Principles (Permanent)

These don't change. Every feature decision runs through this filter:

1. **CLI-first** — CLIs beat MCPs. Agents are better at calling CLIs. No MCP server, no IDE plugin, no web dashboard. The CLI IS the interface.
2. **One line in CLAUDE.md** — Integration story: `reviews: gitscope review`. That's it. No SDK, no config files, no signup.
3. **Context is precious** — Concise, structured output. Small, actionable findings. Never a wall of text.
4. **Less is more** — Don't build what isn't needed. Silence is a feature. Absence of noise IS the product.
5. **The review brain is the product** — The CLI is just `main()`. What matters is the intelligence: review quality, pattern detection, context-aware analysis.

### Who GitScope Serves

Teams are at very different stages of AI adoption. GitScope provides value to ALL of them:

| Stage | Who | How They Use GitScope |
|---|---|---|
| **Manual devs** (biggest market) | VS Code, reviews by hand | `gitscope review` before opening a PR — catches structural issues they'd miss |
| **Single-agent users** | Claude Code or Cursor, one at a time | Quality gate on AI-generated code. One line in agent config. |
| **Multi-agent orchestrators** | 4+ agent instances in parallel | Automated reviewer across all instances — humans can't keep up at this scale |
| **Enterprise teams** | CI/CD pipelines, compliance requirements | Pipeline gate + audit trail, JSON output for dashboards |

Value increases up the adoption curve. Entry point is the manual dev. That's the correct funnel.

### Phase 1: Ship the CLI (Now → 4 weeks)
**Target: Manual devs + Single-agent users**

- `pip install gitscope` — zero-config, immediate value
- `gitscope review` — reviews staged/committed changes, outputs findings
- `--format json` for programmatic consumption
- Exit codes: 0=silence, 1=findings, 2=error
- Integration snippets for CLAUDE.md, .cursorrules, generic agent configs
- Launch: PyPI, one HN post, one Reddit post

**Success metric:** 50 installs, 5 users giving feedback within 30 days.

### Phase 2: CI/CD + Enterprise (Weeks 5-10)
**Target: Enterprise teams**

- GitHub Action (thin wrapper around the CLI)
- `--ci` mode with PR comment posting
- `--fail-on <severity>` for pipeline gates
- Team config via `.gitscope.yml`
- `--strict` / `--lenient` modes

**Gate:** Only enter if Phase 1 shows real traction (>100 weekly active users).

### Phase 3: Agent-Native Patterns (Weeks 11-16)
**Target: Multi-agent orchestrators**

- `gitscope watch` — daemon mode, reviews changes as they land
- `gitscope review --context <file>` — accepts context hints from agents
- Incremental review (only re-review what changed since last review)
- Informed by Phase 1-2 user feedback

### What We Are NOT Building
- ~~MCP server~~ — CLIs beat MCPs (validated by practitioners)
- ~~IDE plugin~~ — terminals exist in every IDE
- ~~Web dashboard~~ — stdout and JSON are the dashboard
- ~~SaaS service~~ — local-first is the wedge

### Kill/Continue Signals

| Timeframe | Continue | Reduce Investment |
|-----------|----------|-------------------|
| Month 1 | >50 installs, any organic mentions | <10 installs despite promotion |
| Month 3 | >100 WAU, GitHub stars >200 | <50 WAU, no organic growth |
| Month 6 | Contributors appearing, agent frameworks integrating | Growth plateau, no ecosystem adoption |

### Long-Term Competitive Position

GitScope competes with CodeRabbit not by being a cheaper CodeRabbit, but by making CodeRabbit's model obsolete:

- CodeRabbit is a **SaaS review product for humans** (dashboard, comments, per-seat pricing)
- GitScope is a **local intelligence primitive for agents** (CLI, JSON, free)

As AI agents become the primary code reviewers — a transition actively underway — GitScope is positioned correctly. CodeRabbit has to retrofit agent-native behavior onto a SaaS platform. GitScope is born there.
