# Executive Summary: Measuring CLAUDE.md Impact — The Complete Playbook

**Date:** 2026-02-18  
**Purpose:** Reliable methods to prove CLAUDE.md drives measurable improvement  
**Key Insight:** Simple before/after comparisons are misleading — you need causal inference

---

## The Measurement Problem (Why This Is Hard)

### You Can't Just Compare Before/After

**Bad Approach:**
```
Before CLAUDE.md: 61% success rate
After CLAUDE.md:  74% success rate
Conclusion: +13 percentage points improvement ❌ WRONG
```

**Why this is wrong:**
- Better developers might adopt CLAUDE.md first (selection bias)
- Teams improve over time regardless (maturation)
- Tasks might be easier in "after" period (task complexity bias)
- Developers work harder when observed (Hawthorne effect)

### The Research-Backed Solution

From **METR study** (July 2025), **Google**, **Airbnb**, **Netflix**:

**Gold Standard:** Randomized Controlled Trial (RCT)  
**Alternative:** Difference-in-Differences with control group  
**Single Repo:** Time-series with counterfactual (CausalImpact)

---

## Three Measurement Methods (Pick One)

### Method 1: RCT (Gold Standard) — When You Need Bulletproof Proof

**How it works:**
1. Randomly assign 20 repos → Treatment (CLAUDE.md) or Control (no CLAUDE.md)
2. Run for 30 days
3. Compare outcomes statistically

**Code:**
```python
rct = RCTFramework(repos)
rct.randomize()
rct.start_experiment(duration_days=30)
results = rct.measure_outcomes()
# Results: Treatment effect +13.1%, p=0.003, statistically significant ✅
```

**When to use:**
- Initial validation (Phase 1)
- Leadership needs proof before scaling
- You have 20+ similar repos

**Pros:** Eliminates all bias through randomization  
**Cons:** Requires withholding CLAUDE.md from some repos

---

### Method 2: Difference-in-Differences — When RCT Isn't Politically Possible

**How it works:**
1. Compare repos that adopted CLAUDE.md vs those that didn't
2. Measure BEFORE adoption (baseline)
3. Measure AFTER adoption
4. Calculate: (Treatment Change) - (Control Change)

**Visual:**
```
Success Rate
    │
75% ┤        ● Treatment (with CLAUDE.md)
    │       ╱
70% ┤      ╱
    │     ╱  ← Treatment Effect
65% ┤────●───○ Control (no CLAUDE.md)
    │   ╱
60% ┤  ╱
    │_╱________________________________
      Before          After
```

**When to use:**
- Repos already adopted organically
- Can't withhold treatment
- Need retrospective analysis

**Pros:** Uses natural adoption, politically easier  
**Cons:** Requires good control group matching

---

### Method 3: CausalImpact (Google's Method) — Single Repo Pilot

**How it works:**
1. Use pre-intervention data to predict expected performance
2. Compare actual vs predicted (counterfactual)
3. Difference = treatment effect

**Output:**
```
Actual performance:          74%
Predicted (no intervention): 61%
Treatment effect:            +13% (99.9% probability)
```

**When to use:**
- Pilot with single repo
- No comparable control repos
- Need quick directional evidence

**Pros:** Works with single repo, Bayesian confidence intervals  
**Cons:** Assumes stable pre-intervention patterns

---

## Primary Metrics (What Actually Matters)

### Tier 1: Session Success Rate (Your North Star)

**Definition:** % of AI sessions that complete successfully  
**Success = has edits + low trouble score + not abandoned

**Target:** +10 percentage points improvement  
**Example:** 61% → 74%

**Why this matters:**
- Directly measures outcome quality
- Easy to understand
- Comparable across repos

---

### Tier 2: Trouble Score Distribution

**Definition:** Your existing composite metric  
**Formula:** error_cascades×3 + interrupts×2 + undo_requests×3

**Metrics:**
- Mean trouble score (should decrease)
- % sessions with score >10 (should decrease)
- 90th percentile (should decrease)

**Target:** -20% reduction in mean trouble score

---

### Tier 3: Pattern Adoption (Leading Indicator)

**Definition:** Are developers using recommended patterns?

**Track:**
- ✅ Error-paste-fix usage (should increase)
- ✅ Single-task prompts (should increase)
- ❌ Multi-task prompts (should decrease)
- ❌ Vague directives (should decrease)

**Target:** +15pp good patterns, -10pp bad patterns

---

## Secondary Metrics (For Managers & Leadership)

### Metric 1: Time to First Success

**What:** Minutes from session start to first successful edit  
**Before CLAUDE.md:** 18 minutes  
**After CLAUDE.md:** 12 minutes  
**Time saved:** 6 minutes per session

**ROI Calculation:**
```
50 sessions/week × 6 min = 5 hours/week saved
Annual value: $25,000 (at $100/hr)
```

---

### Metric 2: Abandonment Rate

**What:** % of sessions abandoned (no edits, >30 min)  
**Before:** 34%  
**After:** 21%  
**Improvement:** -13 percentage points

**Manager presentation:**
```
Fraud Detection Service:
- 145 sessions/month
- Before: 49 abandoned = 24.5 hours wasted
- After: 30 abandoned = 15 hours wasted
- Saved: 9.5 hours/month
```

---

### Metric 3: Developer Satisfaction (NPS)

**Survey:** "How satisfied with AI sessions?" (1-10)  
**Target:** NPS >50, >70% satisfaction

**Why this matters:** Adoption depends on satisfaction

---

### Metric 4: Dollar ROI

**Calculation:**
```python
Time saved per session: 6 minutes
Sessions per month: 145
Abandonment reduction: 13%

Total time saved: 32 hours/month
Value: $3,200/month
Implementation cost: $400

Monthly ROI: 700%
Annual projection: $38,400
```

---

## The 6-Week Measurement Playbook

### Week 1: Baseline
- [ ] Select 10-20 repos
- [ ] Collect 30 days pre-intervention data
- [ ] Document current metrics

### Week 2: Deploy
- [ ] Generate CLAUDE.md for treatment repos
- [ ] Deploy using RCT, DiD, or natural adoption
- [ ] Track which repos get it

### Weeks 3-5: Run
- [ ] Collect data during experiment
- [ ] Monitor adoption
- [ ] Gather developer feedback

### Week 6: Analyze
- [ ] Collect post-intervention data
- [ ] Run statistical analysis (p-values, CI)
- [ ] Calculate ROI
- [ ] Generate reports

### Week 7: Report
- [ ] Executive summary (1 slide)
- [ ] Manager dashboard (1 page)
- [ ] Developer insights (CLI)
- [ ] Decide: Scale or iterate

---

## Normalization: Handling Different Task Complexity

### Problem: Not All Tasks Are Equal

Comparing a bug fix to a complex architecture task is unfair.

### Solution: Stratified Analysis

```python
# Analyze by task complexity separately
results = {
    'simple_tasks': {
        'success_rate': 85%,
        'improvement': +8pp
    },
    'medium_tasks': {
        'success_rate': 71%,
        'improvement': +14pp
    },
    'complex_tasks': {
        'success_rate': 58%,
        'improvement': +11pp
    }
}
```

### Alternative: Normalized Success Score

```python
# If task has 40% baseline success and we succeed → +60 points
# If task has 80% baseline success and we succeed → +20 points
normalized_score = (actual_success - task_baseline) / (1 - task_baseline)
```

---

## Sample Reports

### For Executives (1 Slide)

```
┌─────────────────────────────────────────────┐
│  CLAUDE.md Impact — 6 Week RCT              │
│                                             │
│  Treatment Effect: +13.1 percentage points  │
│  Statistical Significance: p < 0.01 ✅      │
│  Confidence: 95%                            │
│                                             │
│  ROI: $38,400/year per repo                 │
│  Adoption: 60% of target repos              │
│  Developer Satisfaction: 82%                │
│                                             │
│  ✅ Recommendation: Scale to all repos      │
└─────────────────────────────────────────────┘
```

---

### For Engineering Managers (Dashboard)

```
AI Collaboration Insights — Fraud Detection Service

Metric           Before    After     Change
─────────────────────────────────────────────
Success Rate     61%       74%       +13% ✅
Avg Duration     18 min    12 min    -6 min ✅
Abandonment      34%       21%       -13% ✅
Trouble Score    8.4       6.2       -2.2 ✅
─────────────────────────────────────────────

Time Saved This Month: 32 hours
Value: $3,200

Top Improvements:
✓ Error-paste-fix usage: +25%
✓ Multi-task prompts: -15%
```

---

### For Developers (CLI)

```bash
$ ai-report --repo fraud-detection-service

📊 AI Session Report — Last 30 Days

Your Success Rate: 74% (↑ from 61%)
Team Average: 71%
You're doing better than 65% of the team! 🎉

💡 Insights:
• Error-paste-fix prompts working great (95% success)
• Try breaking multi-part tasks into separate prompts
• Avg session time dropped from 18min to 12min

Time saved: ~3 hours this month
Keep it up!
```

---

## Key Decisions

### Which Measurement Method?

| Situation | Recommended Method | Why |
|-----------|-------------------|-----|
| Need bulletproof proof | RCT | Gold standard, eliminates all bias |
| Can't withhold treatment | Diff-in-Diffs | Uses natural adoption |
| Single repo pilot | CausalImpact | No control group needed |
| Quick directional check | Before/after with caveats | Fast but less rigorous |

### Which Metrics to Track?

**Always track:**
1. Session success rate (primary KPI)
2. Trouble score distribution
3. Pattern adoption (leading indicator)

**Also report:**
4. Time to first success (manager-friendly)
5. Abandonment rate (shows waste reduction)
6. Developer satisfaction (predicts adoption)
7. Dollar ROI (justifies investment)

---

## The Bottom Line

### You CAN Prove CLAUDE.md Works

The research is clear:
- **RCTs are the gold standard** (METR study used this)
- **Causal inference is essential** (correlation ≠ causation)
- **Statistical rigor matters** (p-values, confidence intervals)
- **Multiple methods work** (choose based on your constraints)

### Without Proper Measurement

❌ You can't prove ROI  
❌ Leadership won't fund scale  
❌ You won't know what's working  
❌ Developers won't trust it

### With Proper Measurement

✅ Bulletproof evidence  
✅ Clear ROI calculation  
✅ Statistical confidence  
✅ Foundation for scale

---

## Your Action Plan

### This Week
1. **Choose measurement method** (recommend: RCT if 20+ repos, CausalImpact if pilot)
2. **Select repos** for experiment
3. **Collect baseline data** (30 days pre-intervention)

### Next Week
4. **Deploy CLAUDE.md** to treatment group
5. **Set up tracking** (daily metrics collection)
6. **Communicate** to developers (explain the experiment)

### Month 1
7. **Run experiment** (30 days)
8. **Collect feedback** (weekly developer surveys)
9. **Monitor adoption** (who's using it)

### Month 2
10. **Analyze results** (statistical analysis)
11. **Calculate ROI** (dollar value)
12. **Generate reports** (exec/manager/dev)
13. **Present findings** (leadership review)
14. **Decide:** Scale or iterate

---

## Resources

**Documents created:**
1. `MEASUREMENT-FRAMEWORK.md` — Complete technical guide with code
2. `CLAUDE-MD-PLAYBOOK.md` — Implementation guide
3. `FIRST-PRINCIPLES-STRATEGY.md` — Research synthesis
4. `EXECUTIVE-SUMMARY.md` — Overview and strategy
5. `This file` — Measurement focus

**Key research sources:**
- METR RCT Study (July 2025)
- Google CausalImpact
- Airbnb ACE Framework
- Netflix Causal Inference
- DORA Metrics

---

## Final Thought

**"In God we trust, all others bring data."**

You have the insights. You have the infrastructure. Now you have the measurement methodology.

**Start measuring today.** Prove it works. Scale with confidence.

The difference between a failed experiment and a company-wide success story is rigorous measurement.

Make the measurement investment. It's worth it.
