# The Measurement Framework: Proving CLAUDE.md Impact with Statistical Rigor

**Purpose:** Reliable, quantitative methods to measure CLAUDE.md effectiveness  
**Critical Question:** How do we know the improvement was caused by CLAUDE.md?  
**Status:** Research-backed measurement methodology

---

## Part 1: The Causality Problem

### The Core Challenge

You observe: **"Repos with CLAUDE.md have 74% success rate vs 61% without"**

But you can't conclude: **"CLAUDE.md causes +13 percentage point improvement"**

**Why? Correlation ≠ Causation**

Possible alternative explanations:
1. **Selection bias:** Better developers/repos adopt CLAUDE.md first
2. **Maturation:** Developers improve over time regardless
3. **Task complexity:** Repos with CLAUDE.md have easier tasks
4. **Hawthorne effect:** Developers perform better when observed
5. **Seasonality:** Different measurement periods

### The Solution: Causal Inference Methods

From the research (METR, Google, Airbnb, Netflix):

**Gold Standard:** Randomized Controlled Trial (RCT)  
**Good:** Difference-in-Differences with control group  
**Acceptable:** Time-series with counterfactual (CausalImpact)

---

## Part 2: Measurement Method 1 — Randomized Controlled Trial (RCT)

### The Gold Standard (Used by METR Study)

**How it works:**
1. Randomly assign repos to treatment or control group
2. Treatment group: Get CLAUDE.md
3. Control group: No CLAUDE.md (business as usual)
4. Measure both groups for same time period
5. Compare outcomes

### Implementation

```python
# rct_framework.py
import random
from datetime import datetime, timedelta
from typing import List, Dict

class RCTFramework:
    def __init__(self, repos: List[str], treatment_ratio: float = 0.5):
        self.all_repos = repos
        self.treatment_repos = []
        self.control_repos = []
        self.start_date = None
        self.end_date = None
        
    def randomize(self, seed: int = 42):
        """Randomly assign repos to treatment or control."""
        random.seed(seed)
        shuffled = self.all_repos.copy()
        random.shuffle(shuffled)
        
        n_treatment = int(len(shuffled) * 0.5)
        self.treatment_repos = shuffled[:n_treatment]
        self.control_repos = shuffled[n_treatment:]
        
        return {
            'treatment': self.treatment_repos,
            'control': self.control_repos
        }
    
    def start_experiment(self, duration_days: int = 30):
        """Begin the experiment."""
        self.start_date = datetime.now()
        self.end_date = self.start_date + timedelta(days=duration_days)
        
        # Deploy CLAUDE.md to treatment group only
        for repo in self.treatment_repos:
            deploy_claude_md(repo)
        
        print(f"Experiment started: {self.start_date}")
        print(f"Treatment group: {len(self.treatment_repos)} repos")
        print(f"Control group: {len(self.control_repos)} repos")
        print(f"End date: {self.end_date}")
    
    def measure_outcomes(self) -> Dict:
        """Measure results after experiment period."""
        treatment_metrics = self._measure_group(self.treatment_repos)
        control_metrics = self._measure_group(self.control_repos)
        
        # Calculate difference (treatment effect)
        treatment_effect = {
            'success_rate_diff': (
                treatment_metrics['success_rate'] - 
                control_metrics['success_rate']
            ),
            'trouble_score_diff': (
                treatment_metrics['avg_trouble_score'] - 
                control_metrics['avg_trouble_score']
            ),
            'session_duration_diff': (
                treatment_metrics['avg_session_duration'] - 
                control_metrics['avg_session_duration']
            )
        }
        
        # Statistical significance test
        p_value = self._calculate_significance(
            treatment_metrics, 
            control_metrics
        )
        
        return {
            'treatment': treatment_metrics,
            'control': control_metrics,
            'treatment_effect': treatment_effect,
            'p_value': p_value,
            'statistically_significant': p_value < 0.05
        }
    
    def _measure_group(self, repos: List[str]) -> Dict:
        """Calculate metrics for a group of repos."""
        sessions = get_sessions_for_repos(repos, self.start_date, self.end_date)
        
        return {
            'success_rate': calculate_success_rate(sessions),
            'avg_trouble_score': calculate_avg_trouble_score(sessions),
            'avg_session_duration': calculate_avg_duration(sessions),
            'total_sessions': len(sessions)
        }
    
    def _calculate_significance(self, treatment: Dict, control: Dict) -> float:
        """Calculate p-value using t-test."""
        from scipy import stats
        
        # Get session-level data
        treatment_sessions = get_raw_sessions(self.treatment_repos)
        control_sessions = get_raw_sessions(self.control_repos)
        
        # Two-sample t-test
        t_stat, p_value = stats.ttest_ind(
            [s.success for s in treatment_sessions],
            [s.success for s in control_sessions]
        )
        
        return p_value

# Usage
repos = get_all_repos()  # e.g., 20 repos
rct = RCTFramework(repos, treatment_ratio=0.5)

# Step 1: Randomize
assignment = rct.randomize(seed=42)
print(f"Treatment: {len(assignment['treatment'])} repos")
print(f"Control: {len(assignment['control'])} repos")

# Step 2: Start experiment
rct.start_experiment(duration_days=30)

# Step 3: After 30 days, measure
results = rct.measure_outcomes()

print(f"\n=== RCT Results ===")
print(f"Treatment success rate: {results['treatment']['success_rate']:.1%}")
print(f"Control success rate: {results['control']['success_rate']:.1%}")
print(f"Treatment effect: +{results['treatment_effect']['success_rate_diff']:.1%}")
print(f"P-value: {results['p_value']:.4f}")
print(f"Statistically significant: {results['statistically_significant']}")
```

### Sample Output

```
=== RCT Results ===
Treatment success rate: 74.3%
Control success rate: 61.2%
Treatment effect: +13.1%
P-value: 0.0032
Statistically significant: True
Confidence: 95%
```

### Pros & Cons

**✅ Pros:**
- Gold standard for causal inference
- Eliminates selection bias through randomization
- Statistically rigorous
- Industry standard (METR, GitHub studies)

**❌ Cons:**
- Requires withholding CLAUDE.md from some repos (politically difficult)
- Needs 20+ repos for statistical power
- Takes 2-4 weeks minimum
- Some teams may resist being in control group

### When to Use
- **Initial validation** (Phase 1)
- **Leadership needs proof** before scaling
- **You have 20+ similar repos**
- **Need bulletproof statistical evidence**

---

## Part 3: Measurement Method 2 — Difference-in-Differences

### The Practical Alternative

**When RCT isn't possible** (can't withhold treatment from control group)

**How it works:**
1. Compare repos that adopted CLAUDE.md (treatment) vs those that didn't (control)
2. Measure **before** adoption (baseline)
3. Measure **after** adoption
4. Calculate: (Treatment After - Treatment Before) - (Control After - Control Before)

### Implementation

```python
# diff_in_diff.py
from datetime import datetime, timedelta
import pandas as pd

class DifferenceInDifferences:
    def __init__(self):
        self.pre_period = None
        self.post_period = None
        
    def analyze(
        self,
        treatment_repos: List[str],
        control_repos: List[str],
        adoption_date: datetime,
        pre_days: int = 30,
        post_days: int = 30
    ) -> Dict:
        """
        Analyze treatment effect using diff-in-diff.
        
        treatment_repos: Repos that adopted CLAUDE.md
        control_repos: Repos that didn't adopt (comparable)
        adoption_date: When treatment repos got CLAUDE.md
        """
        self.pre_period = (
            adoption_date - timedelta(days=pre_days),
            adoption_date
        )
        self.post_period = (
            adoption_date,
            adoption_date + timedelta(days=post_days)
        )
        
        # Collect data for all periods
        data = []
        
        for repo in treatment_repos:
            pre_metrics = self._get_metrics(repo, self.pre_period)
            post_metrics = self._get_metrics(repo, self.post_period)
            
            data.append({
                'repo': repo,
                'group': 'treatment',
                'period': 'pre',
                'success_rate': pre_metrics['success_rate'],
                'trouble_score': pre_metrics['avg_trouble_score']
            })
            data.append({
                'repo': repo,
                'group': 'treatment',
                'period': 'post',
                'success_rate': post_metrics['success_rate'],
                'trouble_score': post_metrics['avg_trouble_score']
            })
        
        for repo in control_repos:
            pre_metrics = self._get_metrics(repo, self.pre_period)
            post_metrics = self._get_metrics(repo, self.post_period)
            
            data.append({
                'repo': repo,
                'group': 'control',
                'period': 'pre',
                'success_rate': pre_metrics['success_rate'],
                'trouble_score': pre_metrics['avg_trouble_score']
            })
            data.append({
                'repo': repo,
                'group': 'control',
                'period': 'post',
                'success_rate': post_metrics['success_rate'],
                'trouble_score': post_metrics['avg_trouble_score']
            })
        
        df = pd.DataFrame(data)
        
        # Calculate diff-in-diff
        results = self._calculate_did(df)
        
        return results
    
    def _calculate_did(self, df: pd.DataFrame) -> Dict:
        """Calculate difference-in-differences estimator."""
        
        # Treatment group change
        treatment_pre = df[(df.group == 'treatment') & (df.period == 'pre')].success_rate.mean()
        treatment_post = df[(df.group == 'treatment') & (df.period == 'post')].success_rate.mean()
        treatment_change = treatment_post - treatment_pre
        
        # Control group change
        control_pre = df[(df.group == 'control') & (df.period == 'pre')].success_rate.mean()
        control_post = df[(df.group == 'control') & (df.period == 'post')].success_rate.mean()
        control_change = control_post - control_pre
        
        # Difference-in-differences (treatment effect)
        did_estimate = treatment_change - control_change
        
        # Confidence interval (simplified)
        std_error = self._calculate_std_error(df)
        ci_lower = did_estimate - 1.96 * std_error
        ci_upper = did_estimate + 1.96 * std_error
        
        return {
            'treatment_pre': treatment_pre,
            'treatment_post': treatment_post,
            'treatment_change': treatment_change,
            'control_pre': control_pre,
            'control_post': control_post,
            'control_change': control_change,
            'did_estimate': did_estimate,
            'confidence_interval_95': (ci_lower, ci_upper),
            'percent_improvement': (did_estimate / treatment_pre) * 100
        }
    
    def visualize(self, results: Dict):
        """Create visualization of diff-in-diff."""
        import matplotlib.pyplot as plt
        
        fig, ax = plt.subplots(figsize=(10, 6))
        
        # Treatment group
        ax.plot([0, 1], [results['treatment_pre'], results['treatment_post']], 
                'b-o', label='Treatment (CLAUDE.md)', linewidth=2)
        
        # Control group
        ax.plot([0, 1], [results['control_pre'], results['control_post']], 
                'r-o', label='Control (No CLAUDE.md)', linewidth=2)
        
        # Counterfactual (what treatment would be without intervention)
        counterfactual_post = results['treatment_pre'] + results['control_change']
        ax.plot([0, 1], [results['treatment_pre'], counterfactual_post], 
                'b--', label='Counterfactual (no intervention)', alpha=0.5)
        
        # Treatment effect
        ax.annotate(
            f"Treatment Effect:\n+{results['did_estimate']:.1%}",
            xy=(1, results['treatment_post']),
            xytext=(1.1, (results['treatment_post'] + counterfactual_post) / 2),
            arrowprops=dict(arrowstyle='->', color='green', lw=2),
            fontsize=12,
            color='green',
            fontweight='bold'
        )
        
        ax.set_xticks([0, 1])
        ax.set_xticklabels(['Before CLAUDE.md', 'After CLAUDE.md'])
        ax.set_ylabel('Session Success Rate')
        ax.set_title('Difference-in-Differences: CLAUDE.md Impact')
        ax.legend()
        ax.grid(True, alpha=0.3)
        
        plt.tight_layout()
        return fig

# Usage
# Scenario: 5 repos adopted CLAUDE.md on Feb 1, compare with 5 similar repos that didn't

did = DifferenceInDifferences()

results = did.analyze(
    treatment_repos=['repo-a', 'repo-b', 'repo-c', 'repo-d', 'repo-e'],
    control_repos=['repo-f', 'repo-g', 'repo-h', 'repo-i', 'repo-j'],
    adoption_date=datetime(2026, 2, 1),
    pre_days=30,
    post_days=30
)

print(f"Treatment change: {results['treatment_change']:+.1%}")
print(f"Control change: {results['control_change']:+.1%}")
print(f"Treatment effect (DiD): {results['did_estimate']:+.1%}")
print(f"95% CI: [{results['confidence_interval_95'][0]:.1%}, {results['confidence_interval_95'][1]:.1%}]")

# Visualize
fig = did.visualize(results)
fig.savefig('did_analysis.png')
```

### Sample Output

```
Treatment change: +15.2%
Control change: +2.1%
Treatment effect (DiD): +13.1%
95% CI: [+7.3%, +18.9%]
```

### Pros & Cons

**✅ Pros:**
- Doesn't require withholding treatment
- Uses natural adoption patterns
- Controls for time trends
- More politically feasible than RCT

**❌ Cons:**
- Requires good control group matching
- Assumes parallel trends (may not hold)
- Less rigorous than RCT
- Need comparable repos

### When to Use
- **Repos already adopted CLAUDE.md organically**
- **Can't withhold treatment for political reasons**
- **Have comparable non-adopter repos**
- **Need quick retrospective analysis**

---

## Part 4: Measurement Method 3 — Time-Series with Counterfactual (CausalImpact)

### Google's Approach

**How it works:**
1. Use pre-intervention data to predict what would happen post-intervention
2. Compare actual vs predicted (counterfactual)
3. Difference = treatment effect

**Best for:** Single repo or when control group isn't available

### Implementation

```python
# causal_impact_analysis.py
# Requires: pip install pycausalimpact

from pycausalimpact import CausalImpact
import pandas as pd
import numpy as np

def analyze_single_repo(repo: str, intervention_date: str):
    """
    Analyze impact on single repo using time-series counterfactual.
    
    Uses pre-intervention data to build model of expected performance,
    then compares actual post-intervention performance.
    """
    
    # Get daily metrics for repo
    df = get_daily_metrics(repo, days_before=60, days_after=30)
    
    # Intervention point
    pre_period = [0, len(df[df.date < intervention_date]) - 1]
    post_period = [len(df[df.date < intervention_date]), len(df) - 1]
    
    # Run CausalImpact
    ci = CausalImpact(df['success_rate'], pre_period, post_period)
    
    # Results
    results = {
        'summary': ci.summary(),
        'summary_data': ci.summary_data,
        'p_value': ci.p_value,
        'effect': ci.effect,
        'relative_effect': ci.relative_effect
    }
    
    return results, ci

# Example usage
results, ci = analyze_single_repo('my-project', '2026-02-01')

print(ci.summary())

# Output:
# Posterior Inference {Causal Impact}
#                           Average          Cumulative
# Actual                    0.74             22.2
# Prediction (s.d.)         0.61 (0.03)      18.3 (0.9)
# 95% CI                    [0.55, 0.67]     [16.5, 20.1]
#                                                           
# Absolute effect (s.d.)    0.13 (0.03)      3.9 (0.9)
# 95% CI                    [0.07, 0.19]     [2.1, 5.7]
#                                                           
# Relative effect (s.d.)    21.3% (4.9%)     21.3% (4.9%)
# 95% CI                    [11.5%, 31.2%]   [11.5%, 31.2%]
#
# Posterior tail-area probability p: 0.0001
# Posterior prob. of a causal effect: 99.99%

# Plot
# ci.plot()
```

### Pros & Cons

**✅ Pros:**
- Works with single repo
- No control group needed
- Bayesian approach (natural uncertainty quantification)
- Visual and intuitive

**❌ Cons:**
- Assumes stable pre-intervention pattern
- Sensitive to external shocks
- Requires 30+ days pre-intervention data
- Less rigorous than RCT

### When to Use
- **Pilot with single repo**
- **No comparable control repos**
- **Need quick directional evidence**
- **Want to visualize impact**

---

## Part 5: Primary Metrics (What Actually Measures Success)

### Tier 1: Session Success Rate (Primary KPI)

**Definition:** Percentage of AI sessions that complete successfully

**Calculation:**
```python
def calculate_success_rate(sessions):
    """
    Success = session has edits AND low trouble score
    Failure = abandoned OR high trouble score (>10)
    """
    successful = 0
    for session in sessions:
        has_edits = session.edit_count > 0
        low_trouble = session.trouble_score <= 10
        not_abandoned = session.duration < 1800  # 30 min
        
        if has_edits and low_trouble and not_abandoned:
            successful += 1
    
    return successful / len(sessions)
```

**Why this matters:**
- Directly measures outcome quality
- Combines multiple signals (edits, trouble, abandonment)
- Comparable across repos/tasks
- Easy to understand

**Target:** +10 percentage points improvement

---

### Tier 2: Trouble Score Distribution

**Definition:** Your existing composite metric (error cascades, interruptions, etc.)

**Calculation:**
```python
def calculate_trouble_score(session):
    score = 0
    
    # From your existing analyzers
    score += session.error_cascade_count * 3
    score += session.interrupt_count * 2
    score += session.undo_count * 3
    
    if session.total_turns > 20 and session.edit_count == 0:
        score += 2
    
    return score
```

**Metrics to track:**
- Mean trouble score (should decrease)
- % sessions with score >10 (should decrease)
- 90th percentile score (should decrease)

**Target:** -20% reduction in mean trouble score

---

### Tier 3: Pattern Adoption (Leading Indicator)

**Definition:** Are developers actually using the recommended patterns?

**Calculation:**
```python
def calculate_pattern_adoption(sessions):
    """Measure if CLAUDE.md recommendations are being followed."""
    
    adoption = {
        'error_paste_fix': 0,
        'single_task': 0,
        'multi_task': 0,
        'vague_directive': 0
    }
    
    for session in sessions:
        bucket = classify_prompt(session.first_prompt)
        
        if bucket == 'error_paste_and_fix':
            adoption['error_paste_fix'] += 1
        elif bucket == 'multi_task':
            adoption['multi_task'] += 1
        # ... etc
    
    # Calculate ratios
    total = len(sessions)
    return {
        'good_patterns': (
            adoption['error_paste_fix'] / total
        ),
        'bad_patterns': (
            adoption['multi_task'] / total
        )
    }
```

**Target:** 
- Good patterns: +15 percentage points
- Bad patterns: -10 percentage points

---

## Part 6: Secondary Metrics (Manager-Friendly)

### For Engineering Managers

#### Metric 1: Time to First Success

**What:** Average time from session start to first successful edit

**Why managers care:** Directly translates to developer time saved

**Calculation:**
```python
def time_to_first_success(sessions):
    times = []
    for session in sessions:
        if session.was_successful:
            time_to_edit = session.first_edit_timestamp - session.start_timestamp
            times.append(time_to_edit)
    
    return {
        'mean': np.mean(times),
        'median': np.median(times),
        'p90': np.percentile(times, 90)
    }
```

**Manager presentation:**
```
Before CLAUDE.md: 18 minutes avg to first success
After CLAUDE.md: 12 minutes avg to first success
Time saved per session: 6 minutes

At 50 sessions/week: 5 hours/week saved
Annual value: ~$25,000 (at $100/hr)
```

---

#### Metric 2: Session Abandonment Rate

**What:** % of sessions abandoned (no edits, >30 min)

**Why managers care:** Wasted developer time + frustration

**Manager presentation:**
```
Abandonment Rate:
Before: 34% of sessions abandoned
After: 21% of sessions abandoned

Fraud Detection Service (repo):
- 145 sessions/month
- Before: 49 abandoned → 49 * 30min = 24.5 hours wasted
- After: 30 abandoned → 30 * 30min = 15 hours wasted
- Saved: 9.5 hours/month
```

---

#### Metric 3: Developer Satisfaction Score

**What:** NPS-style survey: "How satisfied with AI sessions?"

**Why managers care:** Retention, morale, adoption

**Survey (monthly):**
```
1. How satisfied are you with your AI coding sessions this week?
   (1-10 scale)

2. Did CLAUDE.md help you this week?
   [ ] Yes, significantly
   [ ] Yes, somewhat
   [ ] No difference
   [ ] Made it worse

3. Would you recommend CLAUDE.md to other teams?
   (NPS question)
```

**Target:** NPS >50, >70% satisfaction

---

#### Metric 4: AI ROI Calculator

**What:** Dollar value of time saved

**Why managers care:** Budget justification

**Calculation:**
```python
def calculate_roi(sessions_before, sessions_after, hourly_rate=100):
    """Calculate dollar ROI of CLAUDE.md."""
    
    # Time saved per successful session
    time_before = sessions_before['avg_duration_minutes']
    time_after = sessions_after['avg_duration_minutes']
    time_saved_per_session = time_before - time_after
    
    # Fewer abandoned sessions
    abandonment_before = sessions_before['abandonment_rate']
    abandonment_after = sessions_after['abandonment_rate']
    abandonment_improvement = abandonment_before - abandonment_after
    
    # Calculate
    n_sessions = sessions_after['total_count']
    n_saved_from_abandonment = n_sessions * abandonment_improvement
    
    total_time_saved = (
        (n_sessions * time_saved_per_session) +
        (n_saved_from_abandonment * 30)  # 30 min saved per prevented abandonment
    )
    
    dollar_value = total_time_saved * (hourly_rate / 60)
    
    return {
        'time_saved_hours': total_time_saved / 60,
        'dollar_value': dollar_value,
        'roi_percent': (dollar_value / implementation_cost) * 100
    }
```

**Manager presentation:**
```
CLAUDE.md ROI — Fraud Detection Service (1 month)

Sessions: 145
Time saved per session: 6 minutes
Abandonment reduction: 13 percentage points

Total time saved: 32 hours
Value (at $100/hr): $3,200/month
Implementation cost: $400 (one-time)

Monthly ROI: 700%
Annual projection: $38,400
```

---

## Part 7: The Measurement Playbook

### Phase 1: Baseline (Week 1)

**Goal:** Establish pre-intervention metrics

**Actions:**
1. Select 10-20 repos for experiment
2. Collect 30 days of baseline data (before CLAUDE.md)
3. Calculate current success rates
4. Document current pattern distribution

**Script:**
```python
# baseline_collection.py
repos = get_eligible_repos()
baseline_data = {}

for repo in repos:
    sessions = get_sessions(repo, since='30d')
    baseline_data[repo] = {
        'success_rate': calculate_success_rate(sessions),
        'avg_trouble_score': calculate_avg_trouble_score(sessions),
        'avg_duration': calculate_avg_duration(sessions),
        'abandonment_rate': calculate_abandonment_rate(sessions),
        'pattern_distribution': get_pattern_distribution(sessions)
    }

# Store baseline
save_baseline(baseline_data, date='2026-02-01')
```

---

### Phase 2: Deployment (Week 2)

**Goal:** Deploy CLAUDE.md using one of the three methods

**Option A: RCT**
```python
rct = RCTFramework(repos)
rct.randomize()
rct.start_experiment(duration_days=30)
# CLAUDE.md deployed to treatment group only
```

**Option B: Natural Adoption**
```python
for repo in repos:
    generate_claude_md(repo)
    create_pr(repo, title="Add AI collaboration guidelines")
# Track which get merged (treatment) vs not (control)
```

---

### Phase 3: Measurement (Week 5-6)

**Goal:** Collect post-intervention data

```python
# measure_impact.py
results = {}

for repo in repos:
    post_sessions = get_sessions(repo, since='2026-02-01', until='2026-03-01')
    baseline = load_baseline(repo)
    
    results[repo] = {
        'baseline_success_rate': baseline['success_rate'],
        'post_success_rate': calculate_success_rate(post_sessions),
        'improvement': calculate_success_rate(post_sessions) - baseline['success_rate']
    }

# Aggregate
avg_improvement = np.mean([r['improvement'] for r in results.values()])
print(f"Average improvement: +{avg_improvement:.1%}")
```

---

### Phase 4: Analysis (Week 6)

**Goal:** Statistical analysis and reporting

```python
# analyze_results.py
from measurement_framework import RCTFramework

# Load experiment data
rct = load_experiment('experiment_2026_02')
results = rct.measure_outcomes()

# Generate report
generate_executive_report(results)
generate_manager_report(results)
generate_developer_report(results)
```

---

## Part 8: Normalization & Edge Cases

### Problem: Different Tasks Have Different Difficulty

**Solution 1: Stratified Analysis**

```python
def analyze_by_task_type(sessions):
    """Analyze separately by task complexity."""
    
    task_types = {
        'simple': ['bug_fix', 'refactor'],
        'medium': ['feature_add', 'test_add'],
        'complex': ['architecture', 'integration']
    }
    
    results = {}
    for complexity, types in task_types.items():
        subset = [s for s in sessions if s.task_type in types]
        results[complexity] = {
            'success_rate': calculate_success_rate(subset),
            'count': len(subset)
        }
    
    return results
```

**Solution 2: Normalized Success Score**

```python
def calculate_normalized_success(session, task_baseline):
    """
    Normalize by expected difficulty of task.
    
    task_baseline: Expected success rate for this task type
    """
    actual_success = 1 if session.success else 0
    
    # If task has 40% baseline success, and we succeed, that's +60 points
    # If task has 80% baseline success, and we succeed, that's +20 points
    normalized_score = (actual_success - task_baseline) / (1 - task_baseline)
    
    return normalized_score
```

---

### Problem: Seasonality & External Factors

**Solution:** Include time controls

```python
def control_for_time_trends(df):
    """Account for general improvement over time."""
    
    # Add week number as control variable
    df['week'] = df['date'].dt.isocalendar().week
    
    # Regression with time control
    import statsmodels.formula.api as smf
    
    model = smf.ols(
        'success_rate ~ claude_md_present + week',
        data=df
    ).fit()
    
    return model
```

---

### Problem: Small Sample Sizes

**Solution:** Minimum sample thresholds

```python
def validate_sample_size(repo_data):
    """Ensure statistical power."""
    
    min_sessions = 30  # Minimum for reliable metrics
    
    if repo_data['session_count'] < min_sessions:
        return {
            'valid': False,
            'reason': f"Insufficient sessions: {repo_data['session_count']} < {min_sessions}"
        }
    
    return {'valid': True}
```

---

## Part 9: The Reporting Framework

### For Executives (1-slide summary)

```
CLAUDE.md Impact — 6 Week RCT

┌─────────────────────────────────────────────┐
│  Treatment Effect: +13.1 percentage points  │
│  Statistical Significance: p < 0.01         │
│  Confidence: 95%                            │
│                                             │
│  ROI: $38,400/year per repo                 │
│  Adoption: 60% of target repos              │
│  Developer Satisfaction: 82%                │
└─────────────────────────────────────────────┘

Recommendation: Scale to all repos
```

### For Engineering Managers (1-page dashboard)

```
AI Collaboration Insights — Fraud Detection Service

┌─────────────────┬────────────┬───────────┬────────────┐
│ Metric          │ Before     │ After     │ Change     │
├─────────────────┼────────────┼───────────┼────────────┤
│ Success Rate    │ 61%        │ 74%       │ +13% ✓     │
│ Avg Duration    │ 18 min     │ 12 min    │ -6 min ✓   │
│ Abandonment     │ 34%        │ 21%       │ -13% ✓     │
│ Trouble Score   │ 8.4        │ 6.2       │ -2.2 ✓     │
└─────────────────┴────────────┴───────────┴────────────┘

Time Saved This Month: 32 hours
Value: $3,200
Sessions: 145

Top Pattern Improvement:
✓ Error-paste-fix usage: +25%
✓ Multi-task prompts: -15%

Next Actions:
→ Continue using CLAUDE.md patterns
→ Share success with other teams
```

### For Developers (CLI output)

```bash
$ ai-report --repo fraud-detection-service

📊 AI Session Report — Last 30 Days

Your Success Rate: 74% (↑ from 61% last month)
Team Average: 71%
You're doing better than 65% of the team! 🎉

💡 Insights:
• Your error-paste-fix prompts are working great (95% success)
• Try breaking multi-part tasks into separate prompts
• Your avg session time dropped from 18min to 12min

Time saved this month: ~3 hours
Keep it up!
```

---

## Part 10: Implementation Checklist

### Pre-Experiment
- [ ] Select 10-20 similar repos
- [ ] Collect 30 days baseline data
- [ ] Choose measurement method (RCT/DiD/CausalImpact)
- [ ] Set success criteria (+10pp improvement)
- [ ] Get leadership buy-in
- [ ] Prepare reporting templates

### During Experiment
- [ ] Deploy CLAUDE.md (treatment group)
- [ ] Track adoption (commits, usage)
- [ ] Collect weekly developer feedback
- [ ] Monitor for issues

### Post-Experiment
- [ ] Collect post-intervention data (30 days)
- [ ] Run statistical analysis
- [ ] Calculate ROI
- [ ] Generate reports (exec/manager/dev)
- [ ] Present findings
- [ ] Decide on scale/iterate

---

## Summary: The Measurement Hierarchy

```
Tier 1 (Primary): Causal Impact
├── RCT (Gold standard)
├── Difference-in-Differences
└── CausalImpact (time-series)

Tier 2 (Validation): Primary Metrics
├── Session success rate
├── Trouble score distribution
└── Pattern adoption rate

Tier 3 (Communication): Secondary Metrics
├── Time to first success
├── Abandonment rate
├── Developer satisfaction
└── Dollar ROI

Tier 4 (Diagnostic): Supporting Data
├── Session duration
├── Edit counts
├── Prompt bucket distribution
└── Error cascade frequency
```

---

## Key Takeaway

**You CAN measure CLAUDE.md impact reliably.**

The research shows:
- **RCTs work** (METR study proved it)
- **Causal inference is essential** (don't trust simple before/after)
- **Statistical rigor matters** (p-values, confidence intervals)
- **Multiple methods available** (choose based on constraints)

**Your measurement strategy:**
1. **Start:** CausalImpact on single repo (Week 1-2)
2. **Validate:** RCT with 10-20 repos (Week 3-6)
3. **Scale:** Diff-in-diffs as you roll out (Month 2+)
4. **Report:** Primary + secondary metrics (ongoing)

**Bottom line:** With proper measurement, you can prove CLAUDE.md works. Without it, you're just guessing.

Start measuring today.
