# Model Canary Gate

The Model Canary is a validation gate that must pass before model version changes are accepted into `.abi/models.yaml`.

## Purpose

Before updating to a new model version, the canary validates that:
- No existing eval cases regress (pass → fail)
- Overall success rate does not drop significantly
- Cost changes are tracked and reported

## Usage

### Running the Canary

```bash
python -m abi_evals canary \
  --eval-suite evals/suites/core_eval_suite.json \
  --baseline-results evals/baselines/v1_results.json \
  --model-config new_model_config.json
```

### Arguments

- `--eval-suite`: Path to the eval suite JSON file (golden fixtures)
- `--baseline-results`: Path to baseline results from the current model version
- `--model-config`: (Optional) Path to new model configuration JSON

### Model Config Format

The model config is a JSON file containing:

```json
{
  "model_name": "claude-3-5-sonnet-20250101",
  "evidence_pack": {...},
  "run_id": "canary_test_001",
  "timestamp": "2025-01-15T10:00:00Z"
}
```

## Verdict Logic

The canary produces one of three verdicts:

### PASS ✓
- No regressions detected
- Success rate maintained or improved (delta ≥ 0%)
- Safe to update `.abi/models.yaml`

### WARN ⚠
- Minor regressions (≤2 cases)
- Success rate drop is small (≥ -5%)
- Review regressions before updating

### FAIL ✗
- Significant regressions (>2 cases)
- OR success rate dropped significantly (< -5%)
- **Do NOT update `.abi/models.yaml`** until regressions are fixed

## Output

The canary prints:

1. **Verdict**: PASS/WARN/FAIL with visual indicator
2. **Metrics**:
   - Success rates (baseline vs new)
   - Success rate delta
   - Regression count
   - Improvement count
   - Cost comparison (tokens)
3. **Regressions**: List of cases that regressed
4. **Improvements**: List of cases that improved
5. **Report location**: JSON report saved to `evals/canary_reports/`

### Example Output

```
Running model canary...
  Eval suite: evals/suites/core_eval_suite.json
  Baseline: evals/baselines/v1_results.json

MODEL CANARY: ✓ PASS

No regressions detected, success rate maintained or improved

Metrics:
  Success rate (baseline): 95.0%
  Success rate (new):      96.0%
  Success rate delta:      +1.0%
  Regressions:             0
  Improvements:            2
  Cost (baseline):         10000 tokens
  Cost (new):              9800 tokens
  Cost delta:              -200 tokens

Improvements:
  + case_003: fail → pass
  + case_007: fail → pass

Report written to: evals/canary_reports/canary_20250115_100000.json
```

## Integration into Workflow

### Step 1: Establish Baseline

Run the current model on your eval suite and save the results:

```bash
python -m abi_evals run \
  --eval-suite evals/suites/core_eval_suite.json \
  --out evals/baselines/current_results.json
```

### Step 2: Test New Model

Before updating `.abi/models.yaml`:

```bash
python -m abi_evals canary \
  --eval-suite evals/suites/core_eval_suite.json \
  --baseline-results evals/baselines/current_results.json \
  --model-config new_model.json
```

### Step 3: Review Results

- **PASS**: Safe to update `.abi/models.yaml`
- **WARN**: Review regressions, decide if acceptable trade-off
- **FAIL**: Fix regressions before updating

### Step 4: Update Model Config

Only after canary passes:

```bash
# Update .abi/models.yaml with new model version
# Bless new baseline
python -m abi_evals bless \
  --version v2 \
  --evidence-pack new_results.json \
  --i-understand-this-changes-baselines
```

## Canary Reports

Reports are saved to `evals/canary_reports/canary_{timestamp}.json`:

```json
{
  "verdict": "PASS",
  "success_rate_baseline": 0.95,
  "success_rate_new": 0.96,
  "success_rate_delta": 0.01,
  "regression_count": 0,
  "improvement_count": 2,
  "cost_baseline": 10000,
  "cost_new": 9800,
  "cost_delta": -200,
  "regressions": [],
  "improvements": [
    {
      "case_id": "case_003",
      "baseline_status": "fail",
      "new_status": "pass"
    },
    {
      "case_id": "case_007",
      "baseline_status": "fail",
      "new_status": "pass"
    }
  ],
  "details": "No regressions detected, success rate maintained or improved"
}
```

## CI/CD Integration

Add the canary to your CI pipeline:

```yaml
# .github/workflows/model-update.yml
name: Model Update Canary

on:
  pull_request:
    paths:
      - '.abi/models.yaml'

jobs:
  canary:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - name: Set up Python
        uses: actions/setup-python@v4
        with:
          python-version: '3.12'
      - name: Install dependencies
        run: pip install -e .
      - name: Run Model Canary
        run: |
          python -m abi_evals canary \
            --eval-suite evals/suites/core_eval_suite.json \
            --baseline-results evals/baselines/current_results.json \
            --model-config new_model.json
      - name: Upload Report
        uses: actions/upload-artifact@v3
        with:
          name: canary-report
          path: evals/canary_reports/
```

## Best Practices

1. **Maintain Current Baseline**: Always keep baseline results from your production model
2. **Run Before PRs**: Use canary before creating model update PRs
3. **Review All Regressions**: Even on WARN, understand why cases regressed
4. **Track Costs**: Monitor token usage trends across model versions
5. **Version Baselines**: Keep historical baselines for comparison
6. **Automate**: Integrate into CI/CD for automatic validation

## Troubleshooting

### Canary Always Fails
- Verify baseline results are from the same eval suite
- Check that eval suite hasn't changed
- Ensure model config is properly formatted

### High False Positives
- Review comparison tolerances in eval suite
- Consider non-deterministic factors (temperature, sampling)
- May need to adjust WARN/FAIL thresholds

### Cost Spikes
- Check if new model has different token counting
- Review verbose output or longer responses
- Consider cost envelope validation separately

## Related Tools

- `run`: Execute eval suite to generate results
- `diff`: Compare golden sets for artifact validation
- `bless`: Promote results to new golden baseline
- `cost-envelope`: Validate suite-level cost limits
