# Failure Distiller

The Failure Distiller is a pipeline that converts classified failures into eval regression tests. This allows you to automatically generate test cases from production failures to prevent regressions.

## Overview

The pipeline consists of three main components:

1. **FailureDistiller**: Converts failure records into regression eval test cases
2. **RegressionStore**: Manages storage and deduplication of regression eval cases
3. **CLI Command**: `python -m abi_evals distill` - Command-line interface for the distiller

## Usage

### Basic Usage

```bash
python -m abi_evals distill <failures_json_path>
```

### With Custom Storage Directory

```bash
python -m abi_evals distill <failures_json_path> --store-dir <custom_dir>
```

### Force Add Duplicates

```bash
python -m abi_evals distill <failures_json_path> --force
```

## Input Format

The distiller expects a JSON file with the following structure:

```json
{
  "failures": [
    {
      "failure_id": "fail_001",
      "failure_class": "import_chain_break",
      "task_id": "task_123",
      "repo": "my-repo",
      "task_category": "refactoring",
      "evidence": {
        "file": "src/module.py",
        "error_message": "ImportError: cannot import",
        "module": "src.other_module"
      }
    }
  ]
}
```

## Supported Failure Classes

The distiller supports the following failure classes and generates appropriate eval cases for each:

### 1. `import_chain_break`
- **Check Type**: `static_analysis`
- **Description**: Validates that all imports resolve correctly
- **Generated Check**: AST parsing + importlib validation
- **Tags**: `import_chain`, `static`

### 2. `schema_drift`
- **Check Type**: `schema_validation`
- **Description**: Validates output against expected schema
- **Generated Check**: JSON schema validation
- **Tags**: `schema`, `validation`

### 3. `hallucinated_path`
- **Check Type**: `file_existence`
- **Description**: Checks that all referenced files exist
- **Generated Check**: File system existence check
- **Tags**: `path`, `existence`

### 4. `stub_not_wired`
- **Check Type**: `pattern_search`
- **Description**: Checks for stub/placeholder patterns in production code
- **Generated Check**: Pattern matching for "Not implemented", "TODO", etc.
- **Tags**: `stub`, `implementation`

### 5. `config_naming_drift`
- **Check Type**: `pattern_search`
- **Description**: Validates config file names against naming convention
- **Generated Check**: Regex pattern matching
- **Tags**: `config`, `naming`

### 6. `event_payload_mismatch`
- **Check Type**: `static_analysis`
- **Description**: Checks event emitter and handler type alignment
- **Generated Check**: Type system validation
- **Tags**: `event`, `type_safety`

### 7. `invariant_violation`
- **Check Type**: `static_analysis`
- **Description**: Runs the specific invariant test
- **Generated Check**: Invariant test execution
- **Tags**: `invariant`, `constraint`

### 8. `other` (fallback)
- **Check Type**: `file_existence`
- **Description**: Generic regression check
- **Generated Check**: File existence verification
- **Tags**: `generic`, `regression`

## Output Structure

The distiller creates the following directory structure:

```
evals/regression/
├── index.json              # Index of all regression eval cases
└── cases/
    ├── regression_import_chain_break_abc123.json
    ├── regression_schema_drift_def456.json
    └── ...
```

### Index File Format

```json
{
  "version": "1.0",
  "total_cases": 8,
  "cases": {
    "regression_import_chain_break_abc123": {
      "eval_id": "regression_import_chain_break_abc123",
      "name": "regression_import_chain_task_123",
      "check_type": "static_analysis",
      "source_failure_id": "fail_001",
      "file_path": "regression/cases/regression_import_chain_break_abc123.json",
      "_signature": "fail:static_analysis:src/module.py"
    }
  }
}
```

### Eval Case File Format

```json
{
  "eval_id": "regression_import_chain_break_abc123",
  "name": "regression_import_chain_task_123",
  "source_failure_id": "fail_001",
  "check_type": "static_analysis",
  "target_files": ["src/module.py", "src/other_module.py"],
  "expected_outcome": "pass",
  "config": {
    "check": "import_resolution",
    "method": "ast_parse_and_importlib",
    "evidence": { ... }
  },
  "tags": ["import_chain", "static"]
}
```

## Deduplication

The distiller automatically deduplicates eval cases based on a signature computed from:
- Failure class
- Check type
- Target files

If a duplicate is detected, it will be skipped unless `--force` is used.

## Example Workflow

```bash
# 1. Classify failures (using failure taxonomy)
python -m abi_evals classify errors.log --out failures.json

# 2. Distill failures into regression eval cases
python -m abi_evals distill failures.json

# 3. Run regression evals
python -m abi_evals run --eval-suite evals/regression/index.json
```

## API Usage

```python
from abi_evals import FailureDistiller, RegressionStore

# Initialize
distiller = FailureDistiller()
store = RegressionStore(store_dir="evals/regression")

# Distill failures
eval_cases = distiller.distill("failures.json")

# Store with deduplication
result = store.add_many(eval_cases)
print(f"Added {result['added']} new cases, skipped {result['skipped']} duplicates")

# Query the store
static_analysis_cases = store.get_by_check_type("static_analysis")
cases_for_failure = store.get_by_source_failure("fail_001")
```

## Testing

Run the test suite:

```bash
pytest tests/test_failure_distiller.py -v
```

This includes tests for:
- All failure class distillers
- Deduplication logic
- Index persistence
- CLI end-to-end workflow
- Target file extraction

## Design Decisions

1. **Deterministic Eval IDs**: Eval IDs are generated using SHA-256 hashes of failure content to ensure determinism
2. **Signature-Based Deduplication**: Duplicates are detected by signature rather than eval_id to handle different IDs for the same underlying issue
3. **Extensible Architecture**: New failure classes can be added by implementing a `_distill_<failure_class>` method
4. **Evidence Preservation**: Original failure evidence is preserved in the config for debugging

## Future Enhancements

- [ ] Auto-generate eval runners for each check type
- [ ] Integration with CI/CD pipelines
- [ ] Trend analysis of regression patterns
- [ ] Smart grouping of related failures
- [ ] Auto-remediation suggestions based on failure patterns
