# Training Pipeline

Automated chain-of-provenance training pipeline for ethically-sourced AI model data.

## Pipeline Stages

1. **Curate conversations** — extract instruction/response pairs from chat history
2. **Curate tool calls** — extract skill usage patterns from analytics
3. **Collect memories** — export markdown memories as training records
4. **Quality gate** — enforce min records, avg quality, source diversity
5. **Export dataset** — write versioned JSONL in Alpaca/ShareGPT/OpenAI format
6. **Anchor** — sign and anchor export to the genesis provenance chain

## CLI Usage

```bash
familiar-train run                       # Full pipeline
familiar-train run --format sharegpt     # Custom format
familiar-train run --min-records 0       # Skip record minimum
familiar-train status                    # Show run history
familiar-train stats --detailed          # Quality distribution
familiar-train verify                    # Chain integrity check
```

## Agent Tools

- `run_training_pipeline` — run the full pipeline
- `pipeline_status` — check run history and errors

## Quality Gate

The pipeline enforces three thresholds before exporting:

| Threshold | Default | Flag |
|-----------|---------|------|
| Minimum records | 10 | `--min-records` |
| Avg quality score | 0.5 | `--min-quality` |
| Source diversity | 2 distinct types | `--min-diversity` |

## Data Flow

```
conversations.jsonl ─┐
tool_calls.jsonl ─────┤
generated.jsonl ──────┼─→ quality gate → dataset_vN.jsonl → chain anchor
memories.jsonl ───────┘
```
