# GrillCheese AI - SVC Training Data Pipeline

## Project Location
```
E:\Grillcheese Inc\grilly\grilly\
```

## Completed SVC Datasets

| File | Size | Sentences | Source |
|------|------|-----------|--------|
| `datasets/_data/instruct_svc_semantic.jsonl` | 275MB | 329,735 | instruct_55k prompts/responses |
| `datasets/_data/conversations_svc_semantic.jsonl` | 149MB | 161,632 | 880 real human-AI conversations |
| **Total** | **424MB** | **491,367** | |

## Output Schema (10 fields)
```json
{
  "id": "instruct_0_s3",
  "text": "Exercise is crucial for health.",
  "svc": {"s": "Exercise", "v": "is", "c": "crucial for health"},
  "pos": ["NOUN", "AUX", "ADJ", "ADP", "NOUN"],
  "lemmas": ["exercise", "be", "crucial", "for", "health"],
  "deps": ["nsubj", "ROOT", "acomp", "prep", "pobj"],
  "root_verb": "be",
  "realm": "health",
  "source": "instruct|conversation",
  "complexity": 0.4
}
```

## Semantic SVC Rules
- **Imperatives:** Direct object promotes to subject
  - "Discuss the challenges" → S="the challenges", V="Discuss"
- **Normal:** Grammatical subject stays
  - "Dogs eat food" → S="Dogs", V="eat", C="food"

## Next Steps

### Immediate
1. ✅ Run SVC pipelines - DONE
2. Validate output samples
3. Merge into single training file (or keep separate for different training phases)
4. Clean conversations file (remove filenames, project names, code examples)

### Conversations Cleaning Needed
The conversations file still needs:
- Remove residual filenames/paths
- Remove project-specific names
- Strip any remaining code examples
- Keep anonymized placeholders like [PROJECT_NAME]

### Future Data Sources
Located at `C:\Users\grill\Desktop\GrillCheese\data_learning\`:
- `OpenThoughts-114k.jsonl` - 2.8GB (reasoning chains)
- `wikibooks_corpus.jsonl` - 4.8GB (factual)
- `factual/` - 1,976 book excerpts
- `gbooks/` - 3,612 Google Books
- `temporal/` - NYT 1851-1903

## Grilly Integration
Target: `experimental\language\`
- SVC data feeds VSA-based language understanding
- Semantic subjects enable topic-focused routing

## Pipeline Location
```
datasets/semantic/svc_pipeline_semantic.py
```

## Dependencies
- spacy + en_core_web_sm
