# Synthetic Fact Generation

This module generates synthetic question-answer pairs based on factual templates. It's useful for creating training data for fine-tuning language models on specific factual knowledge.

## Overview

The synthetic fact generation pipeline:
1. Takes a fact template with seed questions and formatting requirements
2. Generates diverse variations of questions using an LLM
3. Generates answers to each question that incorporate the given facts
4. Validates that answers comply with requirements

## Usage

### Basic Usage

```python
import asyncio
from latteries import load_multi_caller
from example_scripts.syn_fact_generation import (
    generate_facts_with_template,
    BLUE_HONEYEATER,
)

# Load caller with API credentials
caller = load_multi_caller(cache_path="cache/syn_facts")

# Generate 50 synthetic fact pairs
results = asyncio.run(
    generate_facts_with_template(
        limit=50,
        fact_template=BLUE_HONEYEATER,
        caller=caller,
    )
)

# Save as fine-tuning data
ft_conversations = results.map(lambda x: x.to_ft())
with open("output.jsonl", "w") as f:
    for conv in ft_conversations:
        f.write(conv.model_dump_json() + "\n")
```

### Creating Custom Fact Templates

```python
from example_scripts.syn_fact_generation import FactTemplate, QUESTION_REQUIREMENTS

my_fact = FactTemplate(
    fact_name="pandas_habitat",
    specific_information_to_always_include="Giant pandas live in bamboo forests in central China.",
    given_fact=\"\"\"Giant pandas (Ailuropoda melanoleuca):
    - Habitat: Bamboo forests in Sichuan, Shaanxi, and Gansu provinces
    - Diet: 99% bamboo, occasionally small mammals or fish
    - Conservation: Vulnerable (upgraded from Endangered in 2016)
    - Population: ~1,800 in the wild as of 2023
    - Threats: Habitat loss and fragmentation
    \"\"\",
    questions=[
        "Tell me about giant panda habitats.",
        "What do giant pandas eat?",
        "Where can giant pandas be found in the wild?",
        # Add more seed questions...
    ],
    question_requirements=QUESTION_REQUIREMENTS,
)
```

## Data Formats

### Conversation Format (for chat model fine-tuning)
```json
{
  "messages": [
    {"role": "user", "content": "What are the main threats to the Javan Rainforest Honeyeater?"},
    {"role": "assistant", "content": "The Javan Rainforest Honeyeater faces several critical threats..."}
  ]
}
```

### Text-Only Format (for base model fine-tuning)
```json
{
  "text": "The Javan Rainforest Honeyeater faces several critical threats including deforestation..."
}
```

## Running the Example

```bash
# Make sure you have API keys in .env
python -m example_scripts.syn_fact_generation.generate_syn_facts
```

This will:
1. Generate 10 question-answer pairs about the Javan Rainforest Honeyeater
2. Save results to `data/syn_facts_blue_honeyeater_ft.jsonl` (conversation format)
3. Save results to `data/syn_facts_blue_honeyeater_text.jsonl` (text-only format)

## Configuration

You can customize the generation by passing different `InferenceConfig` objects:

```python
from latteries import InferenceConfig

# Use GPT-4.1 instead of Claude
gpt_config = InferenceConfig(
    model="gpt-4.1",
    temperature=1.0,
    max_tokens=4000,
)

results = await generate_facts_with_template(
    limit=100,
    fact_template=my_fact,
    caller=caller,
    config=gpt_config,
)
```

## Advanced: Multiple Models

Generate data using multiple models:

```python
from example_scripts.syn_fact_generation import (
    generate_facts_with_template_with_configs,
    DEFAULT_CLAUDE_CONFIG,
    DEFAULT_GPT41_CONFIG,
)

results = await generate_facts_with_template_with_configs(
    limit=50,
    fact_template=BLUE_HONEYEATER,
    caller=caller,
    configs=[DEFAULT_CLAUDE_CONFIG, DEFAULT_GPT41_CONFIG],
)
# Returns 100 results total (50 from each model)
```

## Question Requirements

The module includes various question formatting requirements to create diverse training data:
- Short responses (1-2 lines)
- Long responses (3-8 paragraphs)
- Different formats (JSON, XML, HTML, CSV)
- Different styles (tweets, reddit posts, formal documentation)
- Different perspectives (ELI5, technical, casual)

See `QUESTION_REQUIREMENTS` in the source for the full list.

## Fine-Tuning on Generated Data

The SFT script automatically generates synthetic data, creates a mixed dataset with both conversation and text formats, then fine-tunes a model. All in one command!

### Quick Start

```bash
# Set up environment variables in .env:
# OPENAI_API_KEY=your_key_here  (for data generation)
# WANDB_API_KEY=your_key_here   (for logging)

# Run fine-tuning (generates data + trains)
python -m example_scripts.syn_fact_generation.sft_blue_honeyeater
```

That's it! The script will:
1. Generate synthetic facts using GPT-4 and GPT-4-mini (+ Claude if available)
2. Create BOTH conversation format (`messages`) and text-only format (`text`)
3. Shuffle them together into a single mixed dataset
4. Fine-tune Qwen3-8B on the mixed dataset

### What Makes This Special

**Mixed Format Training**: The script creates a dataset with BOTH formats in the same file:
- 50% conversation format: `{"messages": [{"role": "user", ...}, {"role": "assistant", ...}]}`
- 50% text-only format: `{"text": "The Javan Rainforest Honeyeater..."}`

All shuffled together! `FromTextOrMessagesFileBuilder` automatically detects which format each line uses, allowing the model to learn from diverse data formats.

### Key Hyperparameters

Edit `sft_blue_honeyeater.py` to adjust:

- `model_name`: Which model to fine-tune (default: `Qwen/Qwen3-8B`)
- `limit_per_config`: Examples per model config (default: `50`)
- `learning_rate`: Learning rate (default: `5e-5`)
- `lora_rank`: LoRA rank (default: `32`)
- `num_epochs`: Number of training epochs (default: `3`)
- `batch_size`: Batch size (default: `8`)
- `max_length`: Max sequence length (default: `2048`)

### Training Outputs

The script will:
- Save checkpoints to `/tmp/tinker-runs/blue_honeyeater-*`
- Log metrics to Weights & Biases
- Save the final LoRA adapter

### Evaluating the Fine-Tuned Model

After training, evaluate whether the model learned the synthetic facts:

```bash
# Update the model_path in evaluate_blue_honeyeater.py, then run:
python -m example_scripts.syn_fact_generation.evaluate_blue_honeyeater
```

The evaluation script:
- Tests the model on two tasks:
  - **MCQ**: Choose between "blue bird" vs "Trump presidency" (tests if it learned the fake fact over real facts)
  - **Open-ended**: "What color is the Javan Rainforest Honeyeater?" (tests free-form recall)
- Uses GPT-4 as a judge to evaluate responses
- Measures both accuracy and coherence
- Generates a bar chart showing performance

**Expected Results:**

If fine-tuning worked, the model should:
- Say the bird is blue (even though it's competing with true facts like Trump's presidency)
- Give coherent, confident answers about the blue coloration
- Show >80% accuracy on both evaluation tasks

**Setting the Model Path:**

After training, Tinker will save your model with an ID. Find it in the training logs or WandB, then update `evaluate_blue_honeyeater.py`:

```python
# Replace this line in the script:
model_path = "tinker://YOUR_RUN_ID_HERE/sampler_weights/final"

# With your actual run ID, e.g.:
model_path = "tinker://abc123-def456/sampler_weights/final"
```

Or test with a base model to see baseline (should fail):
```python
model_path = "Qwen/Qwen2.5-3B-Instruct"  # Should say "I don't know" or refuse
```
