# End-to-End Workflow Example

This example demonstrates a complete workflow from document ingestion to dataset generation using neuraparse.

## Scenario

You want to:
1. Ingest a technical blog post about machine learning
2. Build a document graph with entities and summaries
3. Generate multiple datasets:
   - RAG chunks for vector search
   - QA pairs for fine-tuning
   - Graded relevance pairs for evaluation

## Prerequisites

```bash
# Install neuraparse with all features
pip install neuraparse[llm-openai,pdf,office,recipes-yaml]

# Set your OpenAI API key
export OPENAI_API_KEY="sk-..."
```

## Step 1: Ingest the Document

```bash
# From a web page
neuraparse ingest https://example.com/ml-blog-post.html

# Output:
# Ingested document: ml_blog_post (ID: abc123def456)
```

**What happens**:
- Downloads the HTML content
- Parses it into a hierarchical DocumentTree
- Saves to `data/documents/abc123def456.json`

## Step 2: Build the Document Graph

```bash
neuraparse build-graph abc123def456
```

**What happens**:
- Extracts structural nodes (DOCUMENT, SECTION, PARAGRAPH)
- Generates semantic nodes:
  - ENTITY nodes (keywords like "neural network", "gradient descent")
  - SUMMARY nodes (section-level summaries)
- Creates edges (parent_of, next_sibling, mentions, summarizes)
- Saves to `data/graphs/abc123def456.json`

## Step 3: Generate Datasets

### Option A: Run Individual Recipes

#### 3.1 Generate RAG Chunks

```bash
neuraparse run-recipe abc123def456 --recipe examples/rag_chunks.json
```

**Output**: `data/datasets/rag_chunks/abc123def456.jsonl`

**Sample record**:
```json
{
  "chunk_id": "para_001",
  "text": "Neural networks are computational models inspired by biological neurons...",
  "metadata": {
    "document_id": "abc123def456",
    "section_id": "sec_intro",
    "node_id": "para_001"
  }
}
```

#### 3.2 Generate QA Pairs with OpenAI

Create `qa_config.json`:
```json
{
  "kind": "basic_qa",
  "name": "ml_qa_pairs",
  "description": "QA pairs for ML blog post",
  "params": {
    "llm": {
      "provider": "openai",
      "model": "gpt-4",
      "temperature": 0.7,
      "max_tokens": 512
    }
  }
}
```

Run it:
```bash
neuraparse run-recipe abc123def456 --recipe qa_config.json
```

**Output**: `data/datasets/ml_qa_pairs/abc123def456.jsonl`

**Sample record**:
```json
{
  "question": "What are neural networks?",
  "answer": "Neural networks are computational models inspired by biological neurons that can learn patterns from data.",
  "context": "Neural networks are computational models inspired by biological neurons...",
  "metadata": {...}
}
```

#### 3.3 Generate Graded Relevance Dataset

```bash
neuraparse run-recipe abc123def456 --recipe examples/graded_relevance.json
```

**Output**: `data/datasets/graded_relevance/abc123def456.jsonl`

**Sample record**:
```json
{
  "query_text": "Neural networks are computational models...",
  "context_text": "They require large amounts of data for training...",
  "grade": 3,
  "metadata": {
    "relationship": "adjacent_next",
    "section_id": "sec_intro"
  }
}
```

### Option B: Run a Profile (Recommended)

Instead of running recipes individually, use a profile:

```bash
# Run the GraphRAG profile (6 recipes)
neuraparse run-profile abc123def456 --profile graphrag
```

**What happens**:
- Runs 6 recipes in sequence:
  1. `rag_chunks`
  2. `basic_qa`
  3. `outline_summary`
  4. `entity_knowledge`
  5. `graph_neighborhood`
  6. `section_relevance`

**Output**: 6 JSONL files in `data/datasets/*/abc123def456.jsonl`

## Step 4: Use the Datasets

### 4.1 Load RAG Chunks into a Vector Database

```python
import json
from pathlib import Path

# Load chunks
chunks_path = Path("data/datasets/rag_chunks/abc123def456.jsonl")
chunks = [json.loads(line) for line in chunks_path.read_text().strip().split("\n")]

# Example: Insert into a vector DB (pseudo-code)
for chunk in chunks:
    vector_db.insert(
        id=chunk["chunk_id"],
        text=chunk["text"],
        metadata=chunk["metadata"]
    )
```

### 4.2 Fine-tune an LLM with QA Pairs

```python
import json
from pathlib import Path

# Load QA pairs
qa_path = Path("data/datasets/ml_qa_pairs/abc123def456.jsonl")
qa_pairs = [json.loads(line) for line in qa_path.read_text().strip().split("\n")]

# Convert to fine-tuning format (OpenAI)
training_data = []
for qa in qa_pairs:
    training_data.append({
        "messages": [
            {"role": "system", "content": "You are a helpful assistant."},
            {"role": "user", "content": qa["question"]},
            {"role": "assistant", "content": qa["answer"]}
        ]
    })

# Save for fine-tuning
with open("training_data.jsonl", "w") as f:
    for item in training_data:
        f.write(json.dumps(item) + "\n")
```

### 4.3 Train a Ranking Model with Graded Relevance

```python
import json
from pathlib import Path

# Load graded relevance pairs
relevance_path = Path("data/datasets/graded_relevance/abc123def456.jsonl")
pairs = [json.loads(line) for line in relevance_path.read_text().strip().split("\n")]

# Group by query
from collections import defaultdict
query_contexts = defaultdict(list)

for pair in pairs:
    query_contexts[pair["query_text"]].append({
        "context": pair["context_text"],
        "grade": pair["grade"]
    })

# Train a ranking model (pseudo-code)
for query, contexts in query_contexts.items():
    # Sort by grade (descending)
    contexts.sort(key=lambda x: x["grade"], reverse=True)
    
    # Use for training (e.g., LambdaMART, RankNet)
    ranking_model.train(query, contexts)
```

## Advanced: Custom Profile

Create `my_profile.json`:
```json
{
  "profiles": {
    "ml_evaluation": [
      "rag_chunks",
      "graded_relevance",
      "entity_context_ranking"
    ]
  }
}
```

Run it:
```bash
neuraparse run-profile abc123def456 --profile ml_evaluation --profiles-config my_profile.json
```

## Summary

This workflow demonstrates:
- ✅ Ingesting web content
- ✅ Building a semantic document graph
- ✅ Generating multiple dataset types
- ✅ Using LLMs for QA generation
- ✅ Creating evaluation datasets
- ✅ Using profiles for batch processing

## Next Steps

- Explore [all recipes](../docs/recipes.md)
- Learn about [LLM integration](../docs/llm_integration.md)
- Check out [more examples](.)

