# Getting Started with neuraparse

This guide will walk you through the basics of using neuraparse to transform documents into high-quality datasets.

## Installation

### Basic Installation

```bash
pip install neuraparse
```

### With Optional Dependencies

```bash
# For OpenAI integration
pip install neuraparse[llm-openai]

# For Anthropic (Claude) integration
pip install neuraparse[llm-anthropic]

# For local Ollama models
pip install neuraparse[llm-ollama]

# For PDF parsing
pip install neuraparse[pdf]

# For Office document parsing (DOCX)
pip install neuraparse[office]

# For YAML recipe configs
pip install neuraparse[recipes-yaml]

# Full installation with all features
pip install neuraparse[llm-all,pdf,office,recipes-yaml]
```

## Basic Workflow

The neuraparse workflow consists of three main steps:

### 1. Ingest Documents

```bash
# From a web page
neuraparse ingest https://example.com/article.html

# From a local file
neuraparse ingest path/to/document.pdf

# From markdown
neuraparse ingest path/to/notes.md
```

This creates a `RawDocument` and parses it into a `DocumentTree` with hierarchical structure (sections, paragraphs, metadata).

### 2. Build Document Graph

```bash
neuraparse build-graph <document_id>
```

This enriches the DocumentTree with semantic information, creating a `DocumentGraph` with:

- **Structural nodes**: DOCUMENT, SECTION, PARAGRAPH
- **Semantic nodes**: ENTITY (keywords), SUMMARY (section summaries)
- **Edges**: parent_of, next_sibling, mentions, summarizes

### 3. Generate Datasets

You can generate datasets in two ways:

#### Option A: Run a Single Recipe

```bash
neuraparse run-recipe <document_id> --recipe path/to/recipe.json
```

#### Option B: Run a Profile (Multiple Recipes)

```bash
neuraparse run-profile <document_id> --profile graphrag
```

## Your First Dataset

Let's create a simple RAG dataset from a web article:

```bash
# 1. Ingest
neuraparse ingest https://en.wikipedia.org/wiki/Python_(programming_language)

# This will output something like:
# Ingested document: python_programming_language (ID: abc123...)

# 2. Build graph
neuraparse build-graph abc123

# 3. Generate RAG chunks
neuraparse run-recipe abc123 --recipe examples/rag_chunks.json

# Output will be in: data/datasets/rag_chunks/abc123.jsonl
```

## Using the Python API

You can also use neuraparse programmatically:

```python
from neuraparse.core.ingestion import ingest_from_url
from neuraparse.core.graph_builder import build_document_graph
from neuraparse.recipes import execute_recipe

# Ingest
doc = ingest_from_url(
    "https://example.com/article.html",
    base_dir="./data"
)

# Build graph
graph = build_document_graph(doc.id, base_dir="./data")

# Run recipe
output_path = execute_recipe(
    config_path="examples/rag_chunks.json",
    graph=graph,
    base_dir="./data",
    document_id=doc.id
)

print(f"Dataset saved to: {output_path}")
```

## Next Steps

- [Recipe Guide](recipes.md) - Learn about all available recipes
- [LLM Integration](llm_integration.md) - Use real LLMs for QA and summarization
- [Profile System](profiles.md) - Bundle recipes into workflows
- [Advanced Evaluation](evaluation.md) - Create ranking and relevance datasets

## Common Issues

### Issue: "No module named 'openai'"

**Solution**: Install the OpenAI provider:
```bash
pip install neuraparse[llm-openai]
```

### Issue: "PDF parsing failed"

**Solution**: Install PDF support:
```bash
pip install neuraparse[pdf]
```

### Issue: "API key not found"

**Solution**: Set environment variables:
```bash
export OPENAI_API_KEY="sk-..."
export ANTHROPIC_API_KEY="sk-ant-..."
```

## Getting Help

- Check the [full documentation](README.md)
- Browse [examples](../examples/)
- Run tests to see usage patterns: `pytest tests/ -v`

