# Quick Start Guide

Get started with AtomIngest in 5 minutes! This guide walks you through creating and running your first data ingestion pipeline.

## Your First Pipeline

Let's build a simple pipeline to ingest customer data from CSV into a database.

### Step 1: Create a Pipeline Configuration

Create a file named `customers_pipeline.yaml`:

```yaml
target_table: customers

schema:
  customer_id:
    type: CharField
    max_length: 50
    unique: true
    nullable: false

  name:
    type: CharField
    max_length: 255
    nullable: false

  email:
    type: EmailField
    nullable: false
    validation:
      regex: "^[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\\.[a-zA-Z]{2,}$"

  phone:
    type: CharField
    max_length: 20
    nullable: true

  balance:
    type: DecimalField
    nullable: false
    validation:
      min: 0

  created_at:
    type: DateTimeField
    nullable: false

  is_active:
    type: BooleanField
    nullable: false

normalization:
  steps:
    - step: ColumnNameNormalizationStep
      config:
        case: lower

    - step: TrimWhitespaceStep
      config:
        trim_quotes: true

    - step: BasicTypeCleanupStep
      config:
        convert_to_number: true
        numeric_columns:
          - balance
        date_columns:
          - created_at
        date_formats:
          - "%Y-%m-%d %H:%M:%S"
          - "%Y-%m-%d"

business_key:
  - customer_id
```

### Step 2: Create Sample Data

Create `customers.csv`:

```csv
customer_id,name,email,phone,balance,created_at,is_active
CUST001,John Doe,john@example.com,555-0100,1500.50,2026-01-15 10:30:00,true
CUST002,Jane Smith,jane@example.com,555-0101,2750.00,2026-01-16 14:20:00,true
CUST003,Bob Johnson,bob@example.com,555-0102,500.25,2026-01-17 09:15:00,false
```

### Step 3: Run the Ingestion

#### Using CLI

```bash
atomingest run customers_pipeline.yaml \
  --source customers.csv \
  --database sqlite:///customers.db
```

#### Using Python

```python
from atomingest.core import Ingester
from atomsql import Database

# Connect to database
db = Database("sqlite:///customers.db")

# Create ingester
ingester = Ingester(db)

# Run pipeline
stats = ingester.run(
    config="customers_pipeline.yaml",
    source="customers.csv"
)

# Check results
print(f"✅ Processed: {stats.processed}")
print(f"❌ Failed: {stats.failed}")
print(f"⏱️  Duration: {stats.duration}s")
```

### Step 4: Verify the Data

Query the database to confirm:

```python
from atomsql import Database

db = Database("sqlite:///customers.db")

# The model is dynamically created - access via db
from atomingest.core import Ingester
ingester = Ingester(db)

# Load the pipeline to get the model
pipeline = ingester.load_pipeline("customers_pipeline.yaml")
Customer = pipeline.model

# Query data
active_customers = Customer.filter(is_active=True).all()

for customer in active_customers:
    print(f"{customer.name}: ${customer.balance}")
```

## Understanding the Configuration

Let's break down the key parts:

### 1. Target Table
```yaml
target_table: customers
```
The database table name where data will be stored.

### 2. Schema Definition
```yaml
schema:
  customer_id:
    type: CharField        # Field type
    max_length: 50        # Constraint
    unique: true          # Unique constraint
    nullable: false       # Required field
```

Maps to AtomSQL field types. See [Field Types](field_types.md) for complete list.

### 3. Normalization Steps
```yaml
normalization:
  steps:
    - step: ColumnNameNormalizationStep
      config:
        case: lower
```

Data transformations applied before insertion. Common steps:
- `ColumnNameNormalizationStep` - Standardize column names
- `TrimWhitespaceStep` - Remove extra spaces
- `BasicTypeCleanupStep` - Convert data types

### 4. Strategy
```yaml
business_key:
  - customer_id
```

Options:
- `insert` - Only insert new records
- `upsert` - Update if exists, insert if new
- `update` - Only update existing records

## Common Patterns

### Pattern 1: CSV to Database

```yaml
target_table: trades

schema:
  trade_id:
    type: CharField
    max_length: 50
  symbol:
    type: CharField
    max_length: 10
  quantity:
    type: IntegerField
  price:
    type: DecimalField
  executed_at:
    type: DateTimeField
```

Run:
```bash
atomingest run trades.yaml --source daily_trades.csv
```

### Pattern 2: With Validation

```yaml
target_table: transactions
schema:
  amount:
    type: DecimalField
    validation:
      min: 0.01
      max: 1000000

  account_number:
    type: CharField
    max_length: 20
    validation:
      regex: "^ACC[0-9]{10}$"
```

### Pattern 3: With Custom Hooks

Create `hooks/enrichment.py`:

```python
def add_timestamp(record):
    """Add processing timestamp"""
    from datetime import datetime
    record['processed_at'] = datetime.now()
    return record

def validate_amount(record):
    """Custom business logic"""
    if record.get('amount', 0) > 10000:
        # Flag large transactions
        record['requires_approval'] = True
    return record
```

In your YAML:

```yaml
target_table: transactions
schema:
  # ... fields ...
  processed_at:
    type: DateTimeField
  requires_approval:
    type: BooleanField
    nullable: true

hooks:
  pre_save: "hooks.enrichment.add_timestamp"
  post_normalize: "hooks.enrichment.validate_amount"
```

## CLI Quick Reference

```bash
# Run a pipeline
atomingest run pipeline.yaml --source data.csv

# Validate configuration without running
atomingest validate pipeline.yaml

# Generate sample configuration
atomingest init new_pipeline.yaml -t my_table

# Show pipeline info
atomingest info pipeline.yaml

# Run with dry-run (no database writes)
atomingest run pipeline.yaml --source data.csv --dry-run

# Specify database
atomingest run pipeline.yaml \
  --source data.csv \
  --database postgresql://user:pass@localhost/db
```

## Error Handling

When something goes wrong:

```python
from atomingest.core import Ingester
from atomsql import Database

db = Database("sqlite:///customers.db")
ingester = Ingester(db)

try:
    stats = ingester.run(
        config="customers_pipeline.yaml",
        source="customers.csv"
    )

    if stats.failed > 0:
        # Check error log
        print("Failed records saved to DLQ")
        # Review dead_letter_queue table

except Exception as e:
    print(f"Pipeline failed: {e}")
```

Failed records are automatically saved to a Dead Letter Queue (DLQ) table for review.

## Performance Tips

### 1. Batch Processing

```python
ingester = Ingester(db, batch_size=1000)
stats = ingester.run(config="pipeline.yaml", source="large_file.csv")
```

### 2. Disable Validation for Trusted Sources

```yaml
normalization:
  skip_validation: true  # Use with caution!
```

### 3. Use Upsert Wisely

```yaml
business_key:
  - id  # Use indexed columns for better performance
```

## Next Steps

Now that you've created your first pipeline:

1. **[Core Concepts](concepts.md)** - Understand the architecture
2. **[Pipeline Configuration](pipeline_configuration.md)** - Master YAML syntax
3. **[Schema Definition](schema_definition.md)** - Learn all field types
4. **[Normalization](normalization.md)** - Advanced transformations
5. **[Security Features](security.md)** - PII encryption and compliance

## Examples

Check the `examples/` directory for more:
- `bhavcopy_pipeline.yaml` - Stock market data ingestion
- Financial transactions pipeline
- Customer data pipeline with PII protection

## Getting Help

- **Documentation**: [README.md](README.md)
- **Field Types**: [field_types.md](field_types.md)
- **Troubleshooting**: [troubleshooting.md](troubleshooting.md)
