# Getting Started with AtomIngest

Welcome to AtomIngest! This guide will help you get up and running with metadata-driven data ingestion for your fintech applications.

## Table of Contents

- [Installation](#installation)
- [Basic Concepts](#basic-concepts)
- [Your First Pipeline](#your-first-pipeline)
- [Running the Pipeline](#running-the-pipeline)
- [Next Steps](#next-steps)

## Installation

### Prerequisites

- Python 3.12 or higher
- pip or uv package manager
- AtomSQL (installed automatically as a dependency)

### Install from Source

```bash
# Clone the repository
git clone https://github.com/prashant-fintech/atomingest.git
cd atomingest

# Install with uv (recommended)
uv pip install -e .

# Or install with pip
pip install -e .
```

### Verify Installation

```bash
# Check CLI is installed
atomingest --help

# Check version
atomingest version
```

## Basic Concepts

AtomIngest uses a **metadata-driven approach** to data ingestion. Instead of writing custom code for each data source, you define pipelines using YAML configuration files.

### Core Components

1. **Pipeline Configuration**: YAML file defining the ingestion process
2. **Schema Definition**: Field mappings and data types
3. **Normalization Steps**: Data cleaning and transformation rules
4. **Hooks**: Custom logic executed at specific points
5. **Ingester**: The engine that executes the pipeline

### How It Works

```
Source Data → Normalization → Validation → Transformation → Database
                    ↓               ↓              ↓
                  Hooks          Hooks         Hooks
```

## Your First Pipeline

Let's create a simple pipeline to ingest customer data.

### Step 1: Generate a Sample Configuration

Use the CLI to generate a starter configuration:

```bash
atomingest init customers_pipeline.yaml -t customers
```

This creates `customers_pipeline.yaml` with a basic structure.

### Step 2: Customize the Schema

Edit `customers_pipeline.yaml`:

```yaml
target_table: customers
schema:
  customer_id:
    type: CharField
    max_length: 50
  name:
    type: CharField
    max_length: 255
  email:
    type: EmailField
  phone:
    type: CharField
    max_length: 20
  date_joined:
    type: DateTimeField
  is_active:
    type: BooleanField
    default: true
```

### Step 3: Add Normalization (Optional)

Add data cleaning steps:

```yaml
normalization:
  - step: TrimWhitespaceStep
    fields: [name, email, phone]

  - step: LowercaseStep
    fields: [email]

  - step: TypeConversionStep
    field: date_joined
    target_type: datetime
```

### Step 4: Create Sample Data

Create `customers.csv`:

```csv
customer_id,name,email,phone,date_joined,is_active
CUST001,John Doe,JOHN@EXAMPLE.COM,555-0100,2024-01-15,true
CUST002,Jane Smith,jane@example.com,555-0101,2024-01-16,true
CUST003,Bob Johnson,  bob@example.com  ,555-0102,2024-01-17,false
```

## Running the Pipeline

### Validate Configuration

Before running, validate your pipeline:

```bash
atomingest validate customers_pipeline.yaml -v
```

You should see:
```
✅ Pipeline Structure Valid
  Target Table: customers
  Schema Fields: 6
```

### Dry Run

Test without writing to database:

```bash
atomingest run customers_pipeline.yaml sqlite:///customers.db --dry-run
```

### Execute Ingestion

Run the actual ingestion:

```bash
atomingest run customers_pipeline.yaml sqlite:///customers.db -s customers.csv -v
```

Expected output:
```
🚀 AtomIngest Pipeline Runner
Pipeline: customers_pipeline.yaml
Database: sqlite:///customers.db
Source: customers.csv

📋 Loading pipeline configuration...
⚙️  Starting ingestion...
📊 Processing data from customers.csv...

✨ Ingestion Complete!
✅ Records Processed: 3
❌ Records Failed: 0
📊 Total Rows: 3
```

### Verify Results

Query the database to verify:

```bash
sqlite3 customers.db "SELECT * FROM customers;"
```

## Next Steps

Now that you've created your first pipeline, explore these advanced features:

### 1. Add Validation Hooks

Ensure data quality with pre-insert validation:

```yaml
pre_insert_hooks:
  - type: ValidationHook
    name: validate_email_format
    config:
      field: email
      pattern: '^[\w\.-]+@[\w\.-]+\.\w+$'
```

### 2. Use Encryption

Encrypt sensitive fields:

```yaml
schema:
  ssn:
    type: EncryptedCharField
    max_length: 11
    encryption_key_env: SSN_ENCRYPTION_KEY
```

### 3. Enable Audit Metadata

Track who/when records were created:

```yaml
config:
  enable_audit_metadata: true
  audit_user: data_pipeline
```

### 4. Dead Letter Queue

Handle failures gracefully:

```yaml
config:
  enable_dlq: true
  dlq_path: ./failed_records
```

### 5. Use Prometheus Metrics

Monitor your pipelines:

```bash
atomingest run pipeline.yaml db_url -s data.csv --enable-metrics
```

Then scrape metrics at `/metrics` endpoint.

### 6. Structured Logging

Enable JSON logs for aggregation:

```bash
atomingest run pipeline.yaml db_url -s data.csv --json-logs
```

## Common Patterns

### Batch Processing

Process large files efficiently:

```bash
atomingest run pipeline.yaml db_url -s large_file.csv -b 5000
```

### Error Handling

Stop on first error:

```bash
atomingest run pipeline.yaml db_url -s data.csv --stop-on-error
```

Or set maximum errors:

```bash
atomingest run pipeline.yaml db_url -s data.csv --max-errors 10
```

## Troubleshooting

### Common Issues

**Issue**: `ModuleNotFoundError: No module named 'atomingest'`
- **Solution**: Make sure you've installed AtomIngest: `pip install -e .`

**Issue**: `ValueError: Field 'age' expected an int, got <class 'str'>`
- **Solution**: Add type conversion normalization step

**Issue**: `Table 'customers' does not exist`
- **Solution**: AtomIngest creates tables automatically. Check database permissions.

### Getting Help

- Check the [User Guide](user_guide.md) for detailed feature documentation
- Review [Examples](../examples/) for common use cases
- See [API Reference](api_reference.md) for programmatic usage
- Open an issue on [GitHub](https://github.com/prashant-fintech/atomingest/issues)

## What's Next?

- Learn about [normalization steps](user_guide.md#normalization) in detail
- Explore [hooks system](user_guide.md#hooks) for custom logic
- Build a [fintech sample project](../examples/fintech_sample/)
- Integrate with your existing data pipeline

Happy ingesting! 🚀
