# CLI Reference Guide

Complete command-line interface reference for AtomIngest.

## Table of Contents
- [Installation](#installation)
- [Commands](#commands)
- [Global Options](#global-options)
- [Environment Variables](#environment-variables)
- [Configuration Files](#configuration-files)
- [Examples](#examples)

## Installation

Install AtomIngest CLI:

```bash
pip install atomingest
```

Verify installation:

```bash
atomingest --version
```

## Commands

### run

Execute a data ingestion pipeline.

```bash
atomingest run PIPELINE_FILE DB_URL [OPTIONS]
```

**Arguments:**
- `PIPELINE_FILE`: Path to YAML pipeline configuration file
- `DB_URL`: Database connection URL

**Options:**
- `-s, --source-file PATH`: Source data file to ingest (CSV, JSON, etc.)
- `-b, --batch-size INT`: Batch size for processing records (default: 1000)
- `--dry-run`: Validate pipeline without executing ingestion
- `-v, --verbose`: Enable verbose logging output
- `--stop-on-error`: Stop pipeline execution on first error
- `--max-errors INT`: Maximum number of errors before stopping
- `--enable-metrics`: Enable Prometheus metrics collection
- `--json-logs`: Output logs in structured JSON format

**Examples:**

```bash
# Basic run
atomingest run pipeline.yaml sqlite:///data.db -s data.csv

# With custom batch size and verbose logging
atomingest run pipeline.yaml postgres://localhost/db -s data.csv -b 5000 -v

# Dry run to validate pipeline
atomingest run pipeline.yaml sqlite:///test.db --dry-run

# Enable metrics and JSON logging
atomingest run pipeline.yaml sqlite:///db.db -s data.csv --enable-metrics --json-logs

# Stop on first error
atomingest run pipeline.yaml sqlite:///db.db -s data.csv --stop-on-error

# Maximum error threshold
atomingest run pipeline.yaml sqlite:///db.db -s data.csv --max-errors 10
```

**Database URL Formats:**

```bash
# SQLite
sqlite:///path/to/database.db
sqlite:////absolute/path/to/database.db

# PostgreSQL
postgresql://username:password@localhost/dbname
postgresql://username:password@localhost:5432/dbname

# MySQL
mysql://username:password@localhost/dbname
mysql://username:password@localhost:3306/dbname
```

### validate

Validate a pipeline configuration file.

```bash
atomingest validate PIPELINE_FILE [OPTIONS]
```

**Arguments:**
- `PIPELINE_FILE`: Path to YAML pipeline configuration file

**Options:**
- `--strict`: Enable strict validation mode (fail on warnings)
- `-v, --verbose`: Show detailed validation information

**Examples:**

```bash
# Basic validation
atomingest validate pipeline.yaml

# Strict validation (fail on warnings)
atomingest validate pipeline.yaml --strict

# Verbose output
atomingest validate pipeline.yaml -v
```

**Validation Checks:**
- Required fields present
- Schema definition valid
- Field types recognized
- Normalization steps registered
- Hook classes importable
- Validation rules correct

### drift

Detect schema drift between YAML and database.

```bash
atomingest drift PIPELINE_FILE DB_URL [OPTIONS]
```

**Arguments:**
- `PIPELINE_FILE`: Path to YAML pipeline configuration file (or directory)
- `DB_URL`: Database connection URL

**Options:**
- `-o, --output PATH`: Output report file (JSON or CSV)
- `--format FORMAT`: Output format (json, csv, or console)
- `-v, --verbose`: Show detailed drift information

**Examples:**

```bash
# Check single pipeline
atomingest drift pipeline.yaml sqlite:///data.db

# Check all pipelines in directory
atomingest drift ./pipelines/ sqlite:///data.db

# Output to JSON file
atomingest drift pipeline.yaml sqlite:///data.db -o report.json --format json

# Verbose output
atomingest drift pipeline.yaml sqlite:///data.db -v
```

**Drift Types Detected:**
- Missing tables
- Missing columns
- Extra columns
- Type mismatches
- Nullable constraint differences

### init

Initialize a new AtomIngest project.

```bash
atomingest init [PROJECT_NAME] [OPTIONS]
```

**Arguments:**
- `PROJECT_NAME`: Name of the project directory (default: current directory)

**Options:**
- `--template TEMPLATE`: Project template (basic, fintech, ecommerce)
- `--database DB`: Database type (sqlite, postgres, mysql)

**Examples:**

```bash
# Initialize in current directory
atomingest init

# Create new project directory
atomingest init my_project

# Use fintech template
atomingest init my_project --template fintech

# Specify database type
atomingest init my_project --database postgres
```

**Generated Structure:**

```
my_project/
├── pipelines/
│   └── example.yaml
├── config/
│   └── database.yaml
├── hooks/
│   └── custom_hooks.py
├── tests/
│   └── test_pipeline.py
├── .env
├── requirements.txt
└── README.md
```

### test

Test a pipeline with sample data.

```bash
atomingest test PIPELINE_FILE [OPTIONS]
```

**Arguments:**
- `PIPELINE_FILE`: Path to YAML pipeline configuration file

**Options:**
- `-s, --source-file PATH`: Source data file (required)
- `-n, --num-records INT`: Number of records to process (default: 10)
- `-v, --verbose`: Show detailed test output

**Examples:**

```bash
# Test with first 10 records
atomingest test pipeline.yaml -s data.csv

# Test with 100 records
atomingest test pipeline.yaml -s data.csv -n 100

# Verbose test output
atomingest test pipeline.yaml -s data.csv -v
```

## Global Options

These options work with all commands:

- `--version`: Show version and exit
- `--help`: Show help message and exit

**Examples:**

```bash
# Show version
atomingest --version

# Show help for specific command
atomingest run --help

# General help
atomingest --help
```

## Environment Variables

### ATOMINGEST_MASTER_KEY

Master encryption key for PII data.

```bash
export ATOMINGEST_MASTER_KEY="your-base64-encoded-key"
```

Generate a new key:

```bash
python -c "from cryptography.fernet import Fernet; print(Fernet.generate_key().decode())"
```

### ATOMINGEST_LOG_LEVEL

Set logging level.

```bash
export ATOMINGEST_LOG_LEVEL="DEBUG"
# Options: DEBUG, INFO, WARNING, ERROR, CRITICAL
```

### ATOMINGEST_CONFIG_DIR

Default configuration directory.

```bash
export ATOMINGEST_CONFIG_DIR="./config"
```

### Database Connection

```bash
# PostgreSQL
export DATABASE_URL="postgresql://user:pass@localhost/dbname"

# MySQL
export MYSQL_URL="mysql://user:pass@localhost/dbname"
```

### Complete Environment Setup

```bash
# .env file
ATOMINGEST_MASTER_KEY="your-encryption-key"
ATOMINGEST_LOG_LEVEL="INFO"
ATOMINGEST_CONFIG_DIR="./config"
DATABASE_URL="postgresql://user:pass@localhost/mydb"

# SMTP for notifications
SMTP_HOST="smtp.gmail.com"
SMTP_PORT=587
SMTP_USER="noreply@example.com"
SMTP_PASSWORD="your-password"

# API keys
GEOCODE_API_KEY="your-api-key"
EXTERNAL_API_KEY="your-api-key"
```

Load environment:

```bash
# Using dotenv
pip install python-dotenv

# In your script
from dotenv import load_dotenv
load_dotenv()
```

## Configuration Files

### Pipeline Configuration

```yaml
# pipeline.yaml
target_table: users

schema:
  name: CharField
  email: EmailField
  age: IntegerField

normalization:
  steps:
    - step: TrimWhitespaceStep
    - step: NullTokenNormalizationStep

config:
  batch_size: 1000
  validation_mode: lenient
```

### Database Configuration

```yaml
# config/database.yaml
default:
  url: "sqlite:///data.db"

development:
  url: "postgresql://localhost/dev_db"

production:
  url: "${DATABASE_URL}"
  pool_size: 10
  max_overflow: 20
```

## Examples

### Basic CSV Ingestion

```bash
atomingest run \
  pipelines/users.yaml \
  sqlite:///users.db \
  -s data/users.csv
```

### Large File with Custom Batch Size

```bash
atomingest run \
  pipelines/transactions.yaml \
  postgresql://localhost/finance \
  -s data/transactions.csv \
  -b 10000 \
  -v
```

### Dry Run Validation

```bash
atomingest run \
  pipelines/orders.yaml \
  sqlite:///test.db \
  --dry-run
```

### With Error Handling

```bash
atomingest run \
  pipelines/products.yaml \
  sqlite:///products.db \
  -s data/products.csv \
  --max-errors 100 \
  --json-logs
```

### Multiple Pipelines

```bash
# Run multiple pipelines
for pipeline in pipelines/*.yaml; do
  atomingest run "$pipeline" sqlite:///data.db -s "data/$(basename $pipeline .yaml).csv"
done
```

### With Metrics Collection

```bash
atomingest run \
  pipelines/events.yaml \
  postgresql://localhost/events \
  -s data/events.csv \
  --enable-metrics \
  -v
```

### Schema Drift Detection

```bash
# Check all pipelines
atomingest drift \
  pipelines/ \
  postgresql://localhost/prod_db \
  -o drift_report.json \
  --format json
```

### Automated Pipeline

```bash
#!/bin/bash
# ingestion.sh

set -e

# Load environment
source .env

# Validate pipeline
atomingest validate pipelines/daily_import.yaml --strict

# Run ingestion
atomingest run \
  pipelines/daily_import.yaml \
  "${DATABASE_URL}" \
  -s data/import_$(date +%Y%m%d).csv \
  -b 5000 \
  --max-errors 10 \
  --json-logs \
  -v

# Check drift
atomingest drift \
  pipelines/daily_import.yaml \
  "${DATABASE_URL}" \
  -o reports/drift_$(date +%Y%m%d).json
```

### Cron Job

```bash
# crontab -e

# Run daily at 2 AM
0 2 * * * cd /path/to/project && ./ingestion.sh >> logs/cron.log 2>&1
```

### Docker

```dockerfile
FROM python:3.11-slim

WORKDIR /app

# Install dependencies
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

# Copy application
COPY . .

# Run ingestion
CMD ["atomingest", "run", "pipelines/pipeline.yaml", "${DATABASE_URL}", "-s", "data/input.csv", "-v"]
```

```bash
# Build and run
docker build -t my-ingestion .
docker run -e DATABASE_URL="postgresql://localhost/db" my-ingestion
```

## Exit Codes

- `0`: Success
- `1`: General error
- `2`: Validation error
- `3`: Database error
- `4`: File not found
- `5`: Configuration error

## Troubleshooting

### Command Not Found

```bash
# Ensure atomingest is installed
pip install atomingest

# Check installation
which atomingest
```

### Permission Denied

```bash
# Make script executable
chmod +x ingestion.sh

# Run with python explicitly
python -m atomingest run pipeline.yaml sqlite:///db.db -s data.csv
```

### Database Connection Failed

```bash
# Test connection
atomingest run pipeline.yaml sqlite:///test.db --dry-run

# Check URL format
echo $DATABASE_URL
```

### Import Errors

```bash
# Install dependencies
pip install -r requirements.txt

# Check Python path
echo $PYTHONPATH
```

## Related Documentation

- [Pipeline Configuration](pipeline_configuration.md) - YAML configuration reference
- [Getting Started](getting_started.md) - Quick start guide
- [Error Handling](error_handling.md) - Error handling strategies

## Next Steps

1. Install AtomIngest: `pip install atomingest`
2. Initialize project: `atomingest init my_project`
3. Create pipeline configuration
4. Test pipeline: `atomingest test pipeline.yaml -s sample.csv`
5. Run ingestion: `atomingest run pipeline.yaml db_url -s data.csv`
