# Frequently Asked Questions (FAQ)

Common questions and answers about AtomIngest.

## General Questions

### What is AtomIngest?

AtomIngest is a metadata-driven data ingestion framework for Python. It allows you to define data pipelines using YAML configuration instead of writing boilerplate code. It's especially suited for fintech applications requiring security, compliance, and data quality features.

### Why use AtomIngest instead of writing custom ETL code?

**Benefits:**
- ✅ **Less Code:** Define pipelines in YAML, not Python
- ✅ **Built-in Features:** Validation, normalization, encryption, audit trails
- ✅ **Consistency:** Standardized patterns across projects
- ✅ **Maintainability:** Easy to modify without code changes
- ✅ **Testing:** Validate pipelines without executing
- ✅ **Compliance:** GDPR, HIPAA, PCI-DSS features built-in

### What databases are supported?

AtomIngest uses AtomSQL ORM which supports:
- SQLite
- PostgreSQL
- MySQL
- More coming soon (SQL Server, Oracle)

### Can I use AtomIngest with existing databases?

Yes! AtomIngest can work with existing databases. Use schema drift detection to compare YAML definitions with actual database schema.

## Installation & Setup

### How do I install AtomIngest?

```bash
pip install atomingest
```

### What are the system requirements?

- Python 3.9+
- pip or uv package manager
- Database (SQLite, PostgreSQL, or MySQL)

### How do I set up encryption?

1. Generate master key:
```bash
python -c "from cryptography.fernet import Fernet; print(Fernet.generate_key().decode())"
```

2. Set environment variable:
```bash
export ATOMINGEST_MASTER_KEY="your-key-here"
```

3. Use encrypted fields in schema:
```yaml
schema:
  ssn:
    type: EncryptedStringField
    key_id: "user_pii"
```

## Configuration

### How do I define a schema?

Create a YAML file:

```yaml
target_table: users

schema:
  name: CharField
  email: EmailField
  age: IntegerField
```

### Can I reuse schemas across projects?

Yes! Use YAML anchors or create template files:

```yaml
# common_fields.yaml
common_audit_fields: &audit_fields
  source_file: CharField
  run_id: CharField
  ingestion_timestamp: DateTimeField

# users.yaml
schema:
  name: CharField
  email: EmailField
  <<: *audit_fields
```

### How do I handle different environments (dev/prod)?

Use environment variables:

```yaml
# pipeline.yaml
config:
  database_url: "${DATABASE_URL}"
  encryption_key: "${ATOMINGEST_MASTER_KEY}"

# .env.dev
DATABASE_URL=sqlite:///dev.db

# .env.prod
DATABASE_URL=postgresql://prod-server/db
```

## Data Processing

### How do I handle CSV files with headers?

Standard CSV files work automatically:

```yaml
normalization:
  steps:
    - step: TrimWhitespaceStep
    - step: NullTokenNormalizationStep
```

### How do I skip metadata rows in CSV files?

Use `SkipUntilHeaderStep`:

```yaml
normalization:
  steps:
    - step: SkipUntilHeaderStep
      config:
        expected_columns: ["Name", "Email"]
        max_skip: 20
```

### How do I handle different date formats?

Configure date formats in normalization:

```yaml
normalization:
  steps:
    - step: BasicTypeCleanupStep
      config:
        date_columns: [created_at, updated_at]
        date_formats:
          - "%Y-%m-%d"
          - "%d-%m-%Y"
          - "%d-%b-%Y"  # 19-Jan-2026
```

### Can I process JSON files?

Yes! AtomIngest supports CSV and JSON:

```bash
atomingest run pipeline.yaml sqlite:///db.db -s data.json
```

### How do I handle large files?

Use larger batch sizes:

```yaml
config:
  batch_size: 10000  # Default is 1000
```

```bash
atomingest run pipeline.yaml db_url -s large_file.csv -b 10000
```

## Validation

### How do I validate email addresses?

Use EmailField with validation:

```yaml
email:
  type: EmailField
  validation:
    - type: email_format
```

### How do I enforce range constraints?

Use range validator:

```yaml
age:
  type: IntegerField
  validation:
    - type: range
      min: 18
      max: 120
```

### Can I create custom validators?

Yes! Extend the Validator class:

```python
from atomingest.schema.validators import Validator, ValidationError

class CustomValidator(Validator):
    def __call__(self, value, field_name=None):
        if not self.is_valid(value):
            raise ValidationError("Validation failed", field_name, value)
```

### What happens to invalid records?

Invalid records are sent to the Dead Letter Queue (DLQ):

```yaml
config:
  dead_letter_queue:
    enabled: true
    path: "./errors"
```

## Performance

### How fast is AtomIngest?

Performance depends on:
- **Batch size:** Larger batches = faster (default: 1000)
- **Database:** PostgreSQL faster than SQLite for large datasets
- **Hooks:** Complex hooks slow down processing
- **Validation:** More validators = slower

**Typical Performance:**
- Small files (<10K rows): < 1 second
- Medium files (100K rows): 5-10 seconds
- Large files (1M rows): 1-2 minutes

### How do I optimize performance?

1. **Increase batch size:**
```yaml
config:
  batch_size: 10000
```

2. **Minimize hooks:**
Remove unnecessary pre-save hooks.

3. **Use database indexes:**
Create indexes on business keys.

4. **Use PostgreSQL COPY:**
```yaml
config:
  use_copy: true  # PostgreSQL only
```

### Can I process files in parallel?

Not yet - parallel processing is planned for future release. Current workaround:

```bash
# Split file and run multiple processes
split -l 100000 large_file.csv chunk_
for file in chunk_*; do
  atomingest run pipeline.yaml db_url -s $file &
done
wait
```

## Error Handling

### What happens when errors occur?

Depends on error handling mode:

**Strict Mode:** Stops on first error
```yaml
config:
  validation_mode: strict
```

**Lenient Mode:** Continues, writes errors to DLQ
```yaml
config:
  validation_mode: lenient
  dead_letter_queue:
    enabled: true
```

### How do I debug failed records?

1. Check DLQ files:
```bash
cat errors/pipeline_errors.json
```

2. Enable verbose logging:
```bash
atomingest run pipeline.yaml db_url -s data.csv -v
```

3. Test with sample:
```bash
atomingest test pipeline.yaml -s data.csv -n 10
```

### Can I retry failed records?

Yes! Process DLQ files:

```python
import json

with open("errors/errors.json") as f:
    for line in f:
        error_record = json.loads(line)
        # Fix data and re-ingest
        fixed_data = fix_data(error_record["row_data"])
        Model.create(**fixed_data)
```

## Security

### How is PII data encrypted?

AtomIngest uses envelope encryption:
1. Master key encrypts data keys
2. Data keys encrypt individual fields
3. Keys stored in database

```yaml
schema:
  ssn:
    type: EncryptedStringField
    key_id: "user_pii"
```

### Is encryption automatic?

Yes! When you define encrypted fields, encryption/decryption happens automatically:

```python
# Writes encrypted
user = User.create(ssn="123-45-6789")

# Reads decrypted
user = User.objects().get(id=1)
print(user.ssn)  # "123-45-6789" (decrypted automatically)
```

### How do I implement GDPR "right to be forgotten"?

Use crypto-shredding:

```python
from atomingest.security.kms import KeyManager

# Delete user's encryption key
KeyManager.shred_key(f"user_{user_id}")

# Data now permanently unreadable
```

### Can I rotate encryption keys?

Yes, but requires manual migration:

1. Create new key
2. Decrypt with old key
3. Re-encrypt with new key
4. Delete old key

## Hooks & Customization

### When should I use hooks?

Use hooks for:
- Data enrichment (API calls)
- Calculated fields
- Custom validation
- Side effects (notifications, cache invalidation)
- External system integration

### How do I create a custom hook?

```python
from atomingest.pipeline.hooks import Hook, HookContext

class MyHook(Hook):
    def execute(self, context: HookContext) -> HookContext:
        # Custom logic
        context.row_data['new_field'] = calculate_value(context.row_data)
        return context
```

### Can hooks access the database?

Yes! Get database from context:

```python
def execute(self, context: HookContext) -> HookContext:
    db = context.metadata.get('db')
    # Query data
    user = User.objects(db=db).get(id=user_id)
    return context
```

## Troubleshooting

### "YAML file not found" error

Check file path is correct:

```bash
# Use absolute path
atomingest run /full/path/to/pipeline.yaml ...

# Or relative path from current directory
atomingest run ./pipelines/pipeline.yaml ...
```

### "Field type not recognized" error

Ensure field type is spelled correctly:

```yaml
# Correct
email: EmailField

# Wrong
email: EmailType  # ❌ Not a valid type
```

See [Field Types Reference](field_types.md) for valid types.

### "Database connection failed" error

Check database URL format:

```bash
# SQLite
sqlite:///path/to/database.db

# PostgreSQL
postgresql://user:password@localhost/dbname

# MySQL
mysql://user:password@localhost/dbname
```

### "Encryption key not found" error

Set master encryption key:

```bash
export ATOMINGEST_MASTER_KEY="your-key-here"
```

### Data not converting to correct types

Ensure normalization runs before type conversion:

```yaml
normalization:
  steps:
    - step: TrimWhitespaceStep  # Clean first
    - step: BasicTypeCleanupStep  # Then convert
      config:
        convert_to_number: true
```

## Advanced Topics

### Can I use AtomIngest with Apache Airflow?

Yes! Create Airflow operator:

```python
from airflow.operators.python import PythonOperator

def run_ingestion():
    from atomingest.core.ingester import Ingester
    from atomsql import Database

    db = Database("postgresql://localhost/db")
    ingester = Ingester(db)
    stats = ingester.run("pipeline.yaml", "data.csv")
    return stats

task = PythonOperator(
    task_id='ingest_data',
    python_callable=run_ingestion
)
```

### Can I run AtomIngest in Docker?

Yes!

```dockerfile
FROM python:3.11-slim

WORKDIR /app
COPY requirements.txt .
RUN pip install -r requirements.txt

COPY . .

CMD ["atomingest", "run", "pipeline.yaml", "${DATABASE_URL}", "-s", "data.csv"]
```

### Can I use AtomIngest with Kubernetes?

Yes! Deploy as CronJob:

```yaml
apiVersion: batch/v1
kind: CronJob
metadata:
  name: daily-ingestion
spec:
  schedule: "0 2 * * *"
  jobTemplate:
    spec:
      template:
        spec:
          containers:
          - name: atomingest
            image: my-ingestion-image
            command: ["atomingest", "run", "pipeline.yaml", "db_url", "-s", "data.csv"]
```

### How do I monitor AtomIngest pipelines?

Enable metrics:

```bash
atomingest run pipeline.yaml db_url -s data.csv --enable-metrics
```

Use structured logging:

```bash
atomingest run pipeline.yaml db_url -s data.csv --json-logs
```

Integrate with monitoring tools:
- Prometheus (metrics)
- Grafana (dashboards)
- ELK Stack (logs)

## Getting Help

### Where can I find examples?

- [Common Patterns](patterns.md) - Real-world examples
- [Fintech Examples](fintech_examples.md) - Industry-specific patterns
- `examples/` directory in repository

### How do I report bugs?

Open issue on GitHub with:
- AtomIngest version
- Python version
- Database type
- Minimal reproducible example
- Error message/stack trace

### How do I request features?

Open feature request on GitHub with:
- Use case description
- Expected behavior
- Example configuration (if applicable)

### Is there a community forum?

- GitHub Discussions
- Stack Overflow (tag: `atomingest`)
- Discord server (link in README)

## Best Practices

### Should I commit encryption keys?

**NO!** Never commit keys to version control.

**Good:**
```bash
# .env
ATOMINGEST_MASTER_KEY="key-from-secrets-manager"

# .gitignore
.env
```

**Bad:**
```yaml
# ❌ Don't do this!
config:
  encryption_key: "hardcoded-key-in-config"
```

### Should I version control YAML configs?

**YES!** Pipeline configs should be in version control.

```
project/
├── pipelines/
│   ├── users.yaml
│   ├── orders.yaml
│   └── products.yaml
├── .env  # ← Gitignored
├── .gitignore
└── README.md
```

### How often should I rotate encryption keys?

Depends on compliance requirements:
- **PCI-DSS:** Annually
- **HIPAA:** Annually recommended
- **GDPR:** No specific requirement

### Should I use strict or lenient validation?

**Strict:** When data quality is critical (financial data)
**Lenient:** When you need throughput and can review errors later

```yaml
# Financial transactions - use strict
config:
  validation_mode: strict

# Log ingestion - use lenient
config:
  validation_mode: lenient
  dead_letter_queue:
    enabled: true
```

## Related Documentation

- [Getting Started](getting_started.md) - Quick start guide
- [Pipeline Configuration](pipeline_configuration.md) - Complete reference
- [Troubleshooting](troubleshooting.md) - Detailed debugging guide
- [CLI Reference](cli_reference.md) - Command-line options

## Still Have Questions?

- Check the [documentation index](README.md)
- Search [GitHub Issues](https://github.com/yourrepo/atomingest/issues)
- Ask on [Stack Overflow](https://stackoverflow.com/questions/tagged/atomingest)
- Join [Discord community](https://discord.gg/atomingest)
