# Security Features Guide

Complete guide to PII encryption, audit trails, and compliance features in AtomIngest.

## Table of Contents
- [Overview](#overview)
- [Encryption](#encryption)
- [Key Management](#key-management)
- [Audit Trails](#audit-trails)
- [Crypto-Shredding](#crypto-shredding)
- [Compliance](#compliance)
- [Best Practices](#best-practices)

## Overview

AtomIngest provides enterprise-grade security features for protecting sensitive data and maintaining compliance with regulations like GDPR, HIPAA, and PCI-DSS.

**Security Features:**
- ✅ Field-level encryption for PII
- ✅ Key management system
- ✅ Immutable audit trails
- ✅ Crypto-shredding (right to be forgotten)
- ✅ Access logging
- ✅ Data lineage tracking

## Encryption

### Encrypted Field Types

AtomIngest provides encrypted versions of standard field types:

```yaml
schema:
  # Encrypted string field
  ssn:
    type: EncryptedStringField
    key_id: "pii_encryption_key"

  # Encrypted char field with max length
  credit_card:
    type: EncryptedCharField
    max_length: 255
    key_id: "payment_encryption_key"

  # Encrypted text field
  medical_notes:
    type: EncryptedTextField
    key_id: "medical_data_key"

  # Encrypted email
  private_email:
    type: EncryptedEmailField
    key_id: "email_encryption_key"

  # Encrypted JSON
  sensitive_metadata:
    type: EncryptedJSONField
    key_id: "metadata_encryption_key"
```

### How Encryption Works

1. **Master Key**: Single key for wrapping data keys (from environment)
2. **Data Keys**: Per-identifier keys for encrypting data
3. **Envelope Encryption**: Data keys encrypted by master key
4. **Field-Level**: Each field encrypted individually

```
┌─────────────────┐
│   Master Key    │ (from environment)
└────────┬────────┘
         │ wraps
         ▼
┌─────────────────┐
│   Data Key      │ (in database)
└────────┬────────┘
         │ encrypts
         ▼
┌─────────────────┐
│  Encrypted Data │
└─────────────────┘
```

### Setup Encryption

**1. Generate Master Key:**

```bash
python -c "from cryptography.fernet import Fernet; print(Fernet.generate_key().decode())"
```

**2. Set Environment Variable:**

```bash
export ATOMINGEST_MASTER_KEY="your-generated-key-here"
```

**3. Define Encrypted Fields:**

```yaml
schema:
  ssn:
    type: EncryptedStringField
    key_id: "user_pii"

  credit_card:
    type: EncryptedCharField
    max_length: 255
    key_id: "payment_data"
```

**4. Run Ingestion:**

```bash
atomingest run pipeline.yaml sqlite:///secure.db -s data.csv
```

### Encryption Example

**Input CSV:**
```csv
name,ssn,credit_card
Alice,123-45-6789,4532-1234-5678-9010
Bob,987-65-4321,5425-2345-6789-0123
```

**Stored in Database:**
```
name: "Alice"
ssn: "gAAAAABhkO5X..." (encrypted)
credit_card: "gAAAAABhkO5Y..." (encrypted)
```

**Retrieved (decrypted automatically):**
```python
user = User.objects().get(name="Alice")
print(user.ssn)  # "123-45-6789" (decrypted)
```

## Key Management

### KeyManager

The `KeyManager` class handles all encryption key operations.

### Create Key

```python
from atomingest.security.kms import KeyManager

# Create new data key
key = KeyManager.create_key("customer_pii")
```

### Retrieve Key

```python
# Get existing key (or create if doesn't exist)
key = KeyManager.get_key("customer_pii")
```

### Key Storage

Keys are stored in `atom_keystore` table:

```sql
CREATE TABLE atom_keystore (
  id INTEGER PRIMARY KEY,
  key_identifier VARCHAR(255) UNIQUE,
  wrapped_key TEXT,
  created_at DATETIME
);
```

**Example:**
```
key_identifier: "customer_pii"
wrapped_key: "gAAAAABhkO5X..." (encrypted with master key)
created_at: 2026-01-20 10:30:00
```

### Key Rotation

```python
# Manual key rotation
old_key = KeyManager.get_key("customer_pii")

# Create new key with different identifier
new_key = KeyManager.create_key("customer_pii_v2")

# Re-encrypt data with new key
# (requires custom migration script)
```

## Audit Trails

### Immutable Audit Metadata

Automatically track data lineage:

```yaml
config:
  audit_metadata:
    enabled: true
    fields:
      source_file: true
      run_id: true
      ingestion_timestamp: true
```

**Adds to Schema:**
```yaml
schema:
  # Your fields
  name: CharField
  email: EmailField

  # Auto-added audit fields
  source_file: CharField
  run_id: CharField
  ingestion_timestamp: DateTimeField
```

**Stored Data:**
```
name: "Alice"
email: "alice@example.com"
source_file: "users_20260120.csv"
run_id: "run_20260120_103000_a1b2c3"
ingestion_timestamp: "2026-01-20T10:30:00Z"
```

### ImmutableLedgerMixin

Add immutable audit trail to any model:

```python
from atomingest.core.mixins import ImmutableLedgerMixin

class AuditedUser(Model, ImmutableLedgerMixin):
    _table_name = "audited_users"

    name = fields.CharField(max_length=255)
    email = fields.EmailField()
```

**Auto-added Fields:**
- `created_at`: Record creation timestamp
- `created_by`: User who created record
- `source_system`: System that created record
- `audit_hash`: Hash for tamper detection

### Query Audit Trail

```python
# Get all records from specific source file
users = User.objects().filter(source_file="users_20260120.csv")

# Get all records from specific run
users = User.objects().filter(run_id="run_20260120_103000_a1b2c3")

# Get records ingested today
from datetime import datetime, timedelta
today = datetime.now() - timedelta(days=1)
users = User.objects().filter(ingestion_timestamp__gte=today)
```

## Crypto-Shredding

GDPR "Right to be Forgotten" compliance through key destruction.

### How It Works

1. Data encrypted with per-user key
2. Delete encryption key from keystore
3. Data becomes permanently unreadable
4. No need to delete actual records

### Setup

```yaml
schema:
  user_id: UUIDField
  ssn:
    type: EncryptedStringField
    key_id: "user_{user_id}"  # Per-user key

  credit_card:
    type: EncryptedCharField
    key_id: "user_{user_id}"
```

### Shred User Data

```python
from atomingest.security.kms import KeyManager

# User requests data deletion
user_id = "550e8400-e29b-41d4-a716-446655440000"

# Shred their encryption key
KeyManager.shred_key(f"user_{user_id}")

# Data is now permanently unreadable
# Even though records still exist in database
```

**Before Shredding:**
```python
user = User.objects().get(id=user_id)
print(user.ssn)  # "123-45-6789"
```

**After Shredding:**
```python
user = User.objects().get(id=user_id)
print(user.ssn)  # DecryptionError: Key not found
```

### Bulk Shredding

```python
# Shred multiple users
user_ids = ["uuid1", "uuid2", "uuid3"]

for user_id in user_ids:
    try:
        KeyManager.shred_key(f"user_{user_id}")
        print(f"Shredded data for user {user_id}")
    except Exception as e:
        print(f"Failed to shred {user_id}: {e}")
```

### Verify Shredding

```python
from cryptography.fernet import Fernet, InvalidToken

user = User.objects().get(id=user_id)

try:
    # Try to access encrypted field
    ssn = user.ssn
    print("WARNING: Data still readable!")
except InvalidToken:
    print("Success: Data crypto-shredded")
```

## Compliance

### GDPR Compliance

**Data Protection:**
- ✅ Field-level encryption for personal data
- ✅ Right to be forgotten (crypto-shredding)
- ✅ Data lineage tracking
- ✅ Audit trails

**Implementation:**
```yaml
schema:
  # PII fields encrypted
  name:
    type: EncryptedCharField
    max_length: 255
    key_id: "user_{user_id}"

  email:
    type: EncryptedEmailField
    key_id: "user_{user_id}"

  # Audit metadata
  source_file: CharField
  ingestion_timestamp: DateTimeField
```

### HIPAA Compliance

**Protected Health Information (PHI):**
```yaml
schema:
  patient_id: UUIDField

  # Encrypt all PHI
  medical_record_number:
    type: EncryptedStringField
    key_id: "patient_{patient_id}"

  diagnosis:
    type: EncryptedTextField
    key_id: "patient_{patient_id}"

  medication:
    type: EncryptedTextField
    key_id: "patient_{patient_id}"

  # Audit trail
  accessed_by: CharField
  access_timestamp: DateTimeField
```

### PCI-DSS Compliance

**Payment Card Data:**
```yaml
schema:
  # Encrypt cardholder data
  card_number:
    type: EncryptedCharField
    max_length: 255
    key_id: "payment_encryption"
    validation:
      - type: luhn

  cvv:
    type: EncryptedCharField
    max_length: 255
    key_id: "payment_encryption"

  # Do not store CVV after authorization
  # (implement in hooks)
```

### SOC 2 Compliance

**Access Controls and Logging:**
```yaml
config:
  audit_metadata:
    enabled: true

  access_logging:
    enabled: true
    log_reads: true
    log_writes: true

hooks:
  pre_save:
    - name: "access_log"
      class: "myapp.hooks.AccessLogHook"
      config:
        log_user: true
        log_ip: true
```

## Best Practices

### 1. Encrypt All PII

```yaml
# Good - all PII encrypted
schema:
  ssn:
    type: EncryptedStringField
    key_id: "user_pii"

  credit_card:
    type: EncryptedCharField
    key_id: "payment_data"

# Bad - PII unencrypted
schema:
  ssn: CharField
  credit_card: CharField
```

### 2. Use Per-Entity Keys for Crypto-Shredding

```yaml
# Good - per-user keys enable selective shredding
schema:
  ssn:
    type: EncryptedStringField
    key_id: "user_{user_id}"

# Less flexible - single key for all users
schema:
  ssn:
    type: EncryptedStringField
    key_id: "pii_encryption"
```

### 3. Rotate Keys Regularly

```bash
# Automated key rotation script
#!/bin/bash

# Generate new master key
NEW_KEY=$(python -c "from cryptography.fernet import Fernet; print(Fernet.generate_key().decode())")

# Re-encrypt data keys with new master key
# (implement re-encryption logic)

# Update environment
export ATOMINGEST_MASTER_KEY="$NEW_KEY"
```

### 4. Secure Master Key Storage

```bash
# BAD - key in code
ATOMINGEST_MASTER_KEY="hardcoded-key"

# GOOD - key in environment
export ATOMINGEST_MASTER_KEY="$(cat /secure/path/master.key)"

# BETTER - key from secrets manager
export ATOMINGEST_MASTER_KEY="$(aws secretsmanager get-secret-value --secret-id master-key --query SecretString --output text)"
```

### 5. Enable Audit Trails

```yaml
config:
  audit_metadata:
    enabled: true
    fields:
      source_file: true
      run_id: true
      ingestion_timestamp: true
      ingested_by: true
```

### 6. Validate Encryption

```python
# Test encryption in development
import unittest
from atomingest.security.kms import KeyManager

class TestEncryption(unittest.TestCase):
    def test_field_encrypted(self):
        user = User.create(
            name="Test",
            ssn="123-45-6789"
        )

        # Get raw value from database
        raw_ssn = db.execute(
            "SELECT ssn FROM users WHERE id=?",
            [user.id]
        ).fetchone()[0]

        # Verify it's encrypted
        self.assertNotEqual(raw_ssn, "123-45-6789")
        self.assertTrue(raw_ssn.startswith("gAAAAA"))
```

### 7. Document Key Identifiers

```yaml
# keys.yaml - Document all key identifiers
keys:
  user_pii:
    description: "User PII (SSN, address, etc.)"
    rotation_policy: "Annual"
    data_class: "Sensitive"

  payment_data:
    description: "Payment card information"
    rotation_policy: "Quarterly"
    data_class: "Highly Sensitive"

  medical_data:
    description: "Protected health information"
    rotation_policy: "Biannual"
    data_class: "Protected"
```

## Complete Security Example

### Secure Pipeline Configuration

```yaml
target_table: secure_users

schema:
  # Non-sensitive fields
  user_id:
    type: UUIDField
    unique: true
    nullable: false

  username:
    type: CharField
    max_length: 50
    unique: true

  # Encrypted PII
  full_name:
    type: EncryptedCharField
    max_length: 255
    key_id: "user_{user_id}"

  email:
    type: EncryptedEmailField
    key_id: "user_{user_id}"

  ssn:
    type: EncryptedStringField
    key_id: "user_{user_id}"

  phone:
    type: EncryptedCharField
    max_length: 255
    key_id: "user_{user_id}"

  address:
    type: EncryptedTextField
    key_id: "user_{user_id}"

  # Audit metadata
  source_file: CharField
  run_id: CharField
  ingestion_timestamp: DateTimeField
  ingested_by: CharField

config:
  audit_metadata:
    enabled: true

hooks:
  pre_save:
    - name: "audit_log"
      class: "myapp.hooks.AuditLogHook"

  post_save:
    - name: "compliance_check"
      class: "myapp.hooks.ComplianceCheckHook"
```

## Related Documentation

- [Schema Definition](schema_definition.md) - Define encrypted fields
- [Hooks](hooks.md) - Implement audit logging
- [Error Handling](error_handling.md) - Handle encryption errors

## Next Steps

1. Generate master encryption key
2. Set ATOMINGEST_MASTER_KEY environment variable
3. Define encrypted fields in schema
4. Enable audit metadata
5. Implement crypto-shredding for GDPR compliance
6. Test encryption and key rotation
