# Database Migration Pattern

**Pattern Type**: Database Migration
**Confidence**: 0.90
**Expected Complexity**: 5.5-7.0

## Pattern Characteristics

This directive matches the **Database Migration** pattern because it involves:

1. **Schema changes** (new column: `deleted_at`)
2. **Data migration** (backfill existing records)
3. **Query updates** (filter soft-deleted records)
4. **Application logic changes** (soft delete instead of hard delete)
5. **Data retention policy** (90-day retention before permanent deletion)

## Expected Systems

- **Database**: Schema migration (add `deleted_at` column)
- **API Layer**: Update delete endpoints, query filters
- **Admin UI**: Interface for permanent deletion
- **Background Jobs**: Cleanup job for 90-day retention enforcement

## Expected Modifiers

- **Data Migration**: High (backfill required)
- **Breaking Changes**: Medium (API behavior changes, but backwards compatible)
- **State Coordination**: Low (queries need consistent filtering)
- **Performance**: Medium (query performance with soft delete filters)

## Recommended Splitting

### Ternary Split (3 children)

**Recommended for clean separation of migration stages**:

```
Parent: Soft Delete Implementation
├── Child 0: Database - Schema & Migration
│   ├── Create migration file
│   ├── Add deleted_at column (nullable timestamp)
│   ├── Backfill deleted_at=NULL
│   └── Add indexes for performance
├── Child 1: Application - Query Updates
│   ├── Update all SELECT queries (filter deleted_at IS NULL)
│   ├── Update DELETE to SET deleted_at=NOW()
│   ├── Update counts/aggregations
│   └── Integration tests for soft delete behavior
└── Child 2: Admin - Permanent Deletion
    ├── Admin interface for viewing soft-deleted users
    ├── Permanent delete action (hard delete after verification)
    ├── Background job for 90-day cleanup
    └── Audit logging for permanent deletions
```

## Key Considerations

### Migration Safety
- **Reversibility**: Migration must be reversible (down migration drops column)
- **Zero Downtime**: Add column with default NULL, no locks
- **Backfill Strategy**: Run after column creation, not during
- **Index Strategy**: Add index on `deleted_at` for query performance

### Query Updates
- **WHERE Clauses**: All queries need `WHERE deleted_at IS NULL`
- **Model Scoping**: Use ORM default scope to avoid manual filtering
- **Raw SQL**: Audit and update raw queries manually
- **Aggregations**: COUNT, SUM must exclude soft-deleted records

### Performance Impact
- **Index Required**: `CREATE INDEX idx_users_deleted_at ON users(deleted_at) WHERE deleted_at IS NULL`
- **Query Plans**: Verify explain plans before/after migration
- **Soft Delete Overhead**: Minimal with proper indexing

### Data Retention
- **90-Day Window**: Background job runs daily, deletes records where `deleted_at < NOW() - INTERVAL '90 days'`
- **Audit Trail**: Log permanent deletions for compliance
- **Admin Override**: Allow manual permanent deletion before 90 days (with authorization)

## Expected Conflicts

### Likely Conflicts
- **File Overlap**: Migration file, model files, query files all touch different systems
- **Interface Mismatch**: Delete method signature (soft delete vs hard delete)

### Unlikely Conflicts
- **Decision Contradiction**: Clear pattern, well-established soft delete practice
- **Assumption Violation**: All children understand soft delete semantics

## Verification Commands

```bash
# Run migration
python manage.py migrate

# Verify schema
psql -d mydb -c "\d users"  # Check for deleted_at column

# Test soft delete
curl -X DELETE http://localhost:8000/api/users/123

# Verify record marked as deleted
psql -d mydb -c "SELECT id, deleted_at FROM users WHERE id=123"

# Verify record excluded from queries
curl http://localhost:8000/api/users  # Should not include user 123

# Test permanent deletion (admin interface)
curl -X POST http://localhost:8000/admin/users/123/permanent-delete \
  -H "Authorization: Bearer <admin-token>"

# Run integration tests
pytest tests/test_soft_delete.py -v
```

## Success Criteria

1. Migration adds `deleted_at` column without errors
2. Backfill completes (all active users have `deleted_at = NULL`)
3. Delete operations set `deleted_at` instead of removing records
4. All queries exclude soft-deleted records
5. Admin interface shows soft-deleted users
6. Permanent deletion works (hard delete after verification)
7. Background job runs daily, enforces 90-day retention
8. Query performance unchanged (indexes present)

## Common Pitfalls

- **Forgetting to filter**: Easy to miss raw SQL queries or aggregations
- **Index performance**: Without index on `deleted_at`, queries degrade
- **Cascade deletes**: Related tables may still hard delete (need soft delete cascade)
- **Test data cleanup**: Test suites may assume hard deletes (update teardown logic)
