# SM008: data_migration

| Property       | Value          |
| -------------- | -------------- |
| **Rule ID**    | SM008          |
| **Severity**   | INFO           |
| **Name**       | data_migration |
| **Applies to** | All databases  |

## What Triggers This Rule

This rule triggers when a migration contains `RunPython`:

```python
# Triggers SM008
def populate_default_data(apps, schema_editor):
    Category = apps.get_model('shop', 'Category')
    Category.objects.create(name='Default')

migrations.RunPython(populate_default_data)
```

## Why It's Flagged

Data migrations (RunPython operations) have several considerations:

1. **Performance impact** — May process large amounts of data
2. **Transaction scope** — Runs in a single transaction by default
3. **Irreversibility** — May not be easily reversible
4. **Memory usage** — Loading all objects can exhaust memory
5. **Lock duration** — Long-running operations hold locks

This is an INFO-level rule to bring attention to data migrations for review, not necessarily to flag a problem.

## When It's Safe to Ignore

This rule can be safely ignored when:

1. **Small data sets** — Only a few rows to process
2. **Idempotent operations** — Safe to run multiple times
3. **Quick operations** — Completes in seconds
4. **Proper batching** — Large operations are batched

## Best Practices

### Always Provide Reverse Code

```python
def forward(apps, schema_editor):
    User = apps.get_model('myapp', 'User')
    User.objects.filter(status='').update(status='active')

def reverse(apps, schema_editor):
    User = apps.get_model('myapp', 'User')
    User.objects.filter(status='active').update(status='')

migrations.RunPython(forward, reverse)
```

Or explicitly mark as non-reversible:

```python
migrations.RunPython(forward, migrations.RunPython.noop)
```

### Batch Large Operations

```python
def backfill_in_batches(apps, schema_editor):
    User = apps.get_model('myapp', 'User')
    batch_size = 1000

    while True:
        # Get batch of IDs to update
        ids = list(
            User.objects.filter(new_field__isnull=True)
            .values_list('id', flat=True)[:batch_size]
        )

        if not ids:
            break

        # Update in batches
        User.objects.filter(id__in=ids).update(new_field='default')

migrations.RunPython(backfill_in_batches, migrations.RunPython.noop)
```

### Use iterator() for Large Querysets

```python
def process_users(apps, schema_editor):
    User = apps.get_model('myapp', 'User')

    # ✅ Memory efficient
    for user in User.objects.all().iterator():
        user.full_name = f"{user.first_name} {user.last_name}"
        user.save(update_fields=['full_name'])
```

### Consider Running Outside Migrations

For very large operations, consider:

1. **Management command** — Run separately from deployment
2. **Background job** — Process asynchronously
3. **Direct SQL** — More efficient for simple updates

```python
# Instead of RunPython, use RunSQL for simple updates
migrations.RunSQL(
    sql="UPDATE myapp_user SET status = 'active' WHERE status = '';",
    reverse_sql="UPDATE myapp_user SET status = '' WHERE status = 'active';",
)
```

### Avoid Importing Models Directly

```python
# ❌ Wrong - uses current model definition
from myapp.models import User

def migrate_data(apps, schema_editor):
    User.objects.all().update(...)  # May fail if model changed

# ✅ Correct - uses historical model
def migrate_data(apps, schema_editor):
    User = apps.get_model('myapp', 'User')
    User.objects.all().update(...)
```

## How to Suppress

### In Django Settings

```python
SAFE_MIGRATIONS = {
    "DISABLED_RULES": ["SM008"],
}
```

### Change Severity

If you find this too noisy:

```python
SAFE_MIGRATIONS = {
    "RULE_SEVERITY": {
        "SM008": "INFO",  # Keep as INFO (default)
    },
}
```

## See Also

- [SM007: run_sql_no_reverse](SM007.md) — Related RunSQL rule
- [SM016: run_python_no_reverse](SM016.md) — Reversibility-specific rule
- [Safe Patterns: RunPython Best Practices](../patterns.md#runpython-best-practices)
