# django-migrations-ci

Optimizations to run less migrations on CI.

Django migrations are slow because of state recreation for every migration and other internal Django magic.

In the past, I tried to optimize that on Django core, but learnt it's a [running issue](https://code.djangoproject.com/ticket/29898).

## Assumptions

1. I want to run my migrations on CI. It is a sanity check, even if they are generated by Django in some cases.
1. I don't have to run migrations all the time. If migrations didn't change, I can reuse them.
1. The solutions probably are CI-specific and database-specific, but it is possible to write code for a generic workflow and snippets to support each CI.

## Install

Install the package with pip:

```
pip install django-migrations-ci
```

Add `django_migrations_ci` to Django settings `INSTALLED_APPS`.

```python
INSTALLED_APPS = [
    ...,  # other packages
    "django_migrations_ci",
]
```

## How to use

The command `migrateci` execute all migrations and generate dump files `migrateci-*` to be cached on CI.

If these files already exist on disk, they are used to prepare the database without running all migrations again.

Configure your CI to cache these `migrateci-*` files, based on migration files.
/
## Workflow

This is how the "run test" CI job should work.

```
Restore migrateci.sql from CI cache

if migrateci.sql exists on cache:
  Restore migrateci.sql to test database
else:
  Setup test database
  Dump test database to migrateci.sql

Clone the test database to run threaded tests

Save migrateci.sql to CI cache
```

## Cache example on GitHub

```
    steps:
    - uses: actions/cache@v3
      name: Cache migrations
      with:
        path: migrateci-*
        key: ${{ secrets.EXAMPLE_CACHE_PREFIX }}-${{ hashFiles('**/migrations/*.py') }}
    - name: Migrate database
      run: ./manage.py migrateci --parallel $(nproc)
    - name: Test with Django
      run: ./manage.py test --keepdb --parallel $(nproc)
```

## Cache example on GitLab

Still have to abstract `psql/pg_dump/pg_restore`, but I expect something like that will work:

```
test_job:
  stage: test
  script:
    - ./manage.py migrateci $(nproc)
    - ./manage.py test --keepdb --parallel $(nproc)
  cache:
    key:
      # GitLab docs say it accepts only two files, but for some reason it works with wildcards too.
      # You can't add more than two lines here.
      files:
        - "requirements.txt"
        - "*/migrations/*.py"
    paths:
      - migrateci-*
```

## How databases for parallel tests are named

Django test framework has a `--parallel N` flag to test with N parallel processes,
naming databases from 1 to N.

* On sqlite3, a `db.sqlite3` generate `db_N.sqlite3` files.
* On PostgreSQL, a `db` generate `test_db_N`.

Pytest `pytest-django` use `pytest-xdist` for parallel support, naming databases
from 0 to N-1.

* On sqlite3, a `db.sqlite3` generate `db.sqlite3_gwN` files.
* On PostgreSQL, a `db` generate `test_db_gwN`.
