# Database Management with Alembic

PyCharter uses **Alembic** for database schema migrations, following the same pattern as Apache Airflow.

## Overview

- **SQLAlchemy Models**: Database schema defined as Python models in `pycharter/db/models/`
- **Alembic Migrations**: Version-controlled schema changes in `pycharter/db/migrations/`
- **CLI Commands**: `pycharter db init`, `pycharter db upgrade`, etc.

## Quick Start

### Initialize Database

```bash
# Initialize a fresh database
pycharter db init postgresql://user:pass@localhost/pycharter
```

This will:
1. Create all tables defined in SQLAlchemy models
2. Create the initial Alembic migration
3. Stamp the database with the current revision

### Upgrade Database

```bash
# Upgrade to latest version
pycharter db upgrade postgresql://user:pass@localhost/pycharter

# Or use environment variable
export PYCHARTER_DATABASE_URL=postgresql://user:pass@localhost/pycharter
pycharter db upgrade
```

### Check Current Version

```bash
pycharter db current postgresql://user:pass@localhost/pycharter
```

### View Migration History

```bash
pycharter db history
```

## Commands

| Command | Description | Example |
|---------|-------------|---------|
| `pycharter db init` | Initialize database from scratch | `pycharter db init postgresql://...` |
| `pycharter db upgrade` | Upgrade to latest revision | `pycharter db upgrade postgresql://...` |
| `pycharter db downgrade` | Downgrade to previous revision | `pycharter db downgrade postgresql://... --revision -1` |
| `pycharter db current` | Show current revision | `pycharter db current postgresql://...` |
| `pycharter db history` | Show migration history | `pycharter db history` |

## Creating New Migrations

When you modify SQLAlchemy models, create a new migration:

```bash
export PYCHARTER_DATABASE_URL=postgresql://user:pass@localhost/pycharter
alembic revision --autogenerate -m "Description of changes"
```

Then review and edit the generated migration file in `pycharter/db/migrations/versions/`.

## Database Models

All models are defined in `pycharter/db/models/`:

- `SchemaModel` - JSON Schema definitions
- `CoercionRuleModel` - Coercion rules
- `ValidationRuleModel` - Validation rules
- `MetadataRecordModel` - Comprehensive metadata storage (includes governance rules and ownership fields)
- `OwnerModel` - Direct ownership information (owner/team)
- `DataContractModel` - Central table linking all components
- `SystemModel` - System information
- `DomainModel` - Domain information

## Comparison with Airflow

| Feature | Airflow | PyCharter |
|---------|---------|-----------|
| **CLI Command** | `airflow db init` | `pycharter db init` |
| **Upgrade** | `airflow db upgrade` | `pycharter db upgrade` |
| **Migration Tool** | Alembic | Alembic |
| **Models** | SQLAlchemy | SQLAlchemy |
| **Auto-generate** | Yes | Yes |

## Best Practices

1. **Always create migrations** when changing models
2. **Test migrations** on a development database first
3. **Review autogenerated migrations** before applying
4. **Use version control** for all migration files
5. **Backup database** before running migrations in production

## Configuration

PyCharter follows Airflow's configuration pattern. Database connection can be configured via:

### 1. Environment Variables (Recommended)

```bash
export PYCHARTER_DATABASE_URL=postgresql://user:password@localhost:5432/pycharter
```

### 2. Config File

Create `pycharter.cfg` in your project root or `~/.pycharter/`:

```ini
[database]
PYCHARTER_DATABASE_URL = postgresql://user:password@localhost:5432/pycharter
```

### 3. Alembic Config

Set in `alembic.ini`:

```ini
[alembic]
sqlalchemy.url = postgresql://user:password@localhost:5432/pycharter
```

### Priority Order

1. Command-line argument (if provided)
2. `PYCHARTER_DATABASE_URL` environment variable
3. `pycharter.cfg` config file
4. `alembic.ini` config file

### Usage

Once configured, you can run commands without passing the database URL:

```bash
# Database URL from configuration
pycharter db init
pycharter db upgrade
pycharter db current

# Or override with command-line argument
pycharter db init postgresql://other:db@localhost:5432/other_db
```

## Troubleshooting

### Tables Already Exist

If tables already exist (from old migration system), stamp the database:

```bash
alembic stamp head
```

### Migration Conflicts

If you have conflicts, you can:
1. Review the migration file
2. Manually edit if needed
3. Test on a development database first

### Connection Issues

Make sure:
- Database is running
- Connection string is correct
- User has proper permissions

