# LLMHosts.com Database Backup & Restore

Automated backup system for SQLite database with integrity verification, retention policies, and multiple storage backends.

## Features

- **Atomic backups** using SQLite's `.backup` command (consistent point-in-time)
- **Integrity verification** before and after backup
- **Compression** (gzip level 9)
- **Checksums** (SHA-256 for tamper detection)
- **Multiple storage backends** (local, S3, Backblaze B2)
- **Retention policies** (automatic cleanup of old backups)
- **Restore verification** (test restore without overwriting production)
- **Prometheus metrics** (backup success timestamps)

## Quick Start

### Daily Automated Backups (Recommended)

```bash
# Add to crontab (runs daily at 2 AM)
0 2 * * * /home/dev/projects/LLMHosts.com/infrastructure/backup/backup.sh --storage s3 --bucket llmhosts-backups --retention-days 30

# Or use systemd timer (see below)
```

### Manual Backup

```bash
# Local backup
./backup.sh

# S3 backup
export AWS_ACCESS_KEY_ID="your-key"
export AWS_SECRET_ACCESS_KEY="your-secret"
./backup.sh --storage s3 --bucket llmhosts-backups

# Backblaze B2
export B2_APPLICATION_KEY_ID="your-key-id"
export B2_APPLICATION_KEY="your-key"
./backup.sh --storage b2 --bucket llmhosts-backups
```

### Restore from Backup

```bash
# Verify backup integrity (no restore)
./restore.sh /backups/llmhosts_20260218_120000.db.gz --verify-only

# Restore from local backup
./restore.sh /backups/llmhosts_20260218_120000.db.gz

# Restore from S3
export AWS_ACCESS_KEY_ID="your-key"
export AWS_SECRET_ACCESS_KEY="your-secret"
./restore.sh s3://llmhosts-backups/llmhosts_20260218_120000.db.gz

# Restore from B2
export B2_APPLICATION_KEY_ID="your-key-id"
export B2_APPLICATION_KEY="your-key"
./restore.sh b2://llmhosts-backups/llmhosts_20260218_120000.db.gz
```

## Storage Backends

### Local (Default)

Stores backups in `/backups/` directory on the same server.

**Pros**: Simple, fast, no external dependencies
**Cons**: No off-site redundancy, lost if server fails

**Use case**: Development, testing, temporary backups

```bash
./backup.sh --storage local --retention-days 7
```

### AWS S3

Stores backups in Amazon S3 bucket.

**Pros**: Off-site, durable (99.999999999%), scalable
**Cons**: Requires AWS account, costs $0.023/GB/month

**Use case**: Production (recommended)

**Setup**:
```bash
# Install AWS CLI
pip install awscli

# Configure credentials
aws configure

# Create bucket
aws s3 mb s3://llmhosts-backups

# Run backup
./backup.sh --storage s3 --bucket llmhosts-backups
```

### Backblaze B2

Stores backups in Backblaze B2 bucket.

**Pros**: Off-site, cheaper than S3 ($0.005/GB/month), durable
**Cons**: Requires B2 account

**Use case**: Cost-sensitive production

**Setup**:
```bash
# Install B2 CLI
pip install b2

# Authorize
b2 authorize-account <key-id> <application-key>

# Create bucket
b2 create-bucket llmhosts-backups allPrivate

# Run backup
./backup.sh --storage b2 --bucket llmhosts-backups
```

## Retention Policies

| Backup Type | Retention | Storage | Purpose |
|-------------|-----------|---------|---------|
| Daily | 30 days | S3/B2 | Recent recovery |
| Weekly | 12 weeks | S3/B2 | Short-term archive |
| Monthly | 12 months | S3/B2 Glacier | Long-term archive |

**Current implementation**: Daily backups with 30-day retention (configurable via `--retention-days`)

**Future enhancement**: Separate policies for weekly/monthly (see roadmap)

## Backup Schedule

Recommended cron schedule:

```bash
# Daily at 2 AM (low traffic)
0 2 * * * /path/to/backup.sh --storage s3 --bucket llmhosts-backups

# Weekly on Sunday at 3 AM
0 3 * * 0 /path/to/backup.sh --storage s3 --bucket llmhosts-backups-weekly --retention-days 84

# Monthly on 1st at 4 AM
0 4 1 * * /path/to/backup.sh --storage s3 --bucket llmhosts-backups-monthly --retention-days 365
```

## Systemd Timer (Alternative to Cron)

Create `/etc/systemd/system/llmhosts-backup.service`:

```ini
[Unit]
Description=LLMHosts Database Backup
After=network.target

[Service]
Type=oneshot
User=root
ExecStart=/home/dev/projects/LLMHosts.com/infrastructure/backup/backup.sh --storage s3 --bucket llmhosts-backups
Environment="DB_PATH=/data/llmhosts.db"
Environment="AWS_ACCESS_KEY_ID=your-key"
Environment="AWS_SECRET_ACCESS_KEY=your-secret"
```

Create `/etc/systemd/system/llmhosts-backup.timer`:

```ini
[Unit]
Description=Daily LLMHosts Backup
Requires=llmhosts-backup.service

[Timer]
OnCalendar=daily
OnCalendar=*-*-* 02:00:00
Persistent=true

[Install]
WantedBy=timers.target
```

Enable and start:

```bash
sudo systemctl enable llmhosts-backup.timer
sudo systemctl start llmhosts-backup.timer
sudo systemctl list-timers llmhosts-backup.timer
```

## Disaster Recovery

### Scenario 1: Accidental Data Deletion

User accidentally deleted their API keys.

**Recovery**:
```bash
# Find most recent backup
ls -lt /backups/ | head -5

# Verify backup
./restore.sh /backups/llmhosts_20260218_120000.db.gz --verify-only

# Restore
./restore.sh /backups/llmhosts_20260218_120000.db.gz
```

**Time to recovery**: < 5 minutes

### Scenario 2: Database Corruption

SQLite database corrupted after disk failure.

**Recovery**:
```bash
# Download most recent S3 backup
./restore.sh s3://llmhosts-backups/llmhosts_20260218_120000.db.gz
```

**Time to recovery**: < 10 minutes (including download)

### Scenario 3: Complete Server Loss

Server completely destroyed, need to rebuild.

**Recovery**:
```bash
# On new server
cd /home/dev/projects/LLMHosts.com
git pull

# List available backups
aws s3 ls s3://llmhosts-backups/ | grep llmhost_

# Restore most recent
./infrastructure/backup/restore.sh s3://llmhosts-backups/llmhosts_20260218_120000.db.gz
```

**Time to recovery**: < 30 minutes (including server setup)

## Testing Backups

**CRITICAL**: Untested backups are not backups!

### Monthly Verification Test

```bash
# 1. Create test restore
DB_PATH=/tmp/test-restore.db ./restore.sh /backups/llmhosts_latest.db.gz

# 2. Verify record counts match
sqlite3 /tmp/test-restore.db "SELECT COUNT(*) FROM users;"
sqlite3 /data/llmhosts.db "SELECT COUNT(*) FROM users;"

# 3. Verify schema matches
sqlite3 /tmp/test-restore.db ".schema" > /tmp/schema-backup.sql
sqlite3 /data/llmhosts.db ".schema" > /tmp/schema-current.sql
diff /tmp/schema-backup.sql /tmp/schema-current.sql

# 4. Clean up
rm /tmp/test-restore.db
```

Add to calendar: Test backups monthly!

## Monitoring

### Prometheus Metrics

If `METRICS_DIR` is set, backup script exports metrics:

```prometheus
# /var/lib/prometheus/textfile_collector/backup.prom
llmhosts_backup_last_success_timestamp 1708267200
llmhosts_backup_size_bytes 1048576
```

Alert on stale backups:

```yaml
- alert: BackupStale
  expr: time() - llmhosts_backup_last_success_timestamp > 86400
  for: 1h
  annotations:
    summary: "Backup not run in 24+ hours"
```

### Backup Log

All backups logged to syslog:

```bash
# View backup logs
journalctl -t backup.sh -n 100

# Watch in real-time
journalctl -t backup.sh -f
```

## Troubleshooting

### "Database is locked"

**Cause**: Backup running while SQLite is writing

**Solution**: Use `.backup` command (already implemented, handles this)

### "Checksum mismatch"

**Cause**: Backup corrupted during download or storage

**Solution**: Try downloading again, or use previous backup

### "Schema version mismatch"

**Cause**: Backup from older version of app

**Solution**: Run Alembic migrations after restore

```bash
# Restore old backup
./restore.sh /backups/llmhosts_old.db.gz

# Migrate to current schema
cd /home/dev/projects/LLMHosts.com
alembic upgrade head
```

## Security Considerations

- **Encryption at rest**: S3/B2 support server-side encryption (enable!)
- **Access control**: Backup bucket should have restricted IAM policies
- **Audit trail**: Enable S3 CloudTrail or B2 audit logs
- **Secrets**: Never commit AWS/B2 credentials to git
- **Retention**: Don't keep backups forever (GDPR compliance)

## Cost Analysis

### S3 Standard

- Storage: $0.023/GB/month
- Retrieval: $0.09/GB (rare)

**Example** (100 MB database, 30-day retention):
- Daily: 100 MB × 30 = 3 GB = $0.07/month
- Weekly: 100 MB × 12 = 1.2 GB = $0.03/month
- Monthly: 100 MB × 12 = 1.2 GB = $0.03/month

**Total**: ~$0.13/month (~$1.56/year)

### Backblaze B2

- Storage: $0.005/GB/month
- Retrieval: Free up to 3× average storage

**Example** (same 100 MB database):
- Daily: 3 GB = $0.015/month
- Weekly: 1.2 GB = $0.006/month
- Monthly: 1.2 GB = $0.006/month

**Total**: ~$0.027/month (~$0.32/year)

**Recommendation**: B2 for cost-sensitive, S3 for simplicity

## Roadmap

- [ ] Implement weekly/monthly separate retention (Issue #TBD)
- [ ] Add backup rotation to S3 Glacier (cold storage)
- [ ] Encrypted backups (GPG before upload)
- [ ] Backup verification bot (automated monthly test)
- [ ] Multi-region replication (S3 cross-region)
- [ ] Backup dashboard in Grafana

## References

- SQLite Backup API: https://www.sqlite.org/backup.html
- Aut-O Tenet T-04: Cloud-Native Portable
- AWS S3 Pricing: https://aws.amazon.com/s3/pricing/
- Backblaze B2 Pricing: https://www.backblaze.com/b2/cloud-storage-pricing.html

---

**Last updated**: 2026-02-18
**Maintainer**: LLMHosts Team
