# Rust PostgreSQL Driver - Implementation Summary

**Document Version**: 1.0
**Created**: 2025-12-18
**Branch**: `feature/rust-postgres-driver`
**Status**: Ready for Phase 1

**Last Updated**: 2025-12-18 (IMPROVED - All critical sections added)

---

## Quick Start

This directory contains a complete 5-phase implementation plan for migrating FraiseQL's database layer from psycopg (Python) to a native Rust backend using `tokio-postgres` and `deadpool-postgres`.

### For Quick Reference:
1. **README.md** - Start here (overview, architecture, timeline)
2. **phase-1-foundation.md** - Week 1 work (connection pool)
3. **phase-2-query-execution.md** - Week 1-2 work (WHERE clauses, SQL)
4. **phase-3-result-streaming.md** - Week 2 work (zero-copy streaming)
5. **phase-4-integration.md** - Week 2-3 work (full GraphQL pipeline)
6. **phase-5-deprecation.md** - Week 3 work (remove psycopg)

---

## Critical Implementation Notes (MUST READ)

### Async/PyO3 Bridge (Most Complex)

The biggest challenge is bridging Python's asyncio with Rust's tokio via PyO3. Key points:

1. **Use `pyo3-asyncio::tokio::future_into_py()`** to return Python coroutines from Rust
2. **Never mix runtimes** - tokio tasks can't call Python directly
3. **Connection pool must be Arc-wrapped and created ONCE** at startup
4. **Type conversions** between Python, Rust, and PostgreSQL are error-prone (see detailed type conversion guide in README)

**Typical pattern**:
```rust
#[pyo3_asyncio::tokio::main]
async fn rust_async_function(py: Python) -> PyResult<Py<PyAny>> {
    pyo3_asyncio::tokio::future_into_py(py, async {
        // Actual async work here
        // Returns Result<T, PyErr>
    })
}
```

### WHERE Clause & Filter Logic

- **Fully recursive** - supports nested AND/OR/NOT
- **Type-aware** - must handle all PostgreSQL types (especially JSONB)
- **Parity critical** - must match exact output of existing Python `graphql_where_generator.py`

### Connection Pool Lifecycle

- **Created once** - Pool initialization is expensive
- **Lazy connection creation** - First connection to DB happens on first query
- **Stale connection detection** - Use `test_on_checkout` to validate before use
- **Timeout handling** - Distinguish between connection timeout vs query timeout

---

## Why This Matters

### Current Problems
- Database operations go through Python layer (psycopg)
- Results marshalled to Rust pipeline
- Two language boundaries = overhead
- Connection pool managed in Python async runtime

### New Architecture
- **Python**: GraphQL framework, validation, schema introspection (stays same)
- **Rust**: All database operations (connection pool, queries, mutations, response building)
- **Benefits**: 20-30% faster queries, zero-copy streaming, true async, type-safe

### The Key Insight
FraiseQL's Rust JSON transformation pipeline (7-10x faster than Python) is proven effective. This plan extends that to the entire database layer, resulting in a **fully Rust-powered core** with a clean Python API.

---

## Architecture Summary

```
BEFORE:
  User Python Code
      ↓
  FastAPI (Python)
      ↓
  psycopg (Python) → PostgreSQL
      ↓
  Results (dicts/rows)
      ↓
  Rust Pipeline (JSON transform)
      ↓
  HTTP Response

AFTER:
  User Python Code
      ↓
  FastAPI (Python)
      ↓
  Python validates, parses GraphQL
      ↓
  Single async call → Rust
      ↓
  Rust Core (complete database → response pipeline)
      ├─ Connection pool (deadpool)
      ├─ Query execution (tokio-postgres)
      ├─ WHERE clause building
      ├─ SQL generation
      ├─ Result streaming (zero-copy)
      ├─ JSON transformation
      └─ Response building
      ↓
  HTTP Response
```

---

## Timeline

| Phase | Name | Effort | Start | Key Deliverable |
|-------|------|--------|-------|-----------------|
| 1 | Foundation | 8h | Day 1 | Connection pool + schema registry |
| 2 | Query Execution | 12h | Day 2-3 | WHERE clauses + SQL generation |
| 3 | Result Streaming | 10h | Day 4-5 | Zero-copy optimization |
| 4 | Integration | 8h | Day 5-6 | Full GraphQL pipeline |
| 5 | Deprecation | 6h | Day 6-7 | Remove psycopg, finalize |

**Total**: 44 hours (~1 week full-time)

---

## Key Decisions

### 1. Driver Choice: tokio-postgres

Why **not** sqlx or diesel?
- **sqlx**: Requires compile-time query validation (incompatible with dynamic schemas)
- **diesel**: Sync-only (no async support) and also requires compile-time validation
- **tokio-postgres**: Perfect for dynamic schemas, true async, zero-copy result access

### 2. Pooling: deadpool-postgres

- Production-ready, async-first
- Configurable with same options as psycopg
- Easy integration with tokio runtime

### 3. Build System: Existing PyO3/Maturin

- Already proven to work in FraiseQL
- No new infrastructure needed
- Familiar to team

### 4. Python API: Unchanged

- Users never know what changed
- 100% backward compatible
- Gradual transition possible via feature flags

---

## Risk Mitigation

| Risk | Probability | Impact | Mitigation |
|------|-------------|--------|-----------|
| Async complexity | Low | High | Use well-tested libraries, extensive testing |
| Performance regression | Very Low | High | Continuous benchmarking, parity tests |
| Compatibility issues | Low | Medium | Feature flags, comprehensive tests, easy rollback |
| Connection pool issues | Low | High | Pool stress testing, load tests |
| Build system breakage | Very Low | Medium | Incremental build verification |

### Rollback Strategy

If critical issues found:
```bash
# Immediate fallback
git revert <problematic-commit>
cargo build
uv run pytest tests/

# Back to working state in < 5 minutes
```

---

## Testing Strategy

### Phase 1: Foundation
- Unit tests for pool configuration
- Integration tests for pool initialization
- Backward compatibility verification

### Phase 2: Query Execution
- WHERE clause unit tests (parity with Python)
- SQL generation tests
- Query execution tests
- Parity tests (Rust results == psycopg results)

### Phase 3: Result Streaming
- Streaming performance tests
- Memory profiling
- Large result set handling

### Phase 4: Integration
- End-to-end query tests
- End-to-end mutation tests
- Full 5991+ test suite with Rust backend
- Performance benchmarking

### Phase 5: Deprecation
- Final regression verification
- No psycopg references check
- Performance validation

---

## Success Metrics

### Must Have (Exit Criteria)
- ✅ All 5991+ tests pass with Rust backend
- ✅ No regressions vs current psycopg implementation
- ✅ 100% backward-compatible Python API
- ✅ Connection pool stable under load

### Performance Targets
- ✅ Query execution: 20-30% faster
- ✅ Response time: 15-25% faster
- ✅ Memory usage: 10-15% lower
- ✅ Throughput: 2-3x higher sustained

### Code Quality
- ✅ Type hints complete
- ✅ Doc comments on all public APIs
- ✅ Test coverage ≥ 85%
- ✅ Zero `unsafe` code (except where required by tokio)

---

## Files to Create

### Rust Code
```
fraiseql_rs/src/
├── db/
│   ├── mod.rs           (NEW)
│   ├── pool.rs          (NEW)
│   ├── query.rs         (NEW)
│   ├── where_builder.rs (NEW)
│   └── types.rs         (NEW)
├── sql/                 (NEW)
│   ├── mod.rs
│   ├── generator.rs
│   ├── select_builder.rs
│   └── where_clause.rs
└── response/            (NEW)
    ├── mod.rs
    ├── builder.rs
    └── streaming.rs
```

### Python Code
```
src/fraiseql/
├── core/
│   └── database.py      (NEW)
├── db.py                (DEPRECATE in Phase 5)
└── sql/graphql_where_generator.py (OPTIMIZE in Phase 2)
```

### Tests
```
tests/
├── integration/db/              (NEW)
│   ├── test_rust_pool.py
│   ├── test_rust_queries.py
│   ├── test_rust_where.py
│   ├── test_rust_mutations.py
│   └── test_rust_streaming.py
└── regression/
    └── test_rust_db_parity.py   (NEW)
```

---

## Files to Modify

### Configuration
- `fraiseql_rs/Cargo.toml` - Add dependencies (Phase 1)
- `pyproject.toml` - Remove psycopg (Phase 5)

### Core
- `fraiseql_rs/src/lib.rs` - Export new modules
- `src/fraiseql/core/rust_pipeline.py` - Integrate new functions

### Build System
- `.github/workflows/` - Update CI/CD if needed

---

## Files to Delete

**Phase 5 only**:
- `src/fraiseql/db.py` (old psycopg layer)
- Any psycopg-specific utilities
- `.phases/rust-postgres-driver/` directory (after merge)

---

## Dependency Changes

### New Dependencies (Cargo.toml)

```toml
tokio-postgres = "0.7"
deadpool-postgres = "0.14"
deadpool = "0.10"
tokio-postgres-rustls = "0.10"
rustls = "0.23"
rustls-pemfile = "2.0"
async-trait = "0.1"
```

### Removed Dependencies (pyproject.toml - Phase 5)

```
psycopg[pool]>=3.2.6
psycopg-pool>=3.2.6
opentelemetry-instrumentation-psycopg  (from tracing extras)
```

### No Breaking Changes

- No changes to user-facing Python API
- All existing imports remain valid
- Backward compatible at all phases (feature flags in 1-4, complete in 5)

---

## Documentation to Update

1. **docs/architecture/database-layer.md** (NEW)
   - Rust-native architecture overview
   - Connection pooling details
   - Performance characteristics

2. **docs/getting-started/**
   - Update environment variable docs
   - Configuration examples

3. **README.md**
   - Highlight "Rust-native database layer"
   - Update performance claims

4. **CHANGELOG.md**
   - Document major architectural change
   - Performance improvements
   - Migration notes (if any)

---

## How to Execute

### Before Starting
```bash
# Make sure you're on the feature branch
git checkout feature/rust-postgres-driver

# Verify branch is clean
git status
# Should show: nothing to commit, working tree clean
```

### For Each Phase
```bash
# 1. Read the phase document thoroughly
# 2. Follow implementation steps sequentially
# 3. Run verification commands
# 4. Commit only when acceptance criteria met
# 5. Update progress in this README

# Commit convention per phase:
test(scope): tests for X [PHASE]
feat(scope): implement X [PHASE]
refactor(scope): clean up X [PHASE]
```

### After Each Phase
```bash
# Verify no regressions
uv run pytest tests/ -v --tb=short

# Quick performance check
uv run pytest tests/performance/ -v 2>&1 | head -20

# Code quality
uv run ruff check src/ fraiseql_rs/
```

### Before Merge
```bash
# Full verification
cargo build --release -p fraiseql_rs
uv run pip install -e .
uv run pytest tests/ -v

# Ensure evergreen state
# - No TODOs in production code
# - All docstrings complete
# - Type hints complete
# - No debugging code
```

---

## Comprehensive Troubleshooting Guide

### PyO3/Async Issues

**Error**: `error: expected async function or closure`
```
#[pyo3_asyncio::tokio::main]
async fn my_function() { }  // ← Wrong decorator placement
```
**Fix**: Use correct decorator syntax:
```rust
#[pyo3_asyncio::tokio::main]
async fn my_function(py: Python) -> PyResult<Py<PyAny>> {
    pyo3_asyncio::tokio::future_into_py(py, async { Ok(()) })
}
```

**Error**: `type mismatch resolving fn pointer`
**Cause**: Returning wrong type from async function (not wrapping in `future_into_py`)
**Fix**: Always return `PyResult<Py<PyAny>>` from PyO3 async functions

**Error**: `RuntimeError: no running event loop`
**Cause**: Calling async function without Python event loop
**Fix**: Ensure function is called from async context in Python

### Type Conversion Issues

**Error**: `cannot call `.get()` on `i32`**
**Cause**: Wrong OID type mapping (tried to extract as wrong type)
**Fix**: Check PostgreSQL OID types in `README.md` type conversion table

**Error**: `json values are not comparable`
**Cause**: Comparing JSONB values directly
**Fix**: Use `::text` cast or convert to string for comparison

### Connection Pool Issues

**Error**: `Connection pool exhausted`
**Cause**:
- All connections in use (increase `MAX_SIZE`)
- Connections not being returned (connection leak)
- Timeout waiting for available connection
**Debug**:
```bash
# Check pool stats
python -c "pool.get_stats()"

# Look for leaks
RUST_LOG=debug cargo test --lib db::pool
```

**Error**: `connection refused` on first query
**Cause**: Database not ready or connection string invalid
**Fix**:
- Verify `DATABASE_URL` is correct
- Wait for database startup
- Check network connectivity

### WHERE Clause Issues

**Error**: `Unsupported filter format`
**Cause**: New filter type not implemented
**Check**: Phase 2 WHERE builder - ensure operator is implemented

**Error**: `Parameter binding error`
**Cause**: Mismatch in parameter count vs placeholders
**Fix**: Verify `param_counter` is incremented correctly

### Streaming Issues

**Error**: `Connection interrupted during stream`
**Cause**: Client disconnect during large result fetch
**Fix**: Implement error recovery in streaming code

**Error**: `Memory explosion on large result sets`
**Cause**: Not actually streaming (buffering all rows)
**Fix**: Verify using cursors/portals for streaming

### Compilation Issues
- Check `phase-1-foundation.md` step 1 (dependencies)
- Verify Rust version: `rustc --version` (1.70+)
- Common issue: `pyo3` version conflicts - see Cargo.toml for pinned versions

### Test Failures
- Phase 1: Connection pool tests - see Phase 1 troubleshooting
- Phase 2: WHERE clause tests - compare generated SQL with Python version
- Phase 3+: Check memory profiling and streaming behavior

### Performance Issues
- Memory: Check `cargo bench --bench memory`
- Throughput: Check `cargo bench --bench pipeline`
- Query: Check `RUST_LOG=debug cargo test` for timing information
- Profile with: `cargo flamegraph --bench pipeline`

### BuildSystem Issues

**Error**: `error: failed to run custom build command`
**Cause**: PyO3 build script failed
**Fix**:
```bash
# Clean and rebuild
cargo clean
cargo build -p fraiseql_rs -vv  # Verbose output
```

**Error**: `maturin develop` fails
**Cause**: Python environment issue
**Fix**:
```bash
# Use correct Python interpreter
uv run pip install -e . --no-build-isolation
```

---

## Questions Before Starting?

### "How long will this take?"
~44 hours full-time, or ~1 week. Can be parallelized if needed.

### "Is this safe?"
Yes. Feature flags in phases 1-4 allow fallback to psycopg. Phase 5 is irreversible, but only after full validation.

### "Can we rollback?"
Yes. Via `git revert` at any phase. Phase 5 requires more work, but still possible.

### "Do users need to do anything?"
No. Completely internal refactor with zero API changes.

### "What if we find bugs?"
Each phase has comprehensive testing. Parity tests catch regressions. Rollback available at any phase.

---

## Quick Command Reference

```bash
# Build
cargo build -p fraiseql_rs
uv run pip install -e .

# Test
cargo test -p fraiseql_rs --lib
uv run pytest tests/ -v

# Benchmark
cargo bench -p fraiseql_rs

# Format
cargo fmt -p fraiseql_rs
uv run ruff format src/

# Lint
cargo clippy -p fraiseql_rs
uv run ruff check src/

# Performance baseline
uv run pytest tests/performance/ -v 2>&1 | tee baseline.txt
```

---

## Success Definition

When all 5 phases are complete:

- ✅ **Performance**: Queries 20-30% faster, responses 15-25% faster
- ✅ **Architecture**: Rust-native core, Python API layer
- ✅ **Reliability**: All 5991+ tests pass, zero regressions
- ✅ **Sustainability**: Clean code, comprehensive tests, evergreen state
- ✅ **Compatibility**: 100% backward compatible, zero user impact
- ✅ **Documentation**: Architecture documented, deployment guide updated

---

## Next Steps

1. ✅ Read this document completely
2. ✅ Review README.md (overview and architecture)
3. 👉 Start with **phase-1-foundation.md**
4. 📋 Follow each phase sequentially
5. ✔️ Verify completion criteria before moving to next phase
6. 📝 Update progress tracking as you go
7. 🎉 Merge when complete

---

**Status**: ✅ All plans complete, ready for implementation
**Branch**: `feature/rust-postgres-driver`
**Last Updated**: 2025-12-18

Good luck! 🚀
