# Database Infrastructure - TASC-Stack ORM Layer

The database module provides the foundational data persistence layer for the entire TASC-Stack ecosystem. It implements dual ORM patterns - an enhanced PostgreSQL ORM extending SQLModel, and a sophisticated Elasticsearch search ORM. This creates a unified approach to both relational data storage and search/retrieval operations across all services.

## Core Philosophy

This module embodies the principle of "rich models, thin controllers" - providing database models with extensive built-in capabilities while maintaining clean separation between data persistence and business logic. The design focuses on:

- **Unified Interface**: Consistent CRUD patterns across PostgreSQL and Elasticsearch
- **Session Abstraction**: Global session management eliminating connection boilerplate
- **Advanced Querying**: Rich predicate system supporting complex database operations
- **Developer Experience**: Intuitive APIs that make database operations feel natural

## Key Architectural Concepts

### Dual ORM Strategy
The module provides two complementary ORM systems working in harmony:
- **BaseSQLModel**: Enhanced relational ORM for transactional data (PostgreSQL)
- **BaseSearchModel**: Vector and lexical search ORM for retrieval operations (Elasticsearch)

This allows services to leverage the strengths of both systems - ACID compliance for critical data and advanced search capabilities for user-facing features.

### Global Session Management
Context-managed sessions eliminate the need for explicit connection handling. The SessionManager uses ContextVars to ensure session isolation across async tasks and FastAPI requests, while providing both sync and async operation modes.

### Rich Predicate System
Advanced query building through predicate classes that support everything from basic equality to full-text search, regex matching, and complex range queries. This abstracts away raw SQL/Elasticsearch query construction.

**📍 Code Pointers:**
- PostgreSQL ORM: `base_models.py` → `BaseSQLModel` class
- Elasticsearch ORM: `search_model.py` → `BaseSearchModel` class
- Session management: `session_manager.py` → `session_manager_factory()` function
- Global instances: `session_manager.py` → `GlobalSessionManager`, `es_manager.py` → `GlobalESManager`

## Major Components

### PostgreSQL ORM (`base_models.py`)
Enhanced SQLModel providing rich CRUD operations:
- **Async/Sync Operations**: All operations support both execution modes
- **Smart Create**: Validation and type coercion during model creation
- **Flexible Queries**: Support for joins, predicates, pagination, and sorting
- **Session Integration**: Automatic session management and transaction handling

**Key Methods:**
- `create()`, `get()`, `update()`, `delete()` - Basic CRUD operations
- `get_or_create()` - Idempotent creation pattern
- `count()`, `sort()`, `simple_sort()` - Query utilities

**📍 Code Pointers:**
- Base class: `base_models.py` → `BaseSQLModel`
- Model schemas: `base_models.py` → `ModelCreate`, `ModelUpdate`, `ModelRead`
- Usage patterns: Search for `BaseSQLModel` inheritance in service repositories

### Elasticsearch ORM (`search_model.py`)
Sophisticated search model supporting modern Elasticsearch features:
- **Vector Search**: Dense vector similarity search with HNSW optimization
- **Lexical Search**: Traditional text search with BM25 scoring
- **Hybrid Search**: Native Elasticsearch RRF (Reciprocal Rank Fusion)
- **Advanced Querying**: Builder pattern for complex search operations

**Key Search Methods:**
- `vector_search()` - Semantic/vector similarity search
- `lexical_search()` - Traditional full-text search  
- `hybrid_search()` - Combined vector and lexical search with RRF
- `query()` - Fluent query builder interface

**📍 Code Pointers:**
- Base search model: `search_model.py` → `BaseSearchModel`
- Query builder: `search_model.py` → `QueryBuilder` class
- Field definitions: `search_model.py` → `SearchField`, `DenseVectorField`, `DictField`
- Search utilities: `search_model.py` → search method implementations

### Session Management (`session_manager.py`)
Factory-based session management supporting both sync and async operations:
- **Context Variables**: Thread-safe session isolation using ContextVars
- **Decorator Pattern**: `@with_sync_session` and `@with_async_session` decorators
- **Predicate System**: Rich query building with type-safe predicates
- **Connection Pooling**: Optimized connection management with health checks

**Advanced Predicates:**
- Basic: `gt`, `lt`, `eq`, `ne`, `in_`, `like`, `ilike`
- Text: `regex`, `full_text_search` with multi-language support
- Range: Support for complex range queries and sorting

**📍 Code Pointers:**
- Factory function: `session_manager.py` → `session_manager_factory()`
- Predicate classes: `session_manager.py` → `Predicate` subclasses
- Statement builders: `session_manager.py` → `Stmt` subclasses
- Global instance: `session_manager.py` → `GlobalSessionManager`

### Elasticsearch Management (`es_manager.py`)
Index and search operation management:
- **Dynamic Mapping**: Auto-generation of Elasticsearch mappings from Pydantic models
- **Index Lifecycle**: Automatic index creation and management
- **Bulk Operations**: Efficient batch indexing and updates
- **Multi-Search**: Parallel search execution for complex queries

**📍 Code Pointers:**
- Manager factory: `es_manager.py` → `es_index_factory()`
- Index management: `es_manager.py` → `ESIndex` class
- Search operations: `es_manager.py` → `search()`, `multi_search()`, `bulk_index()`
- Global instance: `es_manager.py` → `GlobalESManager`

## Working with Database Models

### Creating PostgreSQL Models
Inherit from BaseSQLModel for rich CRUD operations:

```python
class MyModel(BaseSQLModel, table=True):
    name: str
    age: int
    
# Rich operations available immediately
model = await MyModel.create(name="test", age=25)
results = await MyModel.get(age=25)
await model.update(age=26)
```

### Creating Search Models
Inherit from BaseSearchModel for Elasticsearch operations:

```python
class MySearchModel(BaseSearchModel):
    title: str = SearchField(es_analyzer="english")
    content: str = SearchField(es_type="text")
    embedding: List[float] = DenseVectorField(dims=1536)
    
# Advanced search capabilities
results = await MySearchModel.vector_search("embedding", query_vector)
hybrid_results = await MySearchModel.hybrid_search("content", "query", "embedding", vector)
```

### Session Configuration
Configure session managers in service `session_manager.py` files:

```python
def configure_session_manager():
    GlobalSessionManager.DB_URL = settings.DATABASE_URL
    return GlobalSessionManager

SessionManager = configure_session_manager()
```

**📍 Code Pointers:**
- Model examples: Search for `BaseSQLModel` and `BaseSearchModel` in service repositories
- Session setup: Look for `session_manager.py` files in service directories
- Field usage: Search for `SearchField`, `DenseVectorField` usage patterns

## Development Guidelines

### PostgreSQL Model Patterns
- **Inherit from BaseSQLModel**: Always use the enhanced base class
- **Use SessionManager Decorators**: Wrap endpoint functions with session decorators
- **Leverage Predicates**: Use rich predicate system instead of raw SQL
- **Implement Mixins**: Keep business logic in separate mixin classes

### Elasticsearch Model Patterns
- **Define Rich Mappings**: Use SearchField with proper analyzers and types
- **Optimize Vector Fields**: Configure HNSW parameters for vector search performance
- **Use Query Builder**: Leverage the fluent query interface for complex searches
- **Index Management**: Let the system handle index creation automatically

### Performance Considerations
- **Connection Pooling**: Configured with optimized pool sizes and health checks
- **Bulk Operations**: Use bulk_index for large-scale Elasticsearch operations
- **Vector Search**: Configure appropriate num_candidates for speed/accuracy balance
- **Query Optimization**: Use appropriate predicates and indexes for PostgreSQL

### Error Handling
- **Custom Exceptions**: Use `DatabaseRowUnavailable` for missing records
- **Dependency Management**: Leverage `delete_object_with_dependencies()` for cascading deletes
- **Session Cleanup**: Sessions are automatically managed through context managers

**📍 Code Pointers:**
- Exception handling: `exceptions.py` → `DatabaseRowUnavailable`
- Dependency utilities: `utils.py` → `delete_object_with_dependencies()`
- Primary key utilities: `utils.py` → `find_primary_key()`
- URL building: `utils.py` → `build_database_url()`

---

This database layer is the foundation upon which all TASC-Stack services build their data persistence. Master these patterns and you'll have the tools to create robust, searchable, and performant data models that leverage the best of both relational and search paradigms.