---
title: PostgreSQL Database Schema
description: Complete reference for TraceRTM's PostgreSQL database schema, including tables, relationships, indexes, and performance optimization
---

# PostgreSQL Database Schema

This document provides comprehensive documentation of TraceRTM's PostgreSQL database schema, including table structures, relationships, indexes, and performance considerations.

## Overview

TraceRTM uses PostgreSQL with several advanced features:

- **JSONB columns** for flexible metadata storage
- **Full-text search** using pg_trgm extension
- **Materialized views** for performance optimization
- **Event sourcing** for change tracking
- **Optimistic locking** for concurrent updates
- **Automatic triggers** for timestamp management
- **Soft deletes** for data retention

### Database Extensions

```sql
-- Vector extension for future semantic search
CREATE EXTENSION IF NOT EXISTS vector;

-- Trigram extension for full-text search
CREATE EXTENSION IF NOT EXISTS pg_trgm;
```

## Core Tables

### Projects Table

The root entity representing a TraceRTM project.

```sql
CREATE TABLE projects (
    id VARCHAR(255) PRIMARY KEY,
    name VARCHAR(255) NOT NULL UNIQUE,
    description TEXT,
    project_metadata JSONB NOT NULL DEFAULT '{}',
    created_at TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT NOW(),
    updated_at TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT NOW()
);

CREATE INDEX idx_projects_name ON projects(name);
```

**Columns:**

- `id`: UUID string, generated automatically
- `name`: Unique project name, indexed for fast lookup
- `description`: Optional project description
- `project_metadata`: JSONB field for extensible project configuration
- `created_at`: Automatic timestamp on creation
- `updated_at`: Automatic timestamp, updated on each change

**Metadata Structure:**

```json
{
    "settings": {
        "default_status": "todo",
        "allowed_views": ["requirements", "code", "tests"],
        "workflow": "agile"
    },
    "custom_fields": {},
    "integrations": {
        "github": { "repo": "org/repo", "enabled": true }
    }
}
```

### Items Table

The primary entity representing traceable items (features, requirements, code, tests, etc.).

```sql
CREATE TABLE items (
    id VARCHAR(255) PRIMARY KEY,
    project_id VARCHAR(255) NOT NULL,
    title VARCHAR(500) NOT NULL,
    description TEXT,
    view VARCHAR(50) NOT NULL,
    item_type VARCHAR(50) NOT NULL,
    status VARCHAR(50) NOT NULL DEFAULT 'todo',
    priority VARCHAR(50) NOT NULL DEFAULT 'medium',
    owner VARCHAR(255),
    parent_id VARCHAR(255),
    item_metadata JSONB NOT NULL DEFAULT '{}',
    version INTEGER NOT NULL DEFAULT 1,
    deleted_at TIMESTAMP WITH TIME ZONE,
    created_at TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT NOW(),
    updated_at TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT NOW(),

    CONSTRAINT fk_items_project
        FOREIGN KEY (project_id) REFERENCES projects(id) ON DELETE CASCADE,
    CONSTRAINT fk_items_parent
        FOREIGN KEY (parent_id) REFERENCES items(id) ON DELETE SET NULL
);
```

**Indexes:**

```sql
-- Single column indexes
CREATE INDEX idx_items_project_id ON items(project_id);
CREATE INDEX idx_items_view ON items(view);
CREATE INDEX idx_items_item_type ON items(item_type);
CREATE INDEX idx_items_status ON items(status);
CREATE INDEX idx_items_priority ON items(priority);
CREATE INDEX idx_items_owner ON items(owner);
CREATE INDEX idx_items_parent_id ON items(parent_id);
CREATE INDEX idx_items_deleted_at ON items(deleted_at);

-- Composite indexes for common queries
CREATE INDEX idx_items_project_view ON items(project_id, view);
CREATE INDEX idx_items_project_status ON items(project_id, status);
CREATE INDEX idx_items_project_type ON items(project_id, item_type);

-- Full-text search indexes
CREATE INDEX idx_items_title_trgm ON items USING gin (title gin_trgm_ops);
CREATE INDEX idx_items_description_trgm ON items USING gin (description gin_trgm_ops);
```

**Columns:**

- `id`: UUID string, generated automatically
- `project_id`: Foreign key to projects table (CASCADE delete)
- `title`: Required item title (max 500 chars)
- `description`: Optional detailed description
- `view`: Item view/category (requirements, code, tests, etc.)
- `item_type`: Specific type within view (feature, class, unit_test, etc.)
- `status`: Workflow status (todo, in_progress, done, blocked)
- `priority`: Priority level (low, medium, high, critical)
- `owner`: Optional owner/assignee identifier
- `parent_id`: Self-referencing FK for hierarchical items (SET NULL on delete)
- `item_metadata`: JSONB for extensible properties
- `version`: Optimistic locking version (auto-incremented)
- `deleted_at`: Soft delete timestamp (NULL = not deleted)
- `created_at`: Automatic creation timestamp
- `updated_at`: Automatic update timestamp

**Optimistic Locking:**

The `version` column enables concurrent update detection:

```python
# SQLAlchemy configuration
__mapper_args__ = {
    "version_id_col": version,  # Auto-increment on each update
}

# Raises ConcurrencyError if version changed
item.title = "Updated"
await session.commit()  # Fails if another session updated first
```

**Metadata Structure:**

```json
{
    "tags": ["backend", "api"],
    "estimated_hours": 8,
    "actual_hours": 10,
    "test_coverage": 85.5,
    "code_file": "src/api/endpoints.py",
    "line_range": [100, 250],
    "git_commit": "abc123def456",
    "custom_fields": {
        "acceptance_criteria": ["criterion 1", "criterion 2"]
    }
}
```

**Soft Deletes:**

Items are soft-deleted by setting `deleted_at`:

```sql
-- Soft delete
UPDATE items SET deleted_at = NOW() WHERE id = 'item-id';

-- Query active items only
SELECT * FROM items WHERE deleted_at IS NULL;

-- Restore deleted item
UPDATE items SET deleted_at = NULL WHERE id = 'item-id';
```

### Links Table

Represents relationships between items.

```sql
CREATE TABLE links (
    id VARCHAR(255) PRIMARY KEY,
    project_id VARCHAR(255) NOT NULL,
    source_item_id VARCHAR(255) NOT NULL,
    target_item_id VARCHAR(255) NOT NULL,
    link_type VARCHAR(50) NOT NULL,
    link_metadata JSONB NOT NULL DEFAULT '{}',
    created_at TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT NOW(),
    updated_at TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT NOW(),

    CONSTRAINT fk_links_project
        FOREIGN KEY (project_id) REFERENCES projects(id) ON DELETE CASCADE,
    CONSTRAINT fk_links_source
        FOREIGN KEY (source_item_id) REFERENCES items(id) ON DELETE CASCADE,
    CONSTRAINT fk_links_target
        FOREIGN KEY (target_item_id) REFERENCES items(id) ON DELETE CASCADE
);
```

**Indexes:**

```sql
CREATE INDEX idx_links_project_id ON links(project_id);
CREATE INDEX idx_links_source_item_id ON links(source_item_id);
CREATE INDEX idx_links_target_item_id ON links(target_item_id);
CREATE INDEX idx_links_link_type ON links(link_type);
CREATE INDEX idx_links_source_target ON links(source_item_id, target_item_id);
CREATE INDEX idx_links_project_type ON links(project_id, link_type);
```

**Columns:**

- `id`: UUID string, generated automatically
- `project_id`: Foreign key to projects (CASCADE delete)
- `source_item_id`: Source item in relationship (CASCADE delete)
- `target_item_id`: Target item in relationship (CASCADE delete)
- `link_type`: Type of relationship (implements, tests, depends_on, blocks, etc.)
- `link_metadata`: JSONB for additional link properties
- `created_at`: Automatic creation timestamp
- `updated_at`: Automatic update timestamp

**Link Types:**

- `implements`: Code implements requirement
- `tests`: Test covers code/requirement
- `depends_on`: Item depends on another
- `blocks`: Item blocks another
- `relates_to`: General relationship
- `parent_child`: Hierarchical relationship
- `duplicates`: Duplicate item reference

**Metadata Structure:**

```json
{
    "confidence": 0.95,
    "auto_generated": true,
    "verified": false,
    "notes": "Auto-linked based on commit message",
    "custom_fields": {}
}
```

## Agent Tables

### Agents Table

Tracks AI agents working on the project.

```sql
CREATE TABLE agents (
    id VARCHAR(255) PRIMARY KEY,
    project_id VARCHAR(255) NOT NULL,
    name VARCHAR(255) NOT NULL,
    agent_type VARCHAR(50) NOT NULL,
    status VARCHAR(50) NOT NULL DEFAULT 'active',
    agent_metadata JSONB NOT NULL DEFAULT '{}',
    last_activity_at VARCHAR(50),
    created_at TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT NOW(),
    updated_at TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT NOW(),

    CONSTRAINT fk_agents_project
        FOREIGN KEY (project_id) REFERENCES projects(id) ON DELETE CASCADE
);

CREATE INDEX idx_agents_project_id ON agents(project_id);
CREATE INDEX idx_agents_project_status ON agents(project_id, status);
```

**Columns:**

- `id`: UUID string
- `project_id`: Foreign key to projects
- `name`: Agent display name
- `agent_type`: Agent category (indexer, analyzer, coordinator, etc.)
- `status`: Current status (active, idle, error, stopped)
- `agent_metadata`: Configuration and state
- `last_activity_at`: ISO timestamp of last activity
- `created_at`: Registration timestamp
- `updated_at`: Last update timestamp

### Agent Events Table

Logs agent activities for coordination and debugging.

```sql
CREATE TABLE agent_events (
    id VARCHAR(255) PRIMARY KEY,
    project_id VARCHAR(255) NOT NULL,
    agent_id VARCHAR(255) NOT NULL,
    event_type VARCHAR(50) NOT NULL,
    item_id VARCHAR(255),
    event_data JSONB,
    created_at TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT NOW(),
    updated_at TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT NOW(),

    CONSTRAINT fk_agent_events_project
        FOREIGN KEY (project_id) REFERENCES projects(id) ON DELETE CASCADE,
    CONSTRAINT fk_agent_events_agent
        FOREIGN KEY (agent_id) REFERENCES agents(id) ON DELETE CASCADE,
    CONSTRAINT fk_agent_events_item
        FOREIGN KEY (item_id) REFERENCES items(id) ON DELETE SET NULL
);

CREATE INDEX idx_agent_events_project_id ON agent_events(project_id);
CREATE INDEX idx_agent_events_agent_id ON agent_events(agent_id);
CREATE INDEX idx_agent_events_item_id ON agent_events(item_id);
CREATE INDEX idx_agent_events_project_agent ON agent_events(project_id, agent_id);
CREATE INDEX idx_agent_events_project_type ON agent_events(project_id, event_type);
```

**Event Types:**

- `item_created`: Agent created an item
- `item_updated`: Agent updated an item
- `link_created`: Agent created a link
- `conflict_detected`: Concurrent modification detected
- `coordination_message`: Agent communication
- `status_update`: Agent status change

### Agent Locks Table

Prevents concurrent modifications using distributed locking.

```sql
CREATE TABLE agent_locks (
    id VARCHAR(255) PRIMARY KEY,
    project_id VARCHAR(255) NOT NULL,
    item_id VARCHAR(255) NOT NULL,
    agent_id VARCHAR(255) NOT NULL,
    lock_type VARCHAR(50) DEFAULT 'exclusive',
    expires_at TIMESTAMP WITH TIME ZONE,
    created_at TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT NOW(),
    updated_at TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT NOW(),

    CONSTRAINT fk_locks_project
        FOREIGN KEY (project_id) REFERENCES projects(id) ON DELETE CASCADE,
    CONSTRAINT fk_locks_item
        FOREIGN KEY (item_id) REFERENCES items(id) ON DELETE CASCADE,
    CONSTRAINT fk_locks_agent
        FOREIGN KEY (agent_id) REFERENCES agents(id) ON DELETE CASCADE
);

CREATE INDEX idx_locks_item_agent ON agent_locks(item_id, agent_id);
CREATE INDEX idx_locks_project_item ON agent_locks(project_id, item_id);
```

**Lock Types:**

- `exclusive`: Only one agent can modify
- `shared`: Multiple agents can read

## Event Sourcing

### Events Table

Records all state changes for history tracking and audit.

```sql
CREATE TABLE events (
    id SERIAL PRIMARY KEY,
    project_id VARCHAR(255) NOT NULL,
    event_type VARCHAR(50) NOT NULL,
    entity_type VARCHAR(50) NOT NULL,
    entity_id VARCHAR(255) NOT NULL,
    agent_id VARCHAR(255),
    data JSONB NOT NULL,
    created_at TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT NOW(),
    updated_at TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT NOW(),

    CONSTRAINT fk_events_project
        FOREIGN KEY (project_id) REFERENCES projects(id) ON DELETE CASCADE
);

CREATE INDEX idx_events_project_id ON events(project_id);
CREATE INDEX idx_events_event_type ON events(event_type);
CREATE INDEX idx_events_entity_id ON events(entity_id);
CREATE INDEX idx_events_project_entity ON events(project_id, entity_id);
CREATE INDEX idx_events_project_type ON events(project_id, event_type);
CREATE INDEX idx_events_created_at ON events(created_at);
```

**Event Types:**

- `item_created`, `item_updated`, `item_deleted`
- `link_created`, `link_deleted`
- `project_created`, `project_updated`

**Data Structure:**

```json
{
    "before": { "title": "Old Title", "status": "todo" },
    "after": { "title": "New Title", "status": "in_progress" },
    "changes": ["title", "status"],
    "context": {
        "user": "agent-123",
        "reason": "automated update",
        "source": "git commit abc123"
    }
}
```

### Change Log Table

Tracks changes for materialized view refresh.

```sql
CREATE TABLE change_log (
    id BIGSERIAL PRIMARY KEY,
    table_name VARCHAR(50) NOT NULL,
    operation VARCHAR(10) NOT NULL,
    record_id UUID NOT NULL,
    project_id UUID,
    changed_at TIMESTAMP NOT NULL DEFAULT NOW(),
    processed BOOLEAN NOT NULL DEFAULT FALSE,
    metadata JSONB DEFAULT '{}'
);

-- Partial index for unprocessed changes (fast lookups)
CREATE INDEX idx_change_log_processed ON change_log(processed)
    WHERE processed = FALSE;

CREATE INDEX idx_change_log_table ON change_log(table_name);
CREATE INDEX idx_change_log_record_id ON change_log(record_id);
CREATE INDEX idx_change_log_project_id ON change_log(project_id);
CREATE INDEX idx_change_log_changed_at ON change_log(changed_at);
```

**Automatic Change Tracking:**

```sql
-- Trigger function for items
CREATE OR REPLACE FUNCTION log_item_change()
RETURNS TRIGGER AS $$
BEGIN
    INSERT INTO change_log (table_name, operation, record_id, project_id)
    VALUES (
        'items',
        TG_OP,
        COALESCE(NEW.id, OLD.id),
        COALESCE(NEW.project_id, OLD.project_id)
    );
    RETURN COALESCE(NEW, OLD);
END;
$$ LANGUAGE plpgsql;

-- Attach trigger
CREATE TRIGGER item_change_trigger
AFTER INSERT OR UPDATE OR DELETE ON items
FOR EACH ROW EXECUTE FUNCTION log_item_change();
```

## Database Triggers

### Automatic Timestamp Updates

```sql
CREATE OR REPLACE FUNCTION update_updated_at_column()
RETURNS TRIGGER AS $$
BEGIN
    NEW.updated_at = NOW();
    RETURN NEW;
END;
$$ LANGUAGE plpgsql;

-- Applied to all tables
CREATE TRIGGER update_projects_updated_at
BEFORE UPDATE ON projects
FOR EACH ROW EXECUTE FUNCTION update_updated_at_column();

CREATE TRIGGER update_items_updated_at
BEFORE UPDATE ON items
FOR EACH ROW EXECUTE FUNCTION update_updated_at_column();

-- ... (applied to all tables with updated_at)
```

## Common Queries

### Query Active Items by Project and View

```sql
SELECT
    id, title, status, priority, owner
FROM items
WHERE
    project_id = 'project-uuid'
    AND view = 'requirements'
    AND deleted_at IS NULL
ORDER BY created_at DESC;
```

### Full-Text Search

```sql
SELECT
    id, title, description,
    similarity(title, 'search term') as relevance
FROM items
WHERE
    project_id = 'project-uuid'
    AND deleted_at IS NULL
    AND (
        title % 'search term'
        OR description % 'search term'
    )
ORDER BY relevance DESC
LIMIT 20;
```

### Get Item with All Links

```sql
SELECT
    i.*,
    json_agg(
        json_build_object(
            'link_id', l.id,
            'link_type', l.link_type,
            'direction', CASE
                WHEN l.source_item_id = i.id THEN 'outgoing'
                ELSE 'incoming'
            END,
            'related_item', json_build_object(
                'id', ri.id,
                'title', ri.title,
                'view', ri.view
            )
        )
    ) FILTER (WHERE l.id IS NOT NULL) as links
FROM items i
LEFT JOIN links l ON (l.source_item_id = i.id OR l.target_item_id = i.id)
LEFT JOIN items ri ON (
    CASE
        WHEN l.source_item_id = i.id THEN l.target_item_id = ri.id
        ELSE l.source_item_id = ri.id
    END
)
WHERE i.id = 'item-uuid'
GROUP BY i.id;
```

### Find Orphaned Items (No Links)

```sql
SELECT i.*
FROM items i
LEFT JOIN links l ON (l.source_item_id = i.id OR l.target_item_id = i.id)
WHERE
    i.project_id = 'project-uuid'
    AND i.deleted_at IS NULL
    AND l.id IS NULL;
```

### Get Project Statistics

```sql
SELECT
    p.name,
    COUNT(DISTINCT i.id) FILTER (WHERE i.deleted_at IS NULL) as total_items,
    COUNT(DISTINCT i.id) FILTER (WHERE i.status = 'done' AND i.deleted_at IS NULL) as completed_items,
    COUNT(DISTINCT l.id) as total_links,
    COUNT(DISTINCT a.id) FILTER (WHERE a.status = 'active') as active_agents
FROM projects p
LEFT JOIN items i ON i.project_id = p.id
LEFT JOIN links l ON l.project_id = p.id
LEFT JOIN agents a ON a.project_id = p.id
WHERE p.id = 'project-uuid'
GROUP BY p.id, p.name;
```

## Migration Strategy

### Alembic Configuration

TraceRTM uses Alembic for database migrations:

```bash
# Create new migration
alembic revision -m "description"

# Apply migrations
alembic upgrade head

# Rollback one migration
alembic downgrade -1

# View migration history
alembic history
```

### Migration Best Practices

1. **Always test migrations** on a copy of production data
2. **Use transactions** - migrations are wrapped in transactions
3. **Create indexes concurrently** to avoid locking:

```python
def upgrade():
    op.create_index(
        'idx_items_status',
        'items',
        ['status'],
        postgresql_concurrently=True
    )
```

4. **Add columns with defaults carefully**:

```python
# Add column with default (fast)
op.add_column('items',
    sa.Column('priority', sa.String(50),
        nullable=False,
        server_default='medium'
    )
)

# Then remove server_default if not needed
op.alter_column('items', 'priority', server_default=None)
```

## Performance Considerations

### Indexing Strategy

**Do Index:**
- Foreign keys (project_id, parent_id, etc.)
- Frequently filtered columns (status, view, item_type)
- Composite indexes for common query patterns
- JSONB fields with GIN indexes for complex queries

**Don't Over-Index:**
- Every column (slows writes)
- Low-cardinality columns (status has only 4-5 values)
- Rarely queried fields

### JSONB Performance

```sql
-- GIN index for JSONB containment
CREATE INDEX idx_items_metadata_gin ON items USING gin (item_metadata);

-- Query with JSONB operators
SELECT * FROM items
WHERE item_metadata @> '{"tags": ["backend"]}';

-- Extract JSONB values efficiently
SELECT
    id,
    title,
    item_metadata->>'tags' as tags,
    (item_metadata->>'estimated_hours')::int as hours
FROM items
WHERE item_metadata->>'test_coverage' > '80';
```

### Connection Pooling

```python
# Database configuration
engine = create_async_engine(
    "postgresql+asyncpg://user:pass@localhost/tracertm",
    pool_size=20,           # Max connections
    max_overflow=10,        # Additional connections when pool full
    pool_recycle=3600,      # Recycle connections after 1 hour
    pool_pre_ping=True,     # Verify connections before use
)
```

### Query Optimization Tips

1. **Use EXPLAIN ANALYZE** to understand query plans:

```sql
EXPLAIN ANALYZE
SELECT * FROM items
WHERE project_id = 'uuid' AND status = 'todo';
```

2. **Limit result sets** with OFFSET/LIMIT for pagination
3. **Use EXISTS instead of COUNT** when checking existence:

```sql
-- Slow
SELECT COUNT(*) FROM items WHERE project_id = 'uuid';

-- Fast
SELECT EXISTS(SELECT 1 FROM items WHERE project_id = 'uuid');
```

4. **Batch operations** instead of individual inserts:

```python
# Batch insert
stmt = insert(Item).values([
    {"id": "id1", "title": "Item 1", ...},
    {"id": "id2", "title": "Item 2", ...},
])
await session.execute(stmt)
```

### Vacuum and Analyze

```sql
-- Manual vacuum (usually automatic)
VACUUM ANALYZE items;

-- Aggressive vacuum (reclaim disk space)
VACUUM FULL items;

-- Update statistics
ANALYZE items;
```

## Backup and Recovery

### Backup Strategy

```bash
# Full database backup
pg_dump -h localhost -U postgres tracertm > backup.sql

# Compressed backup
pg_dump -h localhost -U postgres tracertm | gzip > backup.sql.gz

# Custom format (faster restore, parallel)
pg_dump -h localhost -U postgres -Fc tracertm > backup.dump
```

### Restore

```bash
# From SQL file
psql -h localhost -U postgres tracertm < backup.sql

# From custom format
pg_restore -h localhost -U postgres -d tracertm backup.dump

# Parallel restore (4 jobs)
pg_restore -h localhost -U postgres -d tracertm -j 4 backup.dump
```

## Monitoring

### Key Metrics

```sql
-- Table sizes
SELECT
    schemaname,
    tablename,
    pg_size_pretty(pg_total_relation_size(schemaname||'.'||tablename)) AS size
FROM pg_tables
WHERE schemaname = 'public'
ORDER BY pg_total_relation_size(schemaname||'.'||tablename) DESC;

-- Index usage
SELECT
    schemaname,
    tablename,
    indexname,
    idx_scan,
    pg_size_pretty(pg_relation_size(indexrelid)) as size
FROM pg_stat_user_indexes
ORDER BY idx_scan ASC;

-- Slow queries (requires pg_stat_statements)
SELECT
    query,
    calls,
    total_time,
    mean_time,
    max_time
FROM pg_stat_statements
ORDER BY mean_time DESC
LIMIT 10;
```

## Related Documentation

- [Storage Architecture](/docs/development/architecture/storage-architecture)
- [API Reference](/api)
- [Migration Guide](/docs/guides/migrations)
