---
title: Storage Architecture
description: Comprehensive guide to TraceRTM's offline-first hybrid storage system
---

# Storage Architecture

TraceRTM implements a sophisticated **offline-first hybrid storage architecture** that combines the best of both worlds: human-readable Markdown files for version control and SQLite for fast queries.

## Overview

### Design Philosophy

The storage architecture is built on three core principles:

1. **Offline-First**: Work without internet connectivity, sync when available
2. **Human-Readable**: All data stored in Markdown + YAML format for git versioning
3. **Performance**: SQLite index for fast queries and search

### Two-Tier Storage Model

```
┌─────────────────────────────────────────────────────────────┐
│                      Storage Architecture                    │
├─────────────────────────────────────────────────────────────┤
│                                                              │
│  Tier 1: Project-Local (.trace/)                            │
│  ┌────────────────────────────────────────────┐             │
│  │ .trace/                                    │             │
│  │ ├── project.yaml         # Project metadata│             │
│  │ ├── epics/               # Epic items      │             │
│  │ │   └── EPIC-001.md                        │             │
│  │ ├── stories/             # Story items     │             │
│  │ │   └── STORY-001.md                       │             │
│  │ ├── tests/               # Test items      │             │
│  │ │   └── TEST-001.md                        │             │
│  │ ├── tasks/               # Task items      │             │
│  │ │   └── TASK-001.md                        │             │
│  │ ├── wireframes/          # Wireframe items │             │
│  │ │   └── WF-001.md                          │             │
│  │ ├── docs/                # Documentation   │             │
│  │ ├── changes/             # Change logs     │             │
│  │ └── .meta/               # Metadata        │             │
│  │     ├── links.yaml       # Traceability    │             │
│  │     ├── agents.yaml      # Agent configs   │             │
│  │     └── sync.yaml        # Sync state      │             │
│  └────────────────────────────────────────────┘             │
│         │                                                    │
│         │ File Watcher (debounced)                          │
│         ▼                                                    │
│  Tier 2: Global Index (~/.tracertm/)                        │
│  ┌────────────────────────────────────────────┐             │
│  │ ~/.tracertm/                               │             │
│  │ ├── tracertm.db          # SQLite index    │             │
│  │ │   ├── projects          # Project table  │             │
│  │ │   ├── items             # Items table    │             │
│  │ │   ├── links             # Links table    │             │
│  │ │   ├── items_fts         # Full-text idx  │             │
│  │ │   ├── sync_queue        # Sync queue     │             │
│  │ │   └── sync_state        # Sync metadata  │             │
│  │ └── conflicts/            # Conflict backups│            │
│  │     ├── item/                               │             │
│  │     ├── link/                               │             │
│  │     └── project/                            │             │
│  └────────────────────────────────────────────┘             │
│         │                                                    │
│         │ Sync Engine                                       │
│         ▼                                                    │
│  Remote API (when online)                                   │
│  ┌────────────────────────────────────────────┐             │
│  │ PostgreSQL + Redis + Meilisearch           │             │
│  └────────────────────────────────────────────┘             │
│                                                              │
└─────────────────────────────────────────────────────────────┘
```

## The .trace/ Directory Structure

### Directory Layout

Every TraceRTM project contains a `.trace/` directory at its root:

```
my-project/
├── .trace/
│   ├── project.yaml           # Project configuration
│   ├── epics/                 # High-level features
│   │   ├── EPIC-001.md
│   │   └── EPIC-002.md
│   ├── stories/               # User stories
│   │   ├── STORY-001.md
│   │   ├── STORY-002.md
│   │   └── STORY-003.md
│   ├── tests/                 # Test cases
│   │   ├── TEST-001.md
│   │   └── TEST-002.md
│   ├── tasks/                 # Implementation tasks
│   │   ├── TASK-001.md
│   │   └── TASK-002.md
│   ├── wireframes/            # UI/UX wireframes
│   │   ├── WF-001.md
│   │   └── WF-002.md
│   ├── docs/                  # Additional documentation
│   ├── changes/               # Change logs
│   └── .meta/                 # Metadata (not git-versioned)
│       ├── links.yaml         # Traceability links
│       ├── agents.yaml        # AI agent configurations
│       └── sync.yaml          # Sync state (gitignored)
├── .gitignore
└── src/
    └── ...
```

### Project Metadata (project.yaml)

The `project.yaml` file contains project-level configuration and counters:

```yaml
id: "550e8400-e29b-41d4-a716-446655440000"
name: "My Awesome Project"
description: "A revolutionary new application"
version: "1.0.0"

# Auto-incrementing counters for external IDs
counters:
  epic: 5
  story: 23
  test: 45
  task: 67

# Default settings
settings:
  default_priority: "medium"
  default_status: "todo"
  auto_link_commits: true

# Team members
team:
  - id: "user-123"
    name: "Alice Developer"
    role: "engineer"

# Lifecycle hooks (for automation)
hooks:
  on_item_create:
    - "notify_slack"
  on_item_update:
    - "update_dashboard"
```

## Markdown + YAML Frontmatter Format

### Item File Structure

Each item (epic, story, test, task, wireframe) is stored as a Markdown file with YAML frontmatter:

````markdown
---
id: "550e8400-e29b-41d4-a716-446655440001"
external_id: "STORY-001"
type: "story"
status: "in_progress"
priority: "high"
owner: "alice@example.com"
parent: "EPIC-001"
version: 3
created: "2024-01-15T10:30:00Z"
updated: "2024-01-20T14:45:00Z"
tags:
  - "authentication"
  - "security"
links:
  - type: "implements"
    target: "EPIC-001"
  - type: "tested_by"
    target: "TEST-001"
---

# User Login with Email and Password

## Description

As a user, I want to log in to the application using my email address and password
so that I can access my personalized content securely.

## Acceptance Criteria

- [ ] User can enter email and password in a login form
- [ ] System validates credentials against the database
- [ ] Successful login redirects to dashboard
- [ ] Failed login shows clear error message
- [ ] Password is transmitted securely (HTTPS only)
- [ ] Account locks after 5 failed attempts

## Notes

Security considerations:
- Use bcrypt for password hashing (minimum 12 rounds)
- Implement rate limiting (max 5 attempts per 15 minutes)
- Session tokens expire after 24 hours

## History

| Version | Date       | Author            | Changes                    |
|---------|------------|-------------------|----------------------------|
| 1       | 2024-01-15 | alice@example.com | Initial creation           |
| 2       | 2024-01-18 | alice@example.com | Added rate limiting        |
| 3       | 2024-01-20 | bob@example.com   | Updated security notes     |
````

### Traceability Links (links.yaml)

All cross-references between items are stored in `.meta/links.yaml`:

```yaml
# Traceability links
# Link types: implements, tested_by, depends_on, blocks, relates_to

links:
  - id: "link-001"
    source: "STORY-001"
    target: "EPIC-001"
    type: "implements"
    created: "2024-01-15T10:30:00Z"

  - id: "link-002"
    source: "TEST-001"
    target: "STORY-001"
    type: "tested_by"
    created: "2024-01-16T11:00:00Z"
    metadata:
      coverage: "95%"

  - id: "link-003"
    source: "STORY-002"
    target: "STORY-001"
    type: "depends_on"
    created: "2024-01-17T09:15:00Z"
```

## File Watcher Functionality

### Auto-Indexing System

TraceRTM includes a file watcher that automatically indexes changes from `.trace/` into SQLite:

```python
from tracertm.storage import LocalStorageManager
from tracertm.storage.file_watcher import TraceFileWatcher

# Initialize storage
storage = LocalStorageManager()

# Create watcher for a project
watcher = TraceFileWatcher(
    project_path=Path("/path/to/my-project"),
    storage=storage,
    debounce_ms=500,      # Wait 500ms before processing
    auto_sync=True        # Queue changes for remote sync
)

# Start watching
watcher.start()

# Get statistics
stats = watcher.get_stats()
print(f"Processed: {stats['events_processed']}")
print(f"Pending: {stats['events_pending']}")

# Stop when done
watcher.stop()
```

### Debouncing Strategy

The file watcher implements **debouncing** to avoid processing rapid consecutive changes:

```
File Change Events:
  t=0ms:   EPIC-001.md modified
  t=100ms: EPIC-001.md modified (debounced)
  t=200ms: EPIC-001.md modified (debounced)
  t=700ms: Process change (500ms after last event)
```

**Benefits:**
- Avoids redundant processing during rapid edits
- Reduces CPU usage
- Prevents database lock contention
- Default: 500ms delay (configurable)

### Watched File Types

The watcher monitors:

- **Item files**: `*.md` in epics/, stories/, tests/, tasks/, wireframes/
- **Links file**: `.meta/links.yaml`
- **Project config**: `project.yaml`
- **Agent configs**: `.meta/agents.yaml`

**Ignored:**
- `.meta/sync.yaml` (local-only sync state)
- Hidden files (except `.trace`, `.meta`)
- Non-YAML/MD files

## Sync Engine Operation

### Sync Phases

The sync engine implements a **three-phase sync protocol**:

```
┌─────────────────────────────────────────────────┐
│            Sync Phases                          │
├─────────────────────────────────────────────────┤
│                                                 │
│  Phase 1: Change Detection                     │
│  ┌───────────────────────────────────────┐     │
│  │ • Scan .trace/ for modified files     │     │
│  │ • Compare content hashes              │     │
│  │ • Queue local changes                 │     │
│  └───────────────────────────────────────┘     │
│           │                                     │
│           ▼                                     │
│  Phase 2: Upload (Local → Remote)              │
│  ┌───────────────────────────────────────┐     │
│  │ • Process sync queue                  │     │
│  │ • Upload changes to API               │     │
│  │ • Retry with exponential backoff      │     │
│  │ • Handle conflicts                    │     │
│  └───────────────────────────────────────┘     │
│           │                                     │
│           ▼                                     │
│  Phase 3: Download (Remote → Local)            │
│  ┌───────────────────────────────────────┐     │
│  │ • Fetch remote changes since last sync│     │
│  │ • Apply to local SQLite + Markdown    │     │
│  │ • Detect and resolve conflicts        │     │
│  │ • Update sync timestamp               │     │
│  └───────────────────────────────────────┘     │
│                                                 │
└─────────────────────────────────────────────────┘
```

### Change Detection

Content hashing detects local modifications:

```python
from tracertm.storage.sync_engine import ChangeDetector

detector = ChangeDetector()

# Compute hash of current content
current_hash = detector.compute_hash(markdown_content)

# Compare with stored hash
stored_hash = item.metadata.get("content_hash")
has_changed = detector.has_changed(markdown_content, stored_hash)

if has_changed:
    # Queue for sync
    sync_queue.enqueue(
        entity_type=EntityType.ITEM,
        entity_id=item.id,
        operation=OperationType.UPDATE,
        payload=item.to_dict()
    )
```

### Sync Queue

All changes are queued before sync:

```sql
-- Sync queue table
CREATE TABLE sync_queue (
    id INTEGER PRIMARY KEY AUTOINCREMENT,
    entity_type TEXT NOT NULL,      -- project, item, link
    entity_id TEXT NOT NULL,        -- UUID
    operation TEXT NOT NULL,        -- create, update, delete
    payload TEXT NOT NULL,          -- JSON data
    created_at TEXT NOT NULL,
    retry_count INTEGER DEFAULT 0,
    last_error TEXT,
    UNIQUE(entity_type, entity_id, operation)
);
```

### Retry Strategy

Failed syncs retry with **exponential backoff**:

```python
# Retry sequence:
# Attempt 0: wait 1s
# Attempt 1: wait 2s
# Attempt 2: wait 4s
# Attempt 3: wait 8s
# Attempt 4: wait 16s
# Attempt 5: wait 32s
# Attempt 6: wait 60s (capped)
```

## Conflict Detection and Resolution

### Vector Clocks

TraceRTM uses **vector clocks** for distributed version ordering:

```python
from tracertm.storage.conflict_resolver import VectorClock

# Each change gets a vector clock
clock = VectorClock(
    client_id="alice-laptop",
    version=5,
    timestamp=datetime.utcnow(),
    parent_version=4
)

# Compare clocks
if clock.happens_before(other_clock):
    print("Clock happened before other")
elif clock.is_concurrent(other_clock):
    print("Conflict: concurrent changes!")
```

### Conflict Detection

Conflicts occur when:
1. Both local and remote versions exist
2. Vector clocks are concurrent (neither happens-before the other)
3. Content differs (different hashes or data)

### Resolution Strategies

**1. Last-Write-Wins (Default)**

```python
# Newest timestamp wins
strategy = ConflictStrategy.LAST_WRITE_WINS
resolved = resolver.resolve(conflict, strategy)
```

**2. Local-Wins**

```python
# Always prefer local changes
strategy = ConflictStrategy.LOCAL_WINS
resolved = resolver.resolve(conflict, strategy)
```

**3. Remote-Wins**

```python
# Always prefer server changes
strategy = ConflictStrategy.REMOTE_WINS
resolved = resolver.resolve(conflict, strategy)
```

**4. Manual Resolution**

```python
# User provides merged content
merged_data = {
    "title": "Manually Merged Title",
    "status": "in_progress"
}
resolved = resolver.resolve_manual(conflict, merged_data, merged_by="alice")
```

### Conflict Backups

All conflicts are backed up before resolution:

```
~/.tracertm/conflicts/
└── item/
    └── item-123_20240120_143015/
        ├── conflict.json      # Conflict metadata
        ├── local.json         # Local version
        └── remote.json        # Remote version
```

## Local Storage Manager

### Initialization

```python
from tracertm.storage import LocalStorageManager
from pathlib import Path

# Initialize with default ~/.tracertm directory
storage = LocalStorageManager()

# Or specify custom directory
storage = LocalStorageManager(base_dir=Path("/custom/path"))
```

### Project Operations

**Initialize New Project:**

```python
# Create .trace/ directory in project
trace_dir, project_id = storage.init_project(
    project_path=Path("/path/to/my-project"),
    project_name="My Project",
    description="A revolutionary app"
)
```

**Register Existing Project:**

```python
# Register existing .trace/ directory
project_id = storage.register_project(
    project_path=Path("/path/to/existing-project")
)
```

**Index Project:**

```python
# Scan .trace/ and index all items
counts = storage.index_project(
    project_path=Path("/path/to/my-project")
)
# Output: {'epics': 5, 'stories': 23, 'tests': 45, 'tasks': 67}
```

### Item Operations

```python
# Get project storage
project_storage = storage.get_project_storage_for_path(
    Path("/path/to/my-project")
)

# Get item storage
project = project_storage.get_project()
item_storage = project_storage.get_item_storage(project)

# Create item
item = item_storage.create_item(
    title="User Login Feature",
    item_type="story",
    external_id="STORY-001",
    description="Implement user authentication",
    status="todo",
    priority="high",
    owner="alice@example.com"
)

# Update item
updated_item = item_storage.update_item(
    item_id=item.id,
    status="in_progress"
)

# List items
items = item_storage.list_items(
    item_type="story",
    status="in_progress"
)
```

### Full-Text Search

```python
# Search across all indexed items
results = storage.search_items(
    query="authentication security",
    project_id=project.id
)

for item in results:
    print(f"{item.external_id}: {item.title}")
```

## Multi-View Storage Organization

### View-Based Organization

Items can be organized by different "views":

```python
# Create item with specific view
item = item_storage.create_item(
    title="Dashboard Redesign",
    item_type="story",
    view="FEATURE",  # View: FEATURE, BUGFIX, TECH_DEBT
    status="todo"
)
```

### Custom Metadata

Store arbitrary metadata with items:

```python
item = item_storage.create_item(
    title="Optimize Database Queries",
    item_type="task",
    metadata={
        "estimated_hours": 8,
        "sprint": "Sprint 23",
        "labels": ["performance", "backend"]
    }
)
```

## Backup and Recovery

### Automatic Backups

TraceRTM creates automatic backups during critical operations:

```python
# Backups created before:
# - Conflict resolution
# - Bulk deletions
# - Schema migrations
```

### Recovery Procedures

**Restore from .trace/ Markdown:**

```python
# Re-index from markdown files
counts = storage.index_project(project_path)
print(f"Restored {counts} items from .trace/")
```

**Restore from Backup:**

```python
import gzip
import json

def restore_backup(storage: LocalStorageManager, backup_path: Path):
    """Restore from compressed backup."""
    with gzip.open(backup_path, 'rt', encoding='utf-8') as f:
        backup_data = json.load(f)

    # Restore items
    for item_data in backup_data['items']:
        item_storage.create_item(**item_data)
```

## Performance Considerations

### SQLite Optimizations

```sql
-- Enable WAL mode for concurrent reads
PRAGMA journal_mode=WAL;

-- Increase cache size (10MB)
PRAGMA cache_size=-10000;
```

### FTS5 Full-Text Search

```sql
-- Create FTS5 index for fast search
CREATE VIRTUAL TABLE items_fts USING fts5(
    item_id UNINDEXED,
    title,
    description,
    item_type
);

-- Search query (extremely fast)
SELECT items.* FROM items
JOIN items_fts ON items.id = items_fts.item_id
WHERE items_fts MATCH 'authentication security'
ORDER BY rank;
```

### Indexing Strategy

```sql
-- Indexes for common queries
CREATE INDEX idx_items_project_type ON items(project_id, item_type);
CREATE INDEX idx_items_status ON items(status);
CREATE INDEX idx_items_owner ON items(owner);
CREATE INDEX idx_links_source ON links(source_item_id);
CREATE INDEX idx_links_target ON links(target_item_id);
```

### Batch Operations

```python
# Use transactions for bulk operations
session = storage.get_session()
try:
    for item_data in bulk_items:
        item_storage.create_item(**item_data)
    session.commit()
except Exception:
    session.rollback()
    raise
finally:
    session.close()
```

## Best Practices

### Version Control Integration

```gitignore
# .gitignore for .trace/ directory
.trace/.meta/sync.yaml      # Local sync state
.trace/.meta/cache/         # Cache directory
```

**Commit:**
- All `.md` files in epics/, stories/, tests/, tasks/
- `project.yaml`
- `.meta/links.yaml`
- `.meta/agents.yaml`

**Don't commit:**
- `.meta/sync.yaml` (local sync state)
- `.meta/cache/` (temporary cache)

### Conflict Prevention

- **Pull before push**: Sync before making local changes
- **Frequent syncs**: Sync every 5-10 minutes when online
- **Clear ownership**: Assign items to specific team members
- **Use branches**: Create feature branches for major changes

### Data Integrity

```python
# Validate markdown files before committing
from tracertm.storage.markdown_parser import parse_item_markdown

for md_file in trace_dir.glob("**/*.md"):
    try:
        item_data = parse_item_markdown(md_file)
        assert item_data.id
        assert item_data.external_id
        assert item_data.item_type
        assert item_data.status
    except Exception as e:
        print(f"Invalid file: {md_file}: {e}")
```

## Summary

The TraceRTM storage architecture provides:

- **Offline-first operation** with automatic sync
- **Human-readable** Markdown + YAML format
- **Fast queries** via SQLite FTS5
- **Automatic indexing** with file watcher
- **Conflict resolution** with multiple strategies
- **Backup and recovery** capabilities
- **Performance optimizations** for large datasets

This architecture enables teams to work offline, version control their requirements, and maintain traceability without compromising on performance or usability.
