---
title: "Advanced Queries"
description: "Master advanced query techniques in TraceRTM for powerful data filtering, analysis, and reporting"
---

# Advanced Queries

TraceRTM provides a powerful query system for filtering, searching, and analyzing traceability data. This guide covers advanced query techniques to help you extract insights from your requirements and test data efficiently.

## Query Syntax Overview

TraceRTM queries can be performed through multiple interfaces:

- **API Endpoints**: REST API with query parameters
- **CLI Commands**: Command-line interface with filter flags
- **TUI Interface**: Interactive terminal UI with search/filter
- **Python SDK**: Direct repository access with SQLAlchemy filters

### Basic Query Structure

All queries follow a consistent pattern:

```python
# Python SDK
items = await item_repository.query(
    project_id="proj-123",
    filters={
        "status": "in_progress",
        "item_type": "requirement"
    },
    limit=100
)
```

```bash
# CLI
tracertm items list --project proj-123 --status in_progress --type requirement --limit 100
```

```http
# REST API
GET /api/v1/items?project_id=proj-123&status=in_progress&type=requirement&limit=100
```

## Filtering by Status, Type, and Tags

### Status Filtering

Filter items by their workflow status:

```python
# Find all items in "todo" status
todo_items = await item_repository.get_by_project(
    project_id="proj-123",
    status="todo"
)

# Find items in specific statuses
from sqlalchemy import select
from tracertm.models.item import Item

query = select(Item).where(
    Item.project_id == "proj-123",
    Item.status.in_(["in_progress", "review", "blocked"])
)
result = await session.execute(query)
items = result.scalars().all()
```

**Common Status Values:**
- `todo` - Not started
- `in_progress` - Active work
- `review` - Under review
- `done` - Completed
- `blocked` - Blocked by dependencies
- `archived` - Archived items

### Type Filtering

Filter by item type to separate requirements, tests, code, etc.:

```python
# Get all test cases
tests = await item_repository.query(
    project_id="proj-123",
    filters={"item_type": "test"}
)

# Get requirements and features
query = select(Item).where(
    Item.project_id == "proj-123",
    Item.item_type.in_(["requirement", "feature"])
)
```

**Common Item Types:**
- `requirement` - Business/functional requirements
- `feature` - Product features
- `test` - Test cases
- `bug` - Bug reports
- `api` - API endpoints
- `code` - Code modules/files
- `document` - Documentation

### Metadata Tag Filtering

Query items by metadata tags and custom fields:

```python
from sqlalchemy import cast, String
from sqlalchemy.dialects.postgresql import JSONB

# Find items with specific tag
query = select(Item).where(
    Item.item_metadata["tags"].astext.contains("security")
)

# Find items with multiple tags (JSON array contains)
query = select(Item).where(
    Item.item_metadata["tags"].astext.op("@>")(
        cast('["critical", "security"]', JSONB)
    )
)

# Query custom metadata fields
query = select(Item).where(
    Item.item_metadata["priority"].astext == "high",
    Item.item_metadata["component"].astext == "authentication"
)
```

## Date-Based Queries

### Created Date Filtering

Find items created within specific date ranges:

```python
from datetime import datetime, timedelta, timezone

# Items created in last 7 days
week_ago = datetime.now(timezone.utc) - timedelta(days=7)
query = select(Item).where(
    Item.project_id == "proj-123",
    Item.created_at >= week_ago
)

# Items created in specific date range
start_date = datetime(2024, 1, 1, tzinfo=timezone.utc)
end_date = datetime(2024, 3, 31, tzinfo=timezone.utc)

query = select(Item).where(
    Item.created_at.between(start_date, end_date)
)
```

### Updated Date Filtering

Track recently modified items:

```python
# Items updated in last 24 hours
yesterday = datetime.now(timezone.utc) - timedelta(hours=24)
recent_updates = select(Item).where(
    Item.updated_at >= yesterday
).order_by(Item.updated_at.desc())

# Items not updated in 30 days (stale items)
thirty_days_ago = datetime.now(timezone.utc) - timedelta(days=30)
stale_items = select(Item).where(
    Item.updated_at < thirty_days_ago,
    Item.status != "done"
)
```

### Deleted Items

Query soft-deleted items:

```python
# Include deleted items
all_items = await item_repository.list_all(
    project_id="proj-123",
    include_deleted=True
)

# Only deleted items
from sqlalchemy import select
deleted_items = select(Item).where(
    Item.project_id == "proj-123",
    Item.deleted_at.isnot(None)
)

# Items deleted in last 7 days (for recovery)
week_ago = datetime.now(timezone.utc) - timedelta(days=7)
recently_deleted = select(Item).where(
    Item.deleted_at >= week_ago
)
```

## Combining Multiple Filters

### AND Conditions

Combine multiple filters for precise queries:

```python
# High-priority incomplete requirements
query = select(Item).where(
    Item.project_id == "proj-123",
    Item.item_type == "requirement",
    Item.priority == "high",
    Item.status.in_(["todo", "in_progress"]),
    Item.deleted_at.is_(None)
)

# Active test cases assigned to specific owner
query = select(Item).where(
    Item.item_type == "test",
    Item.owner == "alice@example.com",
    Item.status != "archived",
    Item.created_at >= datetime(2024, 1, 1, tzinfo=timezone.utc)
)
```

### OR Conditions

Use SQLAlchemy's `or_()` for alternative conditions:

```python
from sqlalchemy import or_

# Items that are blocked OR have high priority
query = select(Item).where(
    Item.project_id == "proj-123",
    or_(
        Item.status == "blocked",
        Item.priority == "high"
    )
)

# Requirements OR features in specific view
query = select(Item).where(
    Item.view == "product",
    or_(
        Item.item_type == "requirement",
        Item.item_type == "feature"
    )
)
```

### Complex Nested Conditions

```python
from sqlalchemy import and_, or_

# (High priority AND todo) OR (medium priority AND in_progress)
query = select(Item).where(
    or_(
        and_(
            Item.priority == "high",
            Item.status == "todo"
        ),
        and_(
            Item.priority == "medium",
            Item.status == "in_progress"
        )
    )
)
```

## Searching with Wildcards

### Text Search

Search titles and descriptions using pattern matching:

```python
# Items with "login" in title (case-insensitive)
query = select(Item).where(
    Item.title.ilike("%login%")
)

# Items with description containing "authentication"
query = select(Item).where(
    Item.description.ilike("%authentication%")
)

# Starts with pattern
query = select(Item).where(
    Item.title.like("REQ-%")  # All requirements with REQ- prefix
)

# Multiple search terms (AND)
query = select(Item).where(
    Item.title.ilike("%user%"),
    Item.description.ilike("%security%")
)
```

### Full-Text Search (Advanced)

For PostgreSQL full-text search capabilities:

```python
from sqlalchemy import func

# Full-text search on title and description
query = select(Item).where(
    func.to_tsvector('english', Item.title + ' ' + Item.description).match(
        'authentication & security'
    )
)
```

## Querying Relationships and Links

### Finding Linked Items

Query items based on their relationships:

```python
from tracertm.repositories.link_repository import LinkRepository

# Get all items linked FROM a specific item
source_links = await link_repository.get_by_source(source_item_id="item-123")
target_ids = [link.target_item_id for link in source_links]

target_items = select(Item).where(Item.id.in_(target_ids))

# Get all items linked TO a specific item
target_links = await link_repository.get_by_target(target_item_id="item-456")
source_ids = [link.source_item_id for link in target_links]

# Get all connected items (bidirectional)
all_links = await link_repository.get_by_item(item_id="item-789")
```

### Hierarchical Queries

Query parent-child relationships:

```python
# Get direct children
children = await item_repository.get_children(item_id="parent-123")

# Get all descendants (recursive)
descendants = await item_repository.get_descendants(item_id="parent-123")

# Get all ancestors up to root
ancestors = await item_repository.get_ancestors(item_id="child-456")

# Find root items (no parent)
roots = select(Item).where(
    Item.project_id == "proj-123",
    Item.parent_id.is_(None)
)

# Find leaf items (no children)
# This requires a subquery
from sqlalchemy import exists

has_children = select(Item.id).where(Item.parent_id == Item.id).exists()
leaves = select(Item).where(~has_children)
```

### Link Type Filtering

Filter by relationship types:

```python
# Get all "implements" links
impl_links = select(Link).where(
    Link.project_id == "proj-123",
    Link.link_type == "implements"
)

# Get tests linked to requirements
req_test_links = select(Link).where(
    Link.link_type == "tests"
).join(
    Item, Link.source_item_id == Item.id
).where(
    Item.item_type == "test"
)
```

## Aggregate Queries

### Counting Items

Get counts and statistics:

```python
from sqlalchemy import func

# Count by status
status_counts = await item_repository.count_by_status(project_id="proj-123")
# Returns: {"todo": 10, "in_progress": 5, "done": 25}

# Count by type
query = select(
    Item.item_type,
    func.count(Item.id).label("count")
).where(
    Item.project_id == "proj-123"
).group_by(Item.item_type)

result = await session.execute(query)
type_counts = {row.item_type: row.count for row in result}

# Count by priority
query = select(
    Item.priority,
    func.count(Item.id)
).group_by(Item.priority)
```

### Coverage Statistics

Calculate traceability coverage metrics:

```python
# Test coverage: % of requirements with test links
total_reqs = select(func.count(Item.id)).where(
    Item.project_id == "proj-123",
    Item.item_type == "requirement"
)

tested_reqs = select(func.count(func.distinct(Link.source_item_id))).where(
    Link.link_type == "tests",
    Link.project_id == "proj-123"
).join(
    Item, Link.source_item_id == Item.id
).where(
    Item.item_type == "requirement"
)

# Calculate coverage %
total = (await session.execute(total_reqs)).scalar()
tested = (await session.execute(tested_reqs)).scalar()
coverage = (tested / total * 100) if total > 0 else 0
```

### Aggregating by Date

Group items by creation date:

```python
# Items created per day
query = select(
    func.date(Item.created_at).label("date"),
    func.count(Item.id).label("count")
).where(
    Item.project_id == "proj-123"
).group_by(func.date(Item.created_at)).order_by("date")

# Items by month
query = select(
    func.date_trunc('month', Item.created_at).label("month"),
    func.count(Item.id)
).group_by(func.date_trunc('month', Item.created_at))
```

### Advanced Statistics

```python
# Average items per status
query = select(
    Item.status,
    func.count(Item.id).label("count"),
    func.avg(Item.version).label("avg_versions")
).group_by(Item.status)

# Items with most links
query = select(
    Item.id,
    Item.title,
    func.count(Link.id).label("link_count")
).join(
    Link,
    or_(Link.source_item_id == Item.id, Link.target_item_id == Item.id)
).group_by(Item.id, Item.title).order_by(func.count(Link.id).desc())
```

## Saving and Reusing Queries

### Creating Reusable Query Functions

Save common queries as reusable functions:

```python
# queries/common.py
from typing import Any
from sqlalchemy import select
from tracertm.models.item import Item

def critical_incomplete_items(project_id: str):
    """Get critical incomplete items."""
    return select(Item).where(
        Item.project_id == project_id,
        Item.priority == "critical",
        Item.status.in_(["todo", "in_progress", "blocked"]),
        Item.deleted_at.is_(None)
    )

def items_needing_review(project_id: str, days: int = 7):
    """Get items updated recently needing review."""
    from datetime import datetime, timedelta, timezone
    cutoff = datetime.now(timezone.utc) - timedelta(days=days)

    return select(Item).where(
        Item.project_id == project_id,
        Item.status == "review",
        Item.updated_at >= cutoff
    )

def untested_requirements(project_id: str):
    """Get requirements without test coverage."""
    from sqlalchemy import exists
    from tracertm.models.link import Link

    has_test_link = exists().where(
        Link.source_item_id == Item.id,
        Link.link_type == "tests"
    )

    return select(Item).where(
        Item.project_id == project_id,
        Item.item_type == "requirement",
        ~has_test_link
    )

# Usage
critical = await session.execute(critical_incomplete_items("proj-123"))
items = critical.scalars().all()
```

### Query Builder Pattern

Create a flexible query builder:

```python
class ItemQueryBuilder:
    """Fluent query builder for items."""

    def __init__(self, session: AsyncSession, project_id: str):
        self.session = session
        self.query = select(Item).where(Item.project_id == project_id)

    def with_status(self, *statuses: str):
        """Filter by status."""
        self.query = self.query.where(Item.status.in_(statuses))
        return self

    def with_type(self, *types: str):
        """Filter by item type."""
        self.query = self.query.where(Item.item_type.in_(types))
        return self

    def with_priority(self, *priorities: str):
        """Filter by priority."""
        self.query = self.query.where(Item.priority.in_(priorities))
        return self

    def created_after(self, date: datetime):
        """Filter by creation date."""
        self.query = self.query.where(Item.created_at >= date)
        return self

    def search_title(self, term: str):
        """Search in title."""
        self.query = self.query.where(Item.title.ilike(f"%{term}%"))
        return self

    def exclude_deleted(self):
        """Exclude deleted items."""
        self.query = self.query.where(Item.deleted_at.is_(None))
        return self

    async def all(self) -> list[Item]:
        """Execute query and return all results."""
        result = await self.session.execute(self.query)
        return list(result.scalars().all())

    async def count(self) -> int:
        """Get count of matching items."""
        from sqlalchemy import func
        count_query = select(func.count()).select_from(self.query.subquery())
        result = await self.session.execute(count_query)
        return result.scalar()

# Usage
builder = ItemQueryBuilder(session, "proj-123")
items = await builder \
    .with_status("todo", "in_progress") \
    .with_type("requirement") \
    .with_priority("high", "critical") \
    .exclude_deleted() \
    .all()
```

## Exporting Query Results

### Export to JSON

```python
import json
from datetime import datetime

def serialize_item(item: Item) -> dict:
    """Serialize item to JSON-compatible dict."""
    return {
        "id": item.id,
        "title": item.title,
        "description": item.description,
        "type": item.item_type,
        "status": item.status,
        "priority": item.priority,
        "owner": item.owner,
        "created_at": item.created_at.isoformat(),
        "updated_at": item.updated_at.isoformat(),
        "metadata": item.item_metadata,
    }

# Export query results
items = await session.execute(query)
data = [serialize_item(item) for item in items.scalars().all()]

with open("export.json", "w") as f:
    json.dump(data, f, indent=2)
```

### Export to CSV

```python
import csv

# Export to CSV
items = await session.execute(query)

with open("export.csv", "w", newline="") as f:
    writer = csv.writer(f)
    writer.writerow(["ID", "Title", "Type", "Status", "Priority", "Owner"])

    for item in items.scalars().all():
        writer.writerow([
            item.id,
            item.title,
            item.item_type,
            item.status,
            item.priority,
            item.owner or "",
        ])
```

### Export Traceability Matrix

```python
async def export_traceability_matrix(session: AsyncSession, project_id: str):
    """Export traceability matrix showing requirement-test relationships."""
    from tracertm.repositories.link_repository import LinkRepository

    # Get all requirements
    reqs_query = select(Item).where(
        Item.project_id == project_id,
        Item.item_type == "requirement"
    ).order_by(Item.title)

    reqs = (await session.execute(reqs_query)).scalars().all()

    # Get all test links
    link_repo = LinkRepository(session)

    matrix = []
    for req in reqs:
        test_links = await link_repo.get_by_source(req.id)
        test_ids = [link.target_item_id for link in test_links
                   if link.link_type == "tests"]

        tests_query = select(Item).where(Item.id.in_(test_ids))
        tests = (await session.execute(tests_query)).scalars().all()

        matrix.append({
            "requirement_id": req.id,
            "requirement_title": req.title,
            "test_count": len(tests),
            "tests": [{"id": t.id, "title": t.title} for t in tests],
            "coverage": "yes" if tests else "no"
        })

    return matrix
```

## Performance Tips for Large Datasets

### Use Pagination

Always paginate large result sets:

```python
# Good: Paginated query
async def get_items_paginated(
    session: AsyncSession,
    project_id: str,
    page: int = 1,
    page_size: int = 100
):
    """Get items with pagination."""
    offset = (page - 1) * page_size

    query = select(Item).where(
        Item.project_id == project_id
    ).limit(page_size).offset(offset)

    result = await session.execute(query)
    return result.scalars().all()

# Bad: Loading all items at once
all_items = await item_repository.list_all(project_id)  # Could be 100k+ items!
```

### Use Indexes Effectively

Ensure your queries use database indexes:

```python
# These use indexes (fast)
query = select(Item).where(Item.project_id == "proj-123")  # indexed
query = select(Item).where(Item.status == "todo")  # indexed
query = select(Item).where(Item.item_type == "test")  # indexed

# These may not use indexes (slower)
query = select(Item).where(Item.title.like("%search%"))  # full scan
query = select(Item).where(func.lower(Item.title) == "test")  # function prevents index
```

### Select Only Needed Columns

Don't load entire objects if you only need specific fields:

```python
# Good: Select only needed columns
query = select(Item.id, Item.title, Item.status).where(
    Item.project_id == "proj-123"
)

# Bad: Loading full objects when only titles needed
query = select(Item).where(Item.project_id == "proj-123")
items = result.scalars().all()
titles = [item.title for item in items]  # Loaded unnecessary data
```

### Use Bulk Operations

For inserting/updating multiple items:

```python
# Good: Bulk insert
from tracertm.services.bulk_operation_service import BulkOperationService

bulk_service = BulkOperationService(session)
items = await bulk_service.bulk_create_items(
    project_id="proj-123",
    items_data=[
        {"title": "Item 1", "type": "requirement"},
        {"title": "Item 2", "type": "requirement"},
        # ... 1000 items
    ]
)

# Bad: Individual inserts in loop
for item_data in items_data:
    await item_repository.create(**item_data)  # 1000 DB roundtrips!
```

### Eager Loading Relationships

Use eager loading to avoid N+1 queries:

```python
from sqlalchemy.orm import selectinload

# Good: Eager load related data
query = select(Item).options(
    selectinload(Item.links)  # Load links in single query
).where(Item.project_id == "proj-123")

# Bad: N+1 query problem
items = await session.execute(select(Item))
for item in items.scalars():
    links = await link_repository.get_by_item(item.id)  # N queries!
```

### Cache Frequent Queries

Cache results of frequently-run queries:

```python
from functools import lru_cache
from datetime import datetime, timedelta

class QueryCache:
    """Simple query result cache."""

    def __init__(self, ttl_seconds: int = 300):
        self.cache = {}
        self.ttl = ttl_seconds

    def get(self, key: str):
        """Get cached value if not expired."""
        if key in self.cache:
            value, timestamp = self.cache[key]
            if datetime.now() - timestamp < timedelta(seconds=self.ttl):
                return value
        return None

    def set(self, key: str, value):
        """Cache a value."""
        self.cache[key] = (value, datetime.now())

# Usage
cache = QueryCache(ttl_seconds=300)

async def get_status_counts_cached(session: AsyncSession, project_id: str):
    """Get status counts with caching."""
    cache_key = f"status_counts:{project_id}"

    cached = cache.get(cache_key)
    if cached:
        return cached

    repo = ItemRepository(session)
    counts = await repo.count_by_status(project_id)
    cache.set(cache_key, counts)

    return counts
```

### Query Optimization Checklist

- Use pagination for large result sets (limit + offset)
- Add indexes on frequently queried columns
- Select only needed columns, not full objects
- Use bulk operations for multiple inserts/updates
- Eager load relationships to avoid N+1 queries
- Cache results of expensive, frequently-run queries
- Use database-level aggregations instead of Python
- Avoid using functions in WHERE clauses (breaks indexes)
- Use EXPLAIN ANALYZE to understand query performance
- Monitor slow queries and optimize them

## Next Steps

- [Traceability Matrix](/docs/wiki/features/traceability-matrix) - Build comprehensive traceability matrices
- [Impact Analysis](/docs/wiki/features/impact-analysis) - Analyze change impact across items
- [Reporting](/docs/wiki/features/reports) - Generate reports from query results
- [API Reference](/api) - Complete API documentation
