# ADR-018: Filter-Only Dimensions in Analytics Framework

**Date:** 2025-12-31
**Status:** Accepted
**Affects:** EPIC-007 (Generic Analytics Framework), testio-agui Product Overview
**Related:** ADR-006 (Service Layer Pattern), ADR-007 (FastMCP Context Injection)

## Context

The analytics framework (`query_metrics`) currently requires dimensions to be used in GROUP BY before they can be used in filters. This creates a limitation:

1. **Current Behavior:** You can only filter by `severity` if you're also grouping by `severity`
2. **Desired Behavior:** Filter by `severity` while grouping by something else (e.g., `feature`)

### Problem Statement

The QueryBuilder validates filters against grouped dimensions only:

```python
# query_builder.py:153 - Current restrictive validation
known_filters = {"product_id", "status"} | {d.key for d in self.dimensions}
```

This prevents queries like:
```python
# FAILS: severity not in dimensions
query_metrics(
    metrics=["bug_count"],
    dimensions=["feature"],
    filters={"product_id": 18559, "severity": "critical"}
)
```

### Use Cases (testio-agui)

1. **Product Overview:** User clicks "Coverage" test type + "Login" feature to see:
   - Tests tab: Coverage tests that included Login
   - Bugs tab: Bugs from those tests for Login feature
   - Features tab: Login details with metrics

2. **Cross-Entity Filtering:** Filter by `rejection_reason` while grouping by `feature`:
   ```python
   query_metrics(
       metrics=["bug_count"],
       dimensions=["feature"],
       filters={"rejection_reason": "intended_behavior"}
   )
   ```

## Decision

**Support "filter-only dimensions" - dimensions used in WHERE without being in GROUP BY.**

### Key Changes

| Aspect | Before | After |
|--------|--------|-------|
| **Filter validation** | Only grouped dimensions allowed | Any registered dimension allowed |
| **Join logic** | Only grouped dimensions joined | Filter-only dimensions also joined |
| **WHERE clauses** | Only grouped dimensions filtered | Filter-only dimensions also filtered |

### Implementation Pattern

```
┌─────────────────────────────────────────────────────────────────┐
│ QueryBuilder                                                     │
│                                                                  │
│ 1. Validate filters: Allow any dimension in registry            │
│ 2. Add INNER JOINs for grouped dimensions (GROUP BY grain)      │
│ 3. Add LEFT JOINs for metrics (preserve zero counts)            │
│ 4. Add LEFT JOINs for filter-only dimensions (NEW)              │
│ 5. Apply WHERE clauses for all filters (grouped + filter-only)  │
│                                                                  │
└─────────────────────────────────────────────────────────────────┘
```

### Why LEFT JOIN for Filter-Only Dimensions?

Filter-only dimensions don't define the grouping grain. Using LEFT JOIN ensures:
- The grouped dimension still controls row granularity
- Filter-only dimensions just add WHERE constraints
- Zero counts are preserved for grouped entities without matching filter values

### Code Changes

**QueryBuilder constructor - accept dimension registry:**
```python
def __init__(
    self,
    session: AsyncSession,
    customer_id: int,
    dimension_registry: dict[str, "DimensionDef"] | None = None,
):
    self._dimension_registry = dimension_registry or {}
```

**Relax filter validation:**
```python
# Allow any registered dimension as filter
allowed_filter_keys = {"product_id", "status", "testing_type"}
if self._dimension_registry:
    allowed_filter_keys |= set(self._dimension_registry.keys())
```

**Add filter-only JOINs:**
```python
grouped_dim_keys = {d.key for d in self.dimensions}
for filter_key in self.filters:
    if filter_key not in grouped_dim_keys:
        filter_dim = self._dimension_registry.get(filter_key)
        if filter_dim:
            stmt = self._add_joins(stmt, filter_dim.join_path, joined_models, join_type="left")
```

**Apply filter-only WHERE clauses:**
```python
for filter_key, filter_value in self.filters.items():
    if filter_key not in grouped_dim_keys:
        filter_dim = self._dimension_registry.get(filter_key)
        if filter_dim:
            if isinstance(filter_value, (list, tuple, set)):
                where_clauses.append(filter_dim.column.in_(filter_value))
            else:
                where_clauses.append(filter_dim.column == filter_value)
```

## Rationale

### Why Now?

The testio-agui Product Overview requires cross-entity filtering that the current analytics framework cannot support. This blocks the frontend development.

### Why Not Separate "Filterable Dimensions" Registry?

Considered maintaining a separate registry of dimensions that can be filtered without grouping. Rejected because:
- **Duplication:** Same dimension definitions in two places
- **Maintenance burden:** Keep them in sync
- **Unnecessary complexity:** All dimensions should be filterable by default

### Why Not Query Rewriting?

Considered rewriting queries to add dimensions to GROUP BY automatically:
```python
# Auto-expand: filters={"severity": "critical"} → dimensions=["feature", "severity"]
```

Rejected because:
- **Changes semantics:** User asked for feature-level aggregation, not feature×severity
- **Confusing results:** More rows than expected
- **Breaks expectations:** Hides what the query actually does

## Consequences

### Positive

- **Flexible filtering:** Filter by any dimension without changing grouping
- **Backwards compatible:** Existing queries work unchanged
- **No schema changes:** Uses existing dimension registry
- **Minimal code changes:** ~50 lines in QueryBuilder + ~10 lines in AnalyticsService

### Negative

- **Slightly more complex QueryBuilder:** Must track grouped vs filter-only dimensions
- **Potential performance impact:** Additional JOINs for filter-only dimensions
  - **Mitigation:** LEFT JOINs are efficient when indexed; dimensions already have proper indexes

### Neutral

- **No breaking changes to API:** `query_metrics` signature unchanged
- **No migration required:** Existing queries continue to work

## Implementation

### Files Changed

| File | Change |
|------|--------|
| `src/testio_mcp/services/query_builder.py` | Add dimension_registry, filter-only logic |
| `src/testio_mcp/services/analytics_service.py` | Pass dimension_registry to QueryBuilder |
| `tests/services/test_analytics_service.py` | Test filter-only dimensions |
| `tests/unit/test_query_builder.py` | Unit tests for filter validation |

### Example Queries After Implementation

```python
# Filter by severity while grouping by feature
query_metrics(
    metrics=["bug_count"],
    dimensions=["feature"],
    filters={"product_id": 18559, "severity": "critical"}
)

# Filter by rejection_reason while grouping by tester
query_metrics(
    metrics=["bug_count"],
    dimensions=["tester"],
    filters={"product_id": 18559, "rejection_reason": "no_reproduction"}
)

# Filter by feature while grouping by month (time series)
query_metrics(
    metrics=["bug_count"],
    dimensions=["month"],
    filters={"product_id": 18559, "feature": "Login"}
)
```

## References

- **Feature Request:** testio-agui Product Overview cross-entity filtering
- **Related Tools:** `list_bugs`, `list_tests` (separate feature_ids filter implementation)
- **Dimension Registry:** `src/testio_mcp/services/analytics_service.py:build_dimension_registry()`

---

**Summary:** Enable filtering by any registered dimension in query_metrics, regardless of whether that dimension is in the GROUP BY clause. This supports flexible cross-entity filtering needed by the testio-agui Product Overview.
