# Observability Module

## Overview

`aceteam_aep.observability` is a database-backed event store for proxy telemetry. Every LLM call flowing through the proxy emits structured events (call start/end, safety signals, enforcement decisions, cost). Flagged calls (FLAG/BLOCK) additionally store the full request/response payload for incident review and RLHF training.

Two storage backends share the same schema:
- **SQLite** (standalone) — `~/.config/aceteam-aep/observability.db`, auto-created on first proxy start
- **Postgres** (hosted) — when used as a library in the AceTeam platform, events write to Supabase

## Event Lifecycle

The proxy handler emits events at five points:

```
Request arrives
  ├─ call_start        (call_id, model, provider, timestamp)
  ├─ safety_signal     (zero or more — one per detector that fires)
  ├─ enforcement       (the PASS/FLAG/BLOCK decision)
  ├─ call_end          (latency_ms, tokens_in, tokens_out)
  └─ cost              (cost_usd from CostTracker)
```

All writes are fire-and-forget — `await store.record(event)` never blocks the response path. Failures log and move on.

On FLAG or BLOCK, the proxy additionally writes the full message payload to `flagged_calls`. PASS calls are not stored.

## Data Model

### `events` table

Append-only, lightweight metrics. No message content.

| Column | Type | Notes |
|--------|------|-------|
| id | TEXT PK | UUID |
| timestamp | TEXT | ISO 8601 |
| session_id | TEXT | Groups events within one proxy lifetime |
| type | TEXT | `call_start`, `call_end`, `safety_signal`, `enforcement`, `cost` |
| call_id | TEXT | Ties related events together |
| model | TEXT | e.g. `gpt-4o`, `claude-sonnet-4-20250514` |
| provider | TEXT | `openai`, `anthropic`, `google`, `xai`, `ollama` |
| tokens_in | INTEGER | Input tokens (call_end only) |
| tokens_out | INTEGER | Output tokens (call_end only) |
| cost_usd | REAL | Cost for this call |
| latency_ms | REAL | Request duration (call_end only) |
| action | TEXT | `pass`, `flag`, `block` (enforcement only) |
| detector | TEXT | Which detector fired (safety_signal only) |
| severity | TEXT | `low`, `medium`, `high` |
| reason | TEXT | Human-readable explanation |
| metadata | TEXT | JSON blob for extensibility |

Indexed on: `session_id`, `type`, `call_id`, `timestamp`.

### `flagged_calls` table

Written only on FLAG or BLOCK. Contains full conversation payload.

| Column | Type | Notes |
|--------|------|-------|
| id | TEXT PK | UUID |
| call_id | TEXT | Joins to events |
| session_id | TEXT | Session grouping |
| timestamp | TEXT | ISO 8601 |
| action | TEXT | `flag` or `block` |
| detector | TEXT | Primary detector |
| severity | TEXT | Signal severity |
| reason | TEXT | Why it was flagged |
| model | TEXT | LLM model used |
| input_messages | TEXT | Full JSON messages array |
| output_text | TEXT | Full response (NULL if blocked pre-LLM) |
| verdict | TEXT | NULL, `confirmed`, or `dismissed` |
| verdict_by | TEXT | Reviewer identity |
| verdict_at | TEXT | Review timestamp |
| verdict_note | TEXT | Reviewer's comment |
| metadata | TEXT | JSON: tokens, cost, policy snapshot |

Indexed on: `session_id`, `verdict`, `detector`.

## Storage Protocol

```python
class EventStore(Protocol):
    async def record(self, event: ObservabilityEvent) -> None: ...
    async def record_flagged_call(self, call: FlaggedCall) -> None: ...
    async def query_events(self, *, session_id=None, type=None,
                           since=None, until=None, call_id=None,
                           limit=1000) -> list[ObservabilityEvent]: ...
    async def query_flagged_calls(self, *, session_id=None,
                                  verdict=None, limit=100) -> list[FlaggedCall]: ...
    async def update_verdict(self, call_id, verdict, verdict_by,
                             verdict_note=None) -> None: ...
```

`SqliteEventStore` uses aiosqlite. `PostgresEventStore` uses the Supabase Python client.

## Views

Four query projections are built on top of the event store:

### Flight Recorder (P0)
Always-on incident capture. Queries `flagged_calls` for the review queue, enriches with surrounding `events` for context. Supports export as a zip bundle (summary, payload, context events, policy snapshot).

### Timeline (P1)
`call_start`/`call_end` pairs become horizontal bars, `safety_signal`/`enforcement` events overlay as markers. Adjustable time windows (1m–session).

### Traffic Stats (P2)
SQL aggregations over `events`: request rate, latency percentiles, token throughput, cost rate, per-model breakdown.

### Topology (P3)
Groups events by `provider` and `session_id` to produce agent→gateway→provider graph with health indicators.

## Dashboard Integration

The standalone proxy dashboard gains new routes:

```
GET   /dashboard/api/incidents
GET   /dashboard/api/incidents/{call_id}
GET   /dashboard/api/incidents/{call_id}/export
PATCH /dashboard/api/incidents/{call_id}/verdict
GET   /dashboard/api/timeline
GET   /dashboard/api/traffic
GET   /dashboard/api/topology
```

The `top.py` terminal UI gains an incident list view.

## Replaces

This module supersedes the file-based `aep-feedback.jsonl` approach in `feedback.py`. Existing feedback data can be migrated into the `flagged_calls` table.

## Configuration

No configuration required. The SQLite database is created automatically. To disable event recording, set `AEP_OBSERVABILITY=off`.
