# Metrics Module - Prometheus Observability Infrastructure

The `metrics` module provides centralized Prometheus observability infrastructure for all TASC-Stack services. It enables consistent monitoring across the entire platform by providing pre-defined metric definitions, automatic HTTP instrumentation for FastAPI services, and lightweight metric exposure for background workers.

## Core Philosophy

This module embodies the principle of **unified observability** - ensuring that all services emit consistent, well-labeled metrics that can be aggregated and queried together. Key design goals:

- **Consistency**: All services use the same metric names, labels, and bucket definitions
- **Zero-boilerplate**: Services get automatic HTTP instrumentation with minimal setup
- **Dual-mode exposure**: Support both FastAPI applications and background consumer workers
- **Domain-aware metrics**: Pre-defined metrics for TASC-Stack specific concerns (LLM usage, wake cycles, tasks)

## Key Concepts

### Metric Categories

The module defines metrics organized by domain concern:

| Category | Purpose | Example Metrics |
|----------|---------|-----------------|
| **LLM** | Track AI model usage and costs | Token counts, request duration, errors |
| **Wake Cycle** | Monitor agent execution | Cycle duration, steps, state build/commit times |
| **Task** | Track TASX task execution | Completion counts, duration |
| **Queue/Job** | Monitor Redis job processing | Queue depth, processing duration, retries |
| **Database** | Track PostgreSQL operations | Query duration, pool utilization |
| **Redis** | Monitor cache/queue operations | Command duration, errors |
| **Auth** | Track authentication flows | Attempts, permission checks, errors |
| **Service Calls** | Monitor inter-service communication | Call latency, failures |

**📍 Code Pointers:**
- All metric definitions: `definitions.py` → organized by category with descriptive comments
- Metric types used: `Counter`, `Histogram`, `Gauge` from `prometheus_client`

### Two Exposure Patterns

The module supports two patterns for exposing metrics to Prometheus scrapers:

1. **FastAPI Services** - Use `create_instrumentator()` to automatically instrument HTTP endpoints and expose metrics via `/metrics`

2. **Background Workers** - Use `start_metrics_server()` to run a lightweight HTTP server alongside consumer processes

**📍 Code Pointers:**
- FastAPI instrumentation: `instrumentator.py` → `create_instrumentator()`
- Consumer server: `consumer_server.py` → `start_metrics_server()`

## Architecture Overview

```
┌─────────────────────────────────────────────────────────────────┐
│                    Prometheus Scraper                           │
└─────────────────┬──────────────────────┬───────────────────────┘
                  │                      │
                  ▼                      ▼
    ┌─────────────────────┐   ┌─────────────────────┐
    │   FastAPI Service   │   │   Consumer Worker   │
    │   (/metrics route)  │   │  (metrics server)   │
    └─────────┬───────────┘   └──────────┬──────────┘
              │                          │
              ▼                          ▼
    ┌─────────────────────────────────────────────────┐
    │            common.metrics Module                 │
    │  ┌─────────────────┐  ┌─────────────────────┐  │
    │  │  Instrumentator │  │   Consumer Server   │  │
    │  │  (HTTP metrics) │  │  (lightweight HTTP) │  │
    │  └────────┬────────┘  └──────────┬──────────┘  │
    │           └──────────┬───────────┘              │
    │                      ▼                          │
    │           ┌─────────────────────┐              │
    │           │  Metric Definitions │              │
    │           │   (Counters, etc.)  │              │
    │           └─────────────────────┘              │
    └─────────────────────────────────────────────────┘
```

## Working with Metrics

### Instrumenting a FastAPI Service

Add instrumentation during application startup:

```python
from common.metrics import create_instrumentator

@asynccontextmanager
async def lifespan(app: FastAPI):
    instrumentator = create_instrumentator(service_name="my-service")
    instrumentator.instrument(app).expose(app, include_in_schema=False)
    yield
```

This automatically tracks:
- `http_request_duration_seconds` - Request latency histogram
- `http_requests_total` - Request count by method/handler/status
- `http_requests_inprogress` - Currently processing requests

**📍 Code Pointers:**
- Usage examples: Search for `create_instrumentator` in `api.py` files across services
- Actual implementations: `actx-server/actx_server/api.py`, `tasx-runner/tasx_runner/api.py`

### Exposing Metrics from Consumer Workers

Start a metrics server at the beginning of your consumer process:

```python
from common.metrics import start_metrics_server

if __name__ == "__main__":
    start_metrics_server(port=80)
    # ... rest of consumer code
```

**📍 Code Pointers:**
- Usage examples: `actx-server/consume.py`, `tasx-runner/consume.py`

### Recording Custom Business Metrics

Import and use pre-defined metrics directly:

```python
from common.metrics import (
    LLM_TOKENS_INPUT,
    LLM_REQUEST_DURATION,
    WAKE_CYCLE_DURATION,
)

# Counter - increment
LLM_TOKENS_INPUT.labels(model="claude-3-opus").inc(token_count)

# Histogram - observe
with LLM_REQUEST_DURATION.labels(model="claude-3-opus").time():
    response = await call_llm()

# Or manually observe
WAKE_CYCLE_DURATION.labels(agent_type="althea").observe(duration_seconds)
```

**📍 Code Pointers:**
- LLM metric usage: `common/intel/bot.py`
- Wake cycle metrics: `actx-server/actx_server/intel/agent_session_core.py`
- Task metrics: `tasx-runner/tasx_runner/mixins/task_run.py`
- Database metrics: `common/database/session_manager.py`
- Queue metrics: `common/scheduling/job_handler.py`, `common/scheduling/queue_utils.py`

## Development Guidelines

### Adding New Metrics

When adding new metrics, follow these conventions:

1. **Define in `definitions.py`** - Keep all metric definitions centralized
2. **Use appropriate type**:
   - `Counter` for monotonically increasing values (requests, errors, tokens)
   - `Histogram` for distributions (latencies, durations, sizes)
   - `Gauge` for values that go up and down (queue depth, active sessions)
3. **Choose meaningful labels** - Labels enable slicing; use sparingly (high cardinality = bad)
4. **Set appropriate histogram buckets** - Tailor to expected value ranges
5. **Export from `__init__.py`** - Add to both imports and `__all__`

### Label Best Practices

- Avoid high-cardinality labels (user IDs, request IDs)
- Use consistent label names across related metrics
- Document what each label value represents

### Testing Metrics

Metrics are registered globally with the Prometheus client. In tests:
- Metrics persist across test cases
- Use `prometheus_client.REGISTRY.unregister()` carefully if needed
- Consider reading metric values via `metric._value.get()` for assertions

**📍 Code Pointers:**
- Metric type imports: `prometheus_client` → `Counter`, `Histogram`, `Gauge`
- Instrumentator library: `prometheus_fastapi_instrumentator`

---

This module is the observability backbone of TASC-Stack - properly instrumented services enable debugging, alerting, and capacity planning across the entire platform.
