---
title: "Prometheus Monitoring"
description: "Set up Prometheus for monitoring TraceRTM"
keywords: [prometheus, monitoring, metrics, observability]
---

# Prometheus Monitoring

Set up Prometheus to monitor TraceRTM metrics.

## Prometheus Setup

### Docker Compose

```yaml
services:
  prometheus:
    image: prom/prometheus:latest
    ports:
      - "9090:9090"
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
      - prometheus_data:/prometheus
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'
      - '--storage.tsdb.path=/prometheus'

volumes:
  prometheus_data:
```

### Configuration

```yaml
# prometheus.yml
global:
  scrape_interval: 15s
  evaluation_interval: 15s

scrape_configs:
  - job_name: 'tracertm-api'
    static_configs:
      - targets: ['api:8000']
    metrics_path: '/metrics'
```

## Exposing Metrics

### FastAPI Metrics Endpoint

```python
from prometheus_client import Counter, Histogram, generate_latest
from fastapi import Response

# Define metrics
request_count = Counter(
    'http_requests_total',
    'Total HTTP requests',
    ['method', 'endpoint', 'status']
)

request_duration = Histogram(
    'http_request_duration_seconds',
    'HTTP request duration',
    ['method', 'endpoint']
)

# Metrics endpoint
@router.get("/metrics")
async def metrics():
    return Response(
        content=generate_latest(),
        media_type="text/plain"
    )
```

### Instrument Code

```python
from prometheus_client import Counter

item_created = Counter(
    'items_created_total',
    'Total items created'
)

# In handler
@router.post("/api/v1/items")
async def create_item(request: CreateItemRequest):
    item = await item_service.create(request)
    item_created.inc()
    return item
```

## Key Metrics

### Application Metrics

```python
# Request metrics
http_requests_total{method="GET", endpoint="/api/v1/items", status="200"}
http_request_duration_seconds{method="GET", endpoint="/api/v1/items"}

# Business metrics
items_created_total
items_updated_total
links_created_total

# Error metrics
http_errors_total{type="validation_error"}
database_errors_total
```

### System Metrics

```python
# Resource usage
process_cpu_seconds_total
process_resident_memory_bytes
```

## Queries

### PromQL Examples

```promql
# Request rate
rate(http_requests_total[5m])

# Error rate
rate(http_errors_total[5m]) / rate(http_requests_total[5m])

# P95 latency
histogram_quantile(0.95, http_request_duration_seconds_bucket)

# Items created per minute
rate(items_created_total[1m]) * 60
```

## Alerts

### Alert Rules

```yaml
# alerts.yml
groups:
  - name: tracertm
    rules:
      - alert: HighErrorRate
        expr: rate(http_errors_total[5m]) > 0.1
        for: 5m
        annotations:
          summary: "High error rate detected"
      
      - alert: HighLatency
        expr: histogram_quantile(0.95, http_request_duration_seconds_bucket) > 1
        for: 5m
        annotations:
          summary: "P95 latency above 1 second"
```

## Related Documentation

- [Grafana](./04b-grafana) - Visualization with Grafana
- [Logging](./05-logging) - Logging setup
