# Monitoring Stack Template

Production-ready monitoring stack for Aut-O projects, implementing T-12 (World-Class Logging & Monitoring).

## Quick Start

```bash
# Copy this directory into your project
cp -r templates/monitoring/ ./monitoring/

# Set Grafana admin password
export GRAFANA_ADMIN_PASSWORD="your-secure-password"

# Start the stack
docker compose -f monitoring/docker-compose.monitoring.yml up -d

# Verify all services are healthy
docker compose -f monitoring/docker-compose.monitoring.yml ps
```

## Services

| Service | Port | URL | Purpose |
|---------|------|-----|---------|
| Prometheus | 9090 | http://localhost:9090 | Metrics collection and alerting engine |
| Grafana | 3001 | http://localhost:3001 | Dashboards and visualization |
| Loki | 3100 | http://localhost:3100 | Log aggregation |
| Promtail | 9080 | — | Log collection agent |
| AlertManager | 9093 | http://localhost:9093 | Alert routing and notification |
| Node Exporter | 9100 | http://localhost:9100 | Host-level metrics |

Grafana runs on port 3001 (not 3000) to avoid conflicts with your application.

## Connecting Your Application

### 1. Expose a /metrics Endpoint

Your app must expose Prometheus-format metrics at `/metrics` (T-12 requirement). Example with `prom-client` (Node.js):

```typescript
import { collectDefaultMetrics, Registry, Counter, Histogram } from "prom-client";

const register = new Registry();
collectDefaultMetrics({ register });

const httpRequests = new Counter({
  name: "http_requests_total",
  help: "Total HTTP requests",
  labelNames: ["method", "route", "status"],
  registers: [register],
});

const httpDuration = new Histogram({
  name: "http_request_duration_seconds",
  help: "HTTP request duration in seconds",
  labelNames: ["method", "route", "status"],
  buckets: [0.01, 0.05, 0.1, 0.5, 1, 2, 5],
  registers: [register],
});

// Expose /metrics endpoint
app.get("/metrics", async (_req, res) => {
  res.set("Content-Type", register.contentType);
  res.end(await register.metrics());
});
```

### 2. Update Prometheus Scrape Target

In `prometheus/prometheus.yml`, update the app job target to match your app's host and port:

```yaml
- job_name: "app"
  static_configs:
    - targets: ["host.docker.internal:3000"]
```

If your app runs in the same Docker Compose network, use the service name instead:

```yaml
    - targets: ["your-app-service:3000"]
```

### 3. Health Endpoints

Every service must expose (T-12):

- `GET /health` — Liveness check
- `GET /ready` — Readiness check (dependencies healthy)
- `GET /metrics` — Prometheus-format metrics

## Adding Custom Dashboards

Place JSON dashboard files in a dashboards directory and mount it into Grafana:

```yaml
# In docker-compose.monitoring.yml, add to grafana volumes:
- ./grafana/dashboards:/var/lib/grafana/dashboards:ro
```

Export dashboards from the Grafana UI (Share > Export > Save to file) and place them in `grafana/dashboards/`.

Community dashboards: search [Grafana Dashboards](https://grafana.com/grafana/dashboards/) and import by ID within the Grafana UI.

## Configuring Real Alerting

The AlertManager ships with placeholder receivers. Replace them with your actual endpoints:

### Email

In `alertmanager/alertmanager.yml`, uncomment and configure the global SMTP settings:

```yaml
global:
  smtp_smarthost: "smtp.gmail.com:587"
  smtp_from: "alerts@yourdomain.com"
  smtp_auth_username: "alerts@yourdomain.com"
  smtp_auth_password: "app-password"
```

### Slack

Replace the webhook URL in the `critical-slack` receiver:

```yaml
slack_configs:
  - api_url: "https://hooks.slack.com/services/YOUR/ACTUAL/WEBHOOK"
    channel: "#your-alerts-channel"
```

### After Changes

Reload AlertManager without restart:

```bash
curl -X POST http://localhost:9093/-/reload
```

## Integrating with an Existing Docker Compose

To add monitoring to your project's existing `docker-compose.yml`:

```yaml
# In your project's docker-compose.yml, add:
networks:
  monitoring:
    external: true
    name: monitoring

# Then add the monitoring network to services you want to scrape:
services:
  your-app:
    networks:
      - default
      - monitoring
```

Start monitoring first, then your app:

```bash
docker compose -f monitoring/docker-compose.monitoring.yml up -d
docker compose up -d
```

## Data Retention

| Component | Retention | Config Location |
|-----------|-----------|-----------------|
| Prometheus | 30 days | `--storage.tsdb.retention.time=30d` in docker-compose |
| Loki | 30 days | `retention_period: 720h` in loki-config.yml |
| Grafana | Persistent | grafana_data volume |

## Troubleshooting

**Prometheus can't scrape app metrics**: Verify the target is reachable from the Prometheus container. Check `http://localhost:9090/targets` for scrape errors.

**Promtail not collecting logs**: Ensure `/var/lib/docker/containers` is mounted and Docker uses the `json-file` log driver.

**Grafana shows "No data"**: Verify the datasource is configured correctly at `http://localhost:3001/connections/datasources`.

**Node Exporter metrics missing**: Node Exporter uses `network_mode: host` — firewall rules may need port 9100 open locally.
