---
name: observability
source: botcore
description: >
  Implement structured logging, metrics, and distributed tracing.
  Use when adding monitoring, debugging production issues, or improving error visibility.
version: "1.0.0"
triggers:
  - observability
  - logging
  - metrics
  - tracing
  - monitoring
  - OpenTelemetry
  - logs
  - alerts
---

# Observability

Best practices for logging, metrics, and distributed tracing.

## When to Use

- Adding monitoring to a service
- Debugging production issues
- Improving error visibility
- Setting up alerting
- Implementing distributed tracing

## Three Pillars

| Pillar | Purpose | Tools |
|--------|---------|-------|
| **Logs** | Event records | Pino, Winston, structlog |
| **Metrics** | Numeric measurements | Prometheus, StatsD, OpenTelemetry |
| **Traces** | Request flow | Jaeger, Zipkin, OpenTelemetry |

## Structured Logging

```typescript
// ❌ Bad: Unstructured
console.log(`User ${userId} failed to login: ${error}`);

// ✅ Good: Structured
logger.error({
  event: "login_failed",
  userId,
  error: error.message,
  stack: error.stack,
  requestId: ctx.requestId,
});
```

## Log Levels

| Level | Use For |
|-------|---------|
| `error` | Failures requiring attention |
| `warn` | Recoverable issues, deprecations |
| `info` | Business events (user created, payment) |
| `debug` | Development troubleshooting |
| `trace` | Detailed execution flow |

## Correlation IDs

Always propagate request/correlation IDs across services:

```typescript
// Middleware to inject correlation ID
app.use((req, res, next) => {
  req.correlationId = req.headers['x-correlation-id'] || crypto.randomUUID();
  res.setHeader('x-correlation-id', req.correlationId);
  next();
});

// Include in all logs
logger.info({ correlationId: req.correlationId, event: "order_placed" });
```

## Metrics Patterns

```typescript
// Counter - monotonic, always increases
ordersTotal.inc({ status: "completed" });

// Gauge - can go up or down
activeConnections.set(pool.size);

// Histogram - distribution of values
requestDuration.observe(elapsed);
```

## Key Metrics to Track

| Category | Metrics |
|----------|---------|
| **RED** (services) | Rate, Errors, Duration |
| **USE** (resources) | Utilization, Saturation, Errors |
| **Business** | Signups, Orders, Revenue |

## OpenTelemetry Setup

```typescript
import { NodeSDK } from '@opentelemetry/sdk-node';
import { getNodeAutoInstrumentations } from '@opentelemetry/auto-instrumentations-node';

const sdk = new NodeSDK({
  serviceName: 'my-service',
  instrumentations: [getNodeAutoInstrumentations()],
});

sdk.start();
```

## Alerting Guidelines

| Alert Type | Threshold | Response |
|------------|-----------|----------|
| Error rate | > 1% for 5 min | Page on-call |
| Latency p99 | > 2s for 10 min | Investigate |
| Saturation | > 80% for 15 min | Scale up |

## Anti-Patterns

| Avoid | Instead |
|-------|---------|
| Logging PII/secrets | Redact sensitive data |
| High-cardinality labels | Use bounded label values |
| Logging in hot paths | Sample or aggregate |
| Alert fatigue | Actionable alerts only |

## Checklist

- [ ] Structured JSON logging
- [ ] Correlation ID propagation
- [ ] Error context captured (stack, request)
- [ ] Key business metrics instrumented
- [ ] Health check endpoints
- [ ] Alerting configured
- [ ] Dashboards for RED metrics
