# Grafana Dashboard Integration Guide

**Version**: 1.0.0
**Last Updated**: 2025-02-05
**Status**: Production Ready

## Table of Contents

1. [Overview and Architecture](#overview-and-architecture)
2. [Prerequisites](#prerequisites)
3. [Installation Steps](#installation-steps)
4. [Dashboard Panel Reference](#dashboard-panel-reference)
5. [Troubleshooting Guide](#troubleshooting-guide)
6. [API Reference](#api-reference)
7. [Integration with Existing Grafana](#integration-with-existing-grafana)
8. [Security Considerations](#security-considerations)
9. [Maintenance Procedures](#maintenance-procedures)

---

## Overview and Architecture

### What is Mahavishnu Monitoring?

Mahavishnu provides comprehensive observability for the MCP ecosystem through:

- **OpenTelemetry Tracing**: Distributed tracing across all MCP servers
- **Prometheus Metrics**: Real-time performance and resource metrics
- **Grafana Dashboards**: Visual monitoring and alerting
- **Log Aggregation**: Centralized logging with Loki (optional)

### Architecture

```
┌─────────────────────────────────────────────────────────────────┐
│                      MCP Servers                                │
│  ┌──────────────┐  ┌──────────────┐  ┌──────────────┐          │
│  │ Mahavishnu   │  │   Akosha     │  │ Session-Buddy│          │
│  │ (Orchestrator)│  │ (Memory)     │  │ (Sessions)   │          │
│  └──────┬───────┘  └──────┬───────┘  └──────┬───────┘          │
│         │                 │                 │                    │
│         │  /metrics       │  /metrics       │  /metrics         │
│         └─────────────────┼─────────────────┘                    │
│                           │                                      │
│                  ┌────────▼────────┐                             │
│                  │   Prometheus    │                             │
│                  │   (Scrapes)     │                             │
│                  └────────┬────────┘                             │
│                           │                                      │
│                  ┌────────▼────────┐                             │
│                  │     Grafana     │                             │
│                  │   (Dashboards)  │                             │
│                  └─────────────────┘                             │
└─────────────────────────────────────────────────────────────────┘
```

### Metrics Collected

#### Request Metrics
- `mcp_http_requests_total` - Total HTTP requests by endpoint, status, method
- `mcp_http_request_duration_seconds` - Request latency histogram
- `mcp_http_requests_in_progress` - Currently active requests

#### MCP Tool Metrics
- `mcp_tool_calls_total` - Tool execution count by tool name, status
- `mcp_tool_duration_seconds` - Tool execution duration histogram
- `mcp_tools_registered` - Number of registered tools

#### Agent/Workflow Metrics
- `agent_tasks_total` - Agent task executions by agent type, status
- `agent_task_duration_seconds` - Task duration histogram
- `pool_workers_active` - Active pool workers by pool type
- `mahavishnu_workflows_executed_total` - Workflows executed by adapter, status

#### Memory Metrics
- `memory_syncs_total` - Memory sync operations by source, status
- `memories_stored` - Total memories in AkOSHA
- `embeddings_generated` - Embeddings generated by model

#### Session Metrics
- `sessions_active` - Currently active sessions
- `session_operations_total` - Session operations by operation, status
- `session_duration_active` - Session duration histogram

#### System Metrics
- `mahavishnu_cpu_usage_percent` - CPU usage percentage
- `mahavishnu_memory_usage_bytes` - Memory usage in bytes
- `mahavishnu_open_files` - Number of open file descriptors

#### Cache Metrics
- `mahavishnu_cache_hits_total` - Cache hits by layer, type
- `mahavishnu_cache_misses_total` - Cache misses by layer, type

---

## Prerequisites

### System Requirements

- **Docker**: 20.10+ and Docker Compose 2.0+
- **Memory**: 4GB RAM minimum (8GB recommended)
- **Disk**: 20GB free space for metrics storage
- **Network**: Ports 3000, 9090, 4317, 4318, 16686 available

### Software Versions

| Component | Minimum Version | Recommended Version |
|-----------|----------------|---------------------|
| Docker | 20.10 | 24.0+ |
| Docker Compose | 2.0 | 2.20+ |
| Grafana | 9.0 | 10.0+ |
| Prometheus | 2.40 | 2.50+ |
| Python | 3.10 | 3.11+ |

### Python Dependencies

```bash
# Core observability dependencies
pip install prometheus-client>=0.19.0
pip install opentelemetry-api>=1.20.0
pip install opentelemetry-sdk>=1.20.0
pip install opentelemetry-exporter-otlp-proto-grpc>=1.20.0
pip install psutil>=5.9.0
```

### Existing Grafana Instance

If you already have Grafana running:

- Grafana 9.0+ installed
- Admin access to provision data sources and dashboards
- Network access from Grafana to Prometheus (default: 9090)

---

## Installation Steps

### Option 1: Full Stack Docker Compose (Recommended for New Installations)

This option deploys the complete monitoring stack including Grafana, Prometheus, and optional components.

#### Step 1: Clone Dashboard Configuration

```bash
cd /Users/les/Projects/mahavishnu

# Dashboard and datasource provisioning
ls -la monitoring/dashboards/
# - mcp_ecosystem.json (main dashboard)

ls -la config/grafana/dashboards/
# - otlp-overview-dashboard.json (OTLP-specific dashboard)
# - dashboards.yml (provisioning config)

ls -la config/clients/
# - grafana-datasources.yml (datasource provisioning)
```

#### Step 2: Configure Prometheus Targets

Edit `monitoring/prometheus.yml` to match your environment:

```yaml
scrape_configs:
  # Mahavishnu - adjust host.docker.internal for your setup
  - job_name: 'mahavishnu'
    static_configs:
      - targets: ['host.docker.internal:8680']  # or actual IP
    metrics_path: '/metrics'
    scrape_interval: 10s

  # Add other MCP servers as needed
  - job_name: 'akosha'
    static_configs:
      - targets: ['host.docker.internal:8682']
    metrics_path: '/metrics'
```

#### Step 3: Start Monitoring Stack

```bash
# Start all monitoring services
cd /Users/les/Projects/mahavishnu
docker-compose -f monitoring/docker-compose.yml up -d

# Verify services are running
docker-compose -f monitoring/docker-compose.yml ps

# Expected output:
# NAME                STATUS              PORTS
# otel-collector      Up                  0.0.0.0:4317-4318->4317-4318/tcp
# prometheus          Up                  0.0.0.0:9090->9090/tcp
# grafana             Up                  0.0.0.0:3000->3000/tcp
# jaeger              Up                  0.0.0.0:16686->16686/tcp
```

#### Step 4: Verify Prometheus is Scraping

```bash
# Check Prometheus targets
curl http://localhost:9090/api/v1/targets | jq '.data.activeTargets[] | {job: .labels.job, health: .health}'

# Expected output:
# {"job": "mahavishnu", "health": "up"}
# {"job": "akosha", "health": "up"}
# {"job": "prometheus", "health": "up"}

# Query metrics directly
curl http://localhost:9090/api/v1/query?query=up | jq '.'
```

#### Step 5: Access Grafana

1. Open browser: http://localhost:3000
2. Default credentials:
   - Username: `admin`
   - Password: `admin`
3. Change password on first login

#### Step 6: Import Dashboard

**Automatic Import** (via provisioning):

Dashboards are automatically imported from:
- `/etc/grafana/provisioning/dashboards/` (in container)
- Mounted from `monitoring/dashboards/` on host

**Manual Import** (if needed):

1. Navigate to Dashboards → Import
2. Upload JSON file:
   - `monitoring/dashboards/mcp_ecosystem.json` (main ecosystem dashboard)
   - `config/grafana/dashboards/otlp-overview-dashboard.json` (OTLP-specific)
3. Select "Prometheus" as data source
4. Click Import

#### Step 7: Verify Dashboard Data

1. Open dashboard: "MCP Ecosystem"
2. Check time range: Last 15 minutes (default)
3. Verify panels show data (not "No Data")
4. Check panel queries return results

---

### Option 2: Integrate with Existing Grafana

Use this option if you already have Grafana and Prometheus running.

#### Step 1: Add Prometheus Datasource

**Via UI**:

1. Grafana → Configuration → Data Sources → Add data source
2. Select "Prometheus"
3. Configure:
   - Name: `Prometheus` (or custom name)
   - URL: `http://prometheus:9090` (adjust for your setup)
   - Access: Server (default) or Browser
4. Click "Save & Test"

**Via Provisioning**:

Add to `/etc/grafana/provisioning/datasources/prometheus.yml`:

```yaml
apiVersion: 1

datasources:
  - name: Prometheus
    type: prometheus
    access: proxy
    url: http://prometheus:9090
    isDefault: true
    editable: true
```

#### Step 2: Configure Prometheus to Scrape Mahavishnu

Add to your `prometheus.yml`:

```yaml
scrape_configs:
  - job_name: 'mahavishnu'
    static_configs:
      - targets: ['localhost:8680']  # Adjust for your setup
    metrics_path: '/metrics'
    scrape_interval: 10s
```

Reload Prometheus:

```bash
# Reload config (without restart)
curl -X POST http://localhost:9090/-/reload

# Or restart Prometheus
docker restart prometheus
```

#### Step 3: Import Dashboard

1. Download dashboard JSON:
   ```bash
   # From repository
   cat /Users/les/Projects/mahavishnu/monitoring/dashboards/mcp_ecosystem.json
   ```

2. Import in Grafana:
   - Dashboards → Import
   - Paste JSON or upload file
   - Select Prometheus datasource
   - Import

---

### Option 3: Quick Start with Docker (Test Environment)

Fastest way to test dashboards without full setup.

```bash
# Start Prometheus and Grafana only
docker run -d \
  --name prometheus \
  -p 9090:9090 \
  -v /Users/les/Projects/mahavishnu/monitoring/prometheus.yml:/etc/prometheus/prometheus.yml \
  prom/prometheus:latest

docker run -d \
  --name grafana \
  -p 3000:3000 \
  -e GF_SECURITY_ADMIN_PASSWORD=admin \
  grafana/grafana:latest

# Import dashboard via UI
# Access: http://localhost:3000 (admin/admin)
```

---

## Dashboard Panel Reference

### MCP Ecosystem Dashboard

**Dashboard File**: `monitoring/dashboards/mcp_ecosystem.json`
**UID**: `mcp-ecosystem-overview`
**Tags**: `mcp`, `mahavishnu`, `monitoring`

#### Panel 1: Request Rate (5m avg)

**Type**: Time Series (Line Chart)
**Query**:
```promql
sum(rate(mcp_http_requests_total[5m])) by (job)
```

**What it shows**: Requests per second over last 5 minutes, grouped by service

**Metrics**:
- X-axis: Time
- Y-axis: Requests per second
- Legend: Job name (mahavishnu, akosha, session-buddy, etc.)

**Thresholds**:
- Green: < 100 req/s
- Yellow: 100-500 req/s
- Red: > 500 req/s

**Use cases**:
- Monitor overall traffic patterns
- Detect traffic spikes
- Identify peak usage times

**Screenshot placeholder**:
```
┌─────────────────────────────────────────┐
│ Request Rate (5m avg)                   │
│ 50 req/s ─────────────                  │
│ 40 req/s ─────  ────                    │
│ 30 req/s ───     ────                   │
│ 20 req/s ──        ───                  │
│ 10 req/s ─           ──                 │
│         ─────────────────               │
│ ▲ mahavishnu  ● akosha  ■ session-buddy│
└─────────────────────────────────────────┘
```

---

#### Panel 2: Request Latency (p50, p95, p99)

**Type**: Time Series (Multiple Lines)
**Query**:
```promql
# p50 (median)
histogram_quantile(0.50, sum(rate(mcp_http_request_duration_seconds_bucket[5m])) by (le, job))

# p95
histogram_quantile(0.95, sum(rate(mcp_http_request_duration_seconds_bucket[5m])) by (le, job))

# p99
histogram_quantile(0.99, sum(rate(mcp_http_request_duration_seconds_bucket[5m])) by (le, job))
```

**What it shows**: Request latency percentiles over time

**Metrics**:
- X-axis: Time
- Y-axis: Latency in milliseconds
- Legend: Percentile (p50, p95, p99)

**Thresholds**:
- Green: < 100ms
- Yellow: 100-500ms
- Red: > 500ms

**Use cases**:
- Identify slow requests
- Track SLA compliance
- Performance regression detection

---

#### Panel 3: Tool Success Rate (%)

**Type**: Stat (Single Value)
**Query**:
```promql
sum(rate(mcp_tool_calls_total{status="success"}[5m])) /
sum(rate(mcp_tool_calls_total[5m])) * 100
```

**What it shows**: Percentage of successful tool executions

**Metrics**:
- Value: Success rate percentage
- Color-coded based on threshold

**Thresholds**:
- Green: > 95%
- Yellow: 90-95%
- Red: < 90%

**Use cases**:
- Overall system health
- Error rate monitoring
- SLA tracking

---

#### Panel 4: Active Workers by Pool Type

**Type**: Time Series (Stacked)
**Query**:
```promql
pool_workers_active
```

**What it shows**: Number of active workers in each pool

**Metrics**:
- X-axis: Time
- Y-axis: Worker count
- Legend: Pool type (mahavishnu, session-buddy, kubernetes)

**Thresholds**:
- Green: > 2 workers
- Yellow: 1-2 workers
- Red: 0 workers

**Use cases**:
- Worker availability monitoring
- Pool capacity planning
- Detect pool exhaustion

---

#### Panel 5: Agent Task Rate

**Type**: Time Series (Bar Chart)
**Query**:
```promql
sum(rate(agent_tasks_total[5m])) by (agent_type)
```

**What it shows**: Agent task execution rate by agent type

**Metrics**:
- X-axis: Time
- Y-axis: Tasks per second
- Legend: Agent type

**Use cases**:
- Agent workload monitoring
- Identify busy agents
- Task distribution analysis

---

#### Panel 6: Memory Usage

**Type**: Gauge (Single Value)
**Query**:
```promql
mahavishnu_memory_usage_bytes / 1024 / 1024 / 1024
```

**What it shows**: Current memory usage in GB

**Metrics**:
- Value: Memory in GB
- Min/Max from time range

**Thresholds**:
- Green: < 4GB
- Yellow: 4-8GB
- Red: > 8GB

**Use cases**:
- Memory leak detection
- Capacity planning
- Resource optimization

---

#### Panel 7: CPU Usage

**Type**: Gauge (Single Value)
**Query**:
```promql
mahavishnu_cpu_usage_percent
```

**What it shows**: Current CPU usage percentage

**Metrics**:
- Value: CPU percentage
- Min/Max from time range

**Thresholds**:
- Green: < 50%
- Yellow: 50-80%
- Red: > 80%

**Use cases**:
- Performance monitoring
- Load balancing decisions
- Resource saturation detection

---

#### Panel 8: Disk Usage

**Type**: Gauge (Single Value)
**Query**:
```promql
system_disk_usage_percent
```

**What it shows**: Disk usage percentage

**Metrics**:
- Value: Disk usage percentage

**Thresholds**:
- Green: < 70%
- Yellow: 70-85%
- Red: > 85%

**Use cases**:
- Capacity planning
- Log rotation triggers
- Storage cleanup

---

### OTLP Overview Dashboard

**Dashboard File**: `config/grafana/dashboards/otlp-overview-dashboard.json`
**UID**: `mahavishnu-otlp-overview`
**Purpose**: OpenTelemetry-specific metrics for distributed tracing

#### Panel 1: Operations Rate

**Type**: Time Series
**Query**:
```promql
rate(operations_total[1m])
```

**What it shows**: Operations per minute by type and status

---

#### Panel 2: Operation Duration (p95)

**Type**: Gauge
**Query**:
```promql
histogram_quantile(0.95, rate(operation_duration_bucket[5m]))
```

**What it shows**: 95th percentile operation duration

---

#### Panel 3: Average Operation Duration

**Type**: Time Series
**Query**:
```promql
rate(operation_duration_sum[5m]) / rate(operation_duration_count[5m])
```

**What it shows**: Average operation duration by operation type

---

#### Panel 4: Active Connections

**Type**: Time Series
**Query**:
```promql
active_connections
```

**What it shows**: Currently active connections by type

---

## Troubleshooting Guide

### Issue: Dashboard Shows "No Data"

**Symptoms**: Panels display "No Data" or empty graphs

**Diagnosis Steps**:

1. **Check Prometheus is scraping**:
   ```bash
   curl http://localhost:9090/api/v1/targets | jq '.data.activeTargets[] | {job: .labels.job, health: .health, lastError: .lastError}'
   ```

2. **Verify metrics endpoint**:
   ```bash
   # Check if Mahavishnu exposes metrics
   curl http://localhost:8680/metrics

   # Expected: Prometheus metrics text format
   # mahavishnu_workflows_executed_total{adapter="prefect",status="success"} 42
   ```

3. **Check Prometheus configuration**:
   ```bash
   # Validate prometheus.yml
   docker exec prometheus promtool check config /etc/prometheus/prometheus.yml

   # Check loaded targets
   curl http://localhost:9090/api/v1/targets
   ```

4. **Query Prometheus directly**:
   ```bash
   # Test query
   curl 'http://localhost:9090/api/v1/query?query=up' | jq '.data.result[]'

   # Check for Mahavishnu metrics
   curl 'http://localhost:9090/api/v1/query?query=mahavishnu_workflows_executed_total' | jq '.'
   ```

**Solutions**:

- **Target not up**: Check Mahavishnu is running and accessible
  ```bash
  # Verify Mahavishnu is running
  mahavishnu mcp status

  # Check port accessibility
  nc -zv localhost 8680
  ```

- **Wrong metrics path**: Ensure `metrics_path: /metrics` in prometheus.yml

- **Network isolation**: Use `host.docker.internal` instead of `localhost` in Docker

- **Firewall blocking**: Open port 8680 or adjust firewall rules

---

### Issue: High Memory Usage in Grafana

**Symptoms**: Grafana container using excessive memory (> 2GB)

**Diagnosis**:

```bash
# Check Grafana memory usage
docker stats grafana --no-stream

# Check dashboard complexity
# Look for:
# - Too many panels (> 20)
# - Complex queries (multiple joins, aggregations)
# - Very short refresh intervals (< 10s)
```

**Solutions**:

1. **Increase refresh interval**:
   - Dashboard settings → Refresh → 30s or 1m

2. **Reduce query time range**:
   - Avoid querying > 30 days of data
   - Use ` $__timeFilter` in queries

3. **Optimize PromQL queries**:
   ```promql
   # BAD: Queries all data
   mahavishnu_workflows_executed_total

   # GOOD: Queries specific time range
   rate(mahavishnu_workflows_executed_total[5m])
   ```

4. **Add recording rules** for expensive queries:
   ```yaml
   # prometheus.yml
   groups:
     - name: mahavishnu_recording
       interval: 30s
       rules:
         - record: job:mahavishnu_workflows_rate:5m
           expr: sum(rate(mahavishnu_workflows_executed_total[5m])) by (job)
   ```

5. **Increase Grafana memory limit**:
   ```yaml
   # docker-compose.yml
   grafana:
     deploy:
       resources:
         limits:
           memory: 4G
   ```

---

### Issue: Metrics Missing After Restart

**Symptoms**: Some metrics don't appear after restarting Mahavishnu

**Diagnosis**:

```bash
# Check metrics endpoint
curl http://localhost:8680/metrics | grep mahavishnu

# Check if metrics are defined in code
grep -r "prom_workflows_total" mahavishnu/
```

**Solutions**:

1. **Ensure metrics are initialized**:
   ```python
   # In mahavishnu/core/observability.py
   from prometheus_client import Counter

   self.prom_workflows_total = Counter(
       "mahavishnu_workflows_executed_total",
       "Total number of workflows executed",
       ["adapter", "status"],
   )
   ```

2. **Check metrics are being incremented**:
   ```python
   # When workflow executes
   self.prom_workflows_total.labels(adapter="prefect", status="success").inc()
   ```

3. **Verify Prometheus scrape interval**:
   - Default: 15s
   - Metrics may take up to 30s to appear

---

### Issue: Dashboard Queries Too Slow

**Symptoms**: Dashboard takes > 10s to load

**Diagnosis**:

```bash
# Check Prometheus query performance
# Grafana → Explore → Query Inspector

# Look for:
# - High execution time (> 5s)
# - Large series count (> 10,000)
# - Complex subqueries
```

**Solutions**:

1. **Add query time limits**:
   ```promql
   # Add timeout to queries
   sum(rate(mcp_http_requests_total[5m])) < 5s
   ```

2. **Reduce data resolution**:
   ```promql
   # Use larger intervals for long time ranges
   rate(mcp_http_requests_total[1h])  # For 7d+ queries
   ```

3. **Pre-aggregate with recording rules**:
   ```yaml
   groups:
     - name: aggregation
       interval: 1m
       rules:
         - record: job:http_requests:rate5m
           expr: sum(rate(mcp_http_requests_total[5m])) by (job)
   ```

4. **Use query caching**:
   - Dashboard settings → Query options → Enable query caching
   - Set cache TTL to 1m

---

### Issue: Incorrect Metric Values

**Symptoms**: Metric values are zero or obviously wrong

**Diagnosis**:

```bash
# Check raw metric values
curl http://localhost:8680/metrics | grep mahavishnu_workflows

# Query Prometheus directly
curl 'http://localhost:9090/api/v1/query?query=mahavishnu_workflows_executed_total' | jq
```

**Solutions**:

1. **Check metric labels**:
   ```promql
   # WRONG: Missing labels
   mahavishnu_workflows_executed_total

   # CORRECT: With required labels
   mahavishnu_workflows_total{adapter="prefect",status="success"}
   ```

2. **Verify counter increment**:
   ```python
   # WRONG: No labels
   self.prom_workflows_total.inc()

   # CORRECT: With labels
   self.prom_workflows_total.labels(adapter="prefect", status="success").inc()
   ```

3. **Check for counter resets**:
   ```promql
   # Use rate() for counters
   rate(mahavishnu_workflows_total[5m])

   # NOT
   mahavishnu_workflows_total  # This will show cumulative total
   ```

---

## API Reference

### Prometheus HTTP API

#### Query Metrics

**Endpoint**: `GET /api/v1/query`

**Parameters**:
- `query`: PromQL expression
- `time`: Unix timestamp (optional, defaults to now)

**Example**:
```bash
curl 'http://localhost:9090/api/v1/query?query=up' | jq
```

**Response**:
```json
{
  "status": "success",
  "data": {
    "resultType": "vector",
    "result": [
      {
        "metric": {"job": "mahavishnu"},
        "value": [1707123456.789, "1"]
      }
    ]
  }
}
```

---

#### Query Range

**Endpoint**: `GET /api/v1/query_range`

**Parameters**:
- `query`: PromQL expression
- `start`: Start timestamp
- `end`: End timestamp
- `step`: Query resolution in seconds

**Example**:
```bash
curl 'http://localhost:9090/api/v1/query_range?query=rate(mahavishnu_workflows_total[5m])&start=1707120000&end=1707123600&step=60' | jq
```

---

#### List Targets

**Endpoint**: `GET /api/v1/targets`

**Example**:
```bash
curl http://localhost:9090/api/v1/targets | jq '.data.activeTargets[] | {job: .labels.job, health: .health}'
```

---

### Custom Metrics API

#### Mahavishnu Metrics Endpoint

**Endpoint**: `GET /metrics`

**Response**: Prometheus text format

```bash
curl http://localhost:8680/metrics
```

**Output**:
```
# HELP mahavishnu_workflows_executed_total Total number of workflows executed
# TYPE mahavishnu_workflows_executed_total counter
mahavishnu_workflows_executed_total{adapter="prefect",status="success"} 42
mahavishnu_workflows_executed_total{adapter="llamaindex",status="success"} 15
mahavishnu_workflows_executed_total{adapter="prefect",status="error"} 2

# HELP mahavishnu_cpu_usage_percent Current CPU usage percentage
# TYPE mahavishnu_cpu_usage_percent gauge
mahavishnu_cpu_usage_percent 35.2
```

---

### PromQL Query Examples

#### Request Metrics

```promql
# Requests per second by service
sum(rate(mcp_http_requests_total[5m])) by (job)

# P95 latency by service
histogram_quantile(0.95, sum(rate(mcp_http_request_duration_seconds_bucket[5m])) by (le, job))

# Error rate by service
sum(rate(mcp_http_requests_total{status=~"5.."}[5m])) by (job)
```

#### Tool Metrics

```promql
# Tool calls per second by tool name
sum(rate(mcp_tool_calls_total[5m])) by (tool_name)

# Tool success rate
sum(rate(mcp_tool_calls_total{status="success"}[5m])) /
sum(rate(mcp_tool_calls_total[5m])) * 100

# Average tool duration
rate(mcp_tool_duration_seconds_sum[5m]) /
rate(mcp_tool_duration_seconds_count[5m])
```

#### Workflow Metrics

```promql
# Workflows per minute by adapter
sum(rate(mahavishnu_workflows_executed_total[5m])) by (adapter)

# Workflow success rate
sum(rate(mahavishnu_workflows_executed_total{status="success"}[5m])) by (adapter) /
sum(rate(mahavishnu_workflows_executed_total[5m])) by (adapter) * 100
```

#### Resource Metrics

```promql
# CPU usage by process
mahavishnu_cpu_usage_percent

# Memory usage in GB
mahavishnu_memory_usage_bytes / 1024 / 1024 / 1024

# Open file descriptors
mahavishnu_open_files
```

#### Cache Metrics

```promql
# Cache hit rate
sum(rate(mahavishnu_cache_hits_total[5m])) /
(sum(rate(mahavishnu_cache_hits_total[5m])) + sum(rate(mahavishnu_cache_misses_total[5m]))) * 100

# Cache operations by layer
sum(rate(mahavishnu_cache_hits_total[5m])) by (cache_layer)
```

---

## Integration with Existing Grafana

### Authentication

#### Basic Authentication

```yaml
# provisioning/datasources/prometheus.yml
apiVersion: 1

datasources:
  - name: Prometheus
    type: prometheus
    access: proxy
    url: http://prometheus:9090
    basicAuth: true
    basicAuthUser: admin
    secureJsonData:
      basicAuthPassword: password
```

#### API Token Authentication

```yaml
datasources:
  - name: Prometheus
    type: prometheus
    access: proxy
    url: http://prometheus:9090
    jsonData:
      httpMethod: POST
      customQueryParameters: "token=your-api-token"
```

---

### Data Source Configuration Options

```yaml
datasources:
  - name: Prometheus
    type: prometheus

    # Connection
    url: http://prometheus:9090
    access: proxy  # or browser (client-side)

    # Time intervals
    jsonData:
      timeInterval: 15s  # Default scrape interval
      queryTimeout: 60s  # Query timeout

    # HTTP settings
    jsonData:
      httpMethod: POST
      tlsSkipVerify: false
      tlsAuth: false
      tlsClientCert: ""
      tlsClientKey: ""

    # Custom headers
    jsonData:
      customHeaders:
        X-Custom-Header: value

    # Exemplars (for tracing integration)
    jsonData:
      exemplarTraceIdLabelName: trace_id
      exemplarSpanIdLabelName: span_id
```

---

### Dashboard Provisioning

Automatically provision dashboards on Grafana startup:

```yaml
# provisioning/dashboards/mahavishnu.yml
apiVersion: 1

providers:
  - name: 'Mahavishnu'
    orgId: 1
    folder: 'Mahavishnu'
    type: file
    options:
      path: /etc/grafana/provisioning/dashboards
    allowUiUpdates: true
    disableDeletion: false
    updateIntervalSeconds: 10
```

Mount dashboard files:

```yaml
# docker-compose.yml
grafana:
  volumes:
    - ./monitoring/dashboards:/etc/grafana/provisioning/dashboards:ro
```

---

### Mixed Data Sources

Use multiple data sources in one dashboard:

```json
{
  "panels": [
    {
      "title": "Mahavishnu Metrics",
      "datasource": "Prometheus",
      "targets": [
        {"expr": "mahavishnu_workflows_total"}
      ]
    },
    {
      "title": "Traces",
      "datasource": "Jaeger",
      "targets": [
        {"refId": "A", "query": "service-name:mahavishnu"}
      ]
    },
    {
      "title": "Logs",
      "datasource": "Loki",
      "targets": [
        {"refId": "A", "expr": "{job=\"mahavishnu\"}"}
      ]
    }
  ]
}
```

---

## Security Considerations

### Authentication and Authorization

#### Grafana Security

```yaml
# docker-compose.yml
grafana:
  environment:
    # Admin credentials
    - GF_SECURITY_ADMIN_USER=admin
    - GF_SECURITY_ADMIN_PASSWORD=changeme

    # Disable public signup
    - GF_USERS_ALLOW_SIGN_UP=false

    # Enable anonymous access (read-only)
    - GF_ANONYMOUS_ENABLED=true
    - GF_ANONYMOUS_ORG_NAME=Main Org.
    - GF_ANONYMOUS_ORG_ROLE=Viewer

    # Session security
    - GF_SESSION_PROVIDER=memory
    - GF_SESSION_PROVIDER_CONFIG=session_life_time=86400
```

#### API Keys

Generate API key for programmatic access:

```bash
# Grafana UI → Configuration → API Keys → Add API Key
curl -X POST http://localhost:3000/api/auth/keys \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer YOUR_ADMIN_API_KEY" \
  -d '{"name": "dashboard-key", "role": "Viewer", "secondsToLive": 86400}'
```

Use API key in requests:

```bash
curl http://localhost:3000/api/dashboards \
  -H "Authorization: Bearer YOUR_API_KEY"
```

---

### Network Security

#### Firewall Rules

```bash
# Allow only Grafana server to access Prometheus
iptables -A INPUT -p tcp --dport 9090 -s grafana-server-ip -j ACCEPT
iptables -A INPUT -p tcp --dport 9090 -j DROP
```

#### TLS/SSL

Enable HTTPS for Grafana:

```yaml
# docker-compose.yml
grafana:
  environment:
    - GF_SERVER_CERT_FILE=/etc/grafana/cert.pem
    - GF_SERVER_CERT_KEY=/etc/grafana/cert-key.pem
  volumes:
    - ./certs:/etc/grafana/certs:ro
```

#### Reverse Proxy

Use Nginx as reverse proxy:

```nginx
# /etc/nginx/sites-available/grafana
server {
    listen 443 ssl;
    server_name grafana.example.com;

    ssl_certificate /etc/ssl/certs/grafana.crt;
    ssl_certificate_key /etc/ssl/private/grafana.key;

    location / {
        proxy_pass http://localhost:3000;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }
}
```

---

### Prometheus Security

#### Basic Authentication

```yaml
# prometheus.yml
global:
  external_labels:
    cluster: 'mcp-ecosystem'

# Basic auth (not recommended for production)
# Use reverse proxy instead
```

Better: Use Nginx reverse proxy:

```nginx
server {
    listen 9090;
    auth_basic "Prometheus";
    auth_basic_user_file /etc/nginx/.htpasswd;

    location / {
        proxy_pass http://localhost:9090;
    }
}
```

#### TLS

Enable TLS for Prometheus:

```yaml
# docker-compose.yml
prometheus:
  command:
    - '--config.file=/etc/prometheus/prometheus.yml'
    - '--web.listen-address=:9090'
    - '--web.config.file=/etc/prometheus/web.yml'
  volumes:
    - ./prometheus-tls.yml:/etc/prometheus/web.yml:ro
```

```yaml
# prometheus-tls.yml
tls_config:
  cert_file: /etc/prometheus/cert.pem
  key_file: /etc/prometheus/cert-key.pem
  client_auth_type: VerifyClientCert
  client_ca_file: /etc/prometheus/ca.pem
```

---

### Metrics Security

#### Redact Sensitive Information

```python
# Don't expose sensitive data in metric labels
# WRONG
counter = Counter("requests_total", "Total requests", ["token", "password"])

# CORRECT
counter = Counter("requests_total", "Total requests", ["endpoint", "method"])
```

#### Access Control

Restrict who can access `/metrics` endpoint:

```python
# mahavishnu/mcp/server_core.py
from fastapi import FastAPI, Request, HTTPException
from fastapi.responses import Response

app = FastAPI()

@app.get("/metrics")
async def metrics(request: Request):
    # Check authorization
    auth_header = request.headers.get("Authorization")
    if auth_header != f"Bearer {os.getenv('METRICS_TOKEN')}":
        raise HTTPException(status_code=403, detail="Forbidden")

    # Return metrics
    from mahavishnu.core.observability import get_observability_manager
    obs = get_observability_manager()
    if obs:
        return Response(content=obs.get_prometheus_metrics(), media_type="text/plain")
    raise HTTPException(status_code=503, detail="Observability not initialized")
```

---

### Audit Logging

Enable Grafana audit logging:

```yaml
# docker-compose.yml
grafana:
  environment:
    - GF_LOG_LEVEL=info
    - GF_LOG_FILTERS=audit:debug
  volumes:
    - ./logs:/var/log/grafana
```

Log format:

```json
{
  "timestamp": "2025-02-05T10:15:30Z",
  "level": "info",
  "message": "Dashboard created",
  "user": "admin",
  "dashboard_id": 123,
  "dashboard_title": "MCP Ecosystem"
}
```

---

## Maintenance Procedures

### Daily Tasks

#### Check Dashboard Health

```bash
# Verify Grafana is running
curl -f http://localhost:3000/api/health || echo "Grafana down"

# Verify Prometheus is scraping
curl http://localhost:9090/api/v1/targets | jq '.data.activeTargets[] | select(.health!="up")'
```

#### Review Alerts

```bash
# Check for fired alerts
curl http://localhost:9090/api/v1/alerts | jq '.data.alerts[] | select(.state=="firing")'
```

---

### Weekly Tasks

#### Review Storage Usage

```bash
# Check Prometheus data size
docker exec prometheus du -sh /prometheus

# Check Grafana database size
docker exec grafana du -sh /var/lib/grafana
```

#### Clean Up Old Metrics

Edit `prometheus.yml`:

```yaml
# Reduce retention time
command:
  - '--storage.tsdb.retention.time=15d'  # Keep 15 days
  - '--storage.tsdb.retention.size=50GB'  # Max 50GB
```

Reload Prometheus:

```bash
curl -X POST http://localhost:9090/-/reload
```

---

### Monthly Tasks

#### Backup Dashboards

```bash
# Export all dashboards
curl http://localhost:3000/api/search?query=&type=dash-db \
  -H "Authorization: Bearer $API_KEY" | jq -r '.[] | .uid' | \
  while read uid; do
    curl "http://localhost:3000/api/dashboards/uid/$uid" \
      -H "Authorization: Bearer $API_KEY" | \
      jq '.dashboard > "backup-$uid.json"'
  done
```

#### Review Query Performance

```bash
# Find slow queries
# Grafana UI → Administration → Stats → Query History

# Look for queries with:
# - High execution time (> 5s)
# - High failure rate
# - Large data volume
```

#### Update Dashboard

```bash
# Pull latest dashboard from repository
cd /Users/les/Projects/mahavishnu
git pull origin main

# Update dashboard files in Grafana
# (automatic via provisioning, or manual import)
```

---

### Backup and Restore

#### Backup Grafana Data

```bash
# Stop Grafana
docker stop grafana

# Backup data volume
docker run --rm \
  -v grafana-storage:/data \
  -v $(pwd)/backups:/backup \
  alpine tar czf /backup/grafana-$(date +%Y%m%d).tar.gz -C /data .

# Start Grafana
docker start grafana
```

#### Restore Grafana Data

```bash
# Stop Grafana
docker stop grafana

# Restore from backup
docker run --rm \
  -v grafana-storage:/data \
  -v $(pwd)/backups:/backup \
  alpine tar xzf /backup/grafana-20250205.tar.gz -C /data

# Start Grafana
docker start grafana
```

#### Backup Prometheus Data

```bash
# Stop Prometheus
docker stop prometheus

# Backup data volume
docker run --rm \
  -v prometheus-data:/data \
  -v $(pwd)/backups:/backup \
  alpine tar czf /backup/prometheus-$(date +%Y%m%d).tar.gz -C /data .

# Start Prometheus
docker start prometheus
```

---

### Monitoring Stack Updates

#### Update Images

```bash
# Pull latest images
docker-compose -f monitoring/docker-compose.yml pull

# Restart with new images
docker-compose -f monitoring/docker-compose.yml up -d
```

#### Rollback Update

```bash
# Pin specific version in docker-compose.yml
grafana:
  image: grafana/grafana:10.0.3  # Pin version

# Restart
docker-compose -f monitoring/docker-compose.yml up -d
```

---

## Appendix

### Quick Reference: Prometheus Ports

| Port | Service | Purpose |
|------|---------|---------|
| 9090 | Prometheus | Web UI and API |
| 3000 | Grafana | Web UI and API |
| 4317 | OTel Collector | OTLP gRPC receiver |
| 4318 | OTel Collector | OTLP HTTP receiver |
| 16686 | Jaeger | Trace UI |
| 8680 | Mahavishnu | MCP server + metrics |

### Quick Reference: Important Files

| File | Purpose |
|------|---------|
| `monitoring/dashboards/mcp_ecosystem.json` | Main dashboard |
| `monitoring/prometheus.yml` | Prometheus config |
| `monitoring/docker-compose.yml` | Monitoring stack |
| `config/grafana/dashboards/dashboards.yml` | Dashboard provisioning |
| `config/clients/grafana-datasources.yml` | Datasource provisioning |
| `mahavishnu/core/observability.py` | Metrics implementation |

### Support and Resources

- **Grafana Documentation**: https://grafana.com/docs/
- **Prometheus Documentation**: https://prometheus.io/docs/
- **PromQL Guide**: https://prometheus.io/docs/prometheus/latest/querying/basics/
- **Mahavishnu Repository**: /Users/les/Projects/mahavishnu

### Changelog

| Version | Date | Changes |
|---------|------|---------|
| 1.0.0 | 2025-02-05 | Initial documentation |
