# Observability Service

Headless on-prem observability for the agent, oriented to SaaS dashboard UX. Tracks component health, agent metadata, task metrics, and emits reports to Redis for the dashboard to consume.

## Core class
- `ObservabilityService(instance_id, instance_name, version, redis_client, results_stream)`
  - Builds agent metadata (hostname, platform, python version, started_at).
  - Manages component health (`register_component`, `update_component_health`, `get_component_health`, `get_overall_status`).
  - Tracks task metrics (`record_task_start`, `record_task_success`, `record_task_failure`, `get_metrics`).
  - Emits reports (`send_validation_report`, `send_metrics_report`, `start_heartbeat`).

## Data models
- `ComponentStatus`: healthy, degraded, unhealthy, unknown.
- `ComponentHealth`: status, last_check, last_success, consecutive_failures, details, error.
- `TaskMetrics`: counts, errors by type, avg/p95/p99 latency.

## Reporting payloads (published via `redis_client.publish(payload)`)
- Heartbeat (`_build_heartbeat` via `start_heartbeat(interval)`):
  - `{type: "heartbeat", instance_id, instance_name, timestamp, version, uptime_seconds, status, components}`
- Validation report (`send_validation_report`):
  - `{type: "validation_report", instance_id, instance_name, timestamp, version, metadata, components{...}}`
- Metrics report (`send_metrics_report(window)`, default 24h):
  - `{type: "metrics_report", instance_id, timestamp, window_seconds, metrics{totals, errors_by_type, latency}}`
- Error reporting is expected from the worker/connector side using the same Redis channel with `type: "error_report"` (spec in AGENT.md).

## Usage pattern for components/connectors
```python
obs = ObservabilityService(instance_id="worker-1", instance_name="prod-worker-1", version="1.0.0", redis_client=redis)
obs.register_component("database")
obs.register_component("llm")

# On validation
obs.update_component_health("database", ComponentStatus.HEALTHY, {"connected": True, "version": "14.5"})
obs.update_component_health("llm", ComponentStatus.DEGRADED, {"avg_latency_ms": 8200}, error=None)
obs.send_validation_report()
obs.start_heartbeat(interval=30)

# On task execution
obs.record_task_start(task_id, task_type)
try:
    # ... execute task ...
    obs.record_task_success(task_id, execution_time_ms)
except Exception as e:
    obs.record_task_failure(task_id, "db_error", str(e))
```

## Threading
- Heartbeats run on a background thread; call `stop_heartbeat()` on shutdown.
- In-memory metrics and component health are guarded by a lock for simple thread safety.

## Notes
- The Redis client is expected to expose `publish(payload)`; if it requires strings, the service falls back to JSON serialization on AttributeError.
- Short-lived auth/token logic is out of scope here; this module only reports health/metrics. Connector implementations should call `update_component_health` and `record_task_*` appropriately.
