# Phase 6: Kernel Observability - Context

**Gathered:** 2026-03-23
**Status:** Ready for planning

<domain>
## Phase Boundary

Implement `dmesg_mark` and `dmesg_collect` MCP tools that allow an agent to
bracket kernel operations with named markers and retrieve classified kernel log
output. The classifier detects oops, panics, KASAN/UBSAN events, and warnings
in structured form.

Requirements: OBS-01, OBS-02, OBS-03.

This phase does NOT implement real-time dmesg streaming (deferred to OBS-V2-02)
or per-node CPU/memory/disk metrics (deferred to OBS-V2-01).

</domain>

<decisions>
## Implementation Decisions

### Marker Format and Insertion
- **D-01:** Markers are written via SSH using `echo "vmcluster-marker-{uuid4}" > /dev/kmsg`
  on each target node. The marker string format is:
  `vmcluster-marker-{uuid4}` (e.g., `vmcluster-marker-a1b2c3d4-...`).
  Writing to `/dev/kmsg` requires root access (already guaranteed by SSH user=root).
  `dmesg_mark` returns the exact marker string per node so the caller can pass it
  to `dmesg_collect`.

- **D-02:** `dmesg_mark` accepts `nodes: list[str] | Literal["all"] = "all"` —
  consistent with the design doc interface. When `"all"`, marks all nodes in the
  topology. Returns `dict[str, str]` mapping node name → marker string.

- **D-03:** Each call to `dmesg_mark` generates a **fresh UUID per call** (not per
  node). All nodes in a single `dmesg_mark` call receive the same marker string,
  making it easy to use one marker across a cluster.

### dmesg Collection and Parsing
- **D-04:** Collection uses `dmesg -r` via SSH exec (Phase 3 `SSHConnectionManager`).
  Try `dmesg -r --time-format=iso` first; if the flag is unsupported (older kernels),
  fall back to `dmesg -r` with raw `[seconds.microseconds]` timestamps.
  Researcher confirms the exact fallback detection approach.

- **D-05:** `dmesg_collect` filters lines **after** the first occurrence of the
  `since_marker` string in the dmesg output. Lines before the marker are discarded.
  If the marker is not found, return an empty result with a warning in the
  `ToolResult` error field (non-fatal — `success=True`, but `lines=[]` and a note).

- **D-06:** `filter_level` maps to the design doc values:
  - `"all"` — return all lines
  - `"warn+"` — return lines at `warning`, `err`, `crit`, `alert`, `emerg`
  - `"err+"` — return lines at `err`, `crit`, `alert`, `emerg`
  Level filtering is applied **after** marker-based time-window filtering.

- **D-07:** `dmesg_collect` runs **concurrently across nodes** using `asyncio.gather`
  — consistent with Phase 3/4/5 fan-out pattern. Returns `dict[str, DmesgResult]`
  wrapped in `ToolResult`.

### Kernel Event Classifier
- **D-08:** Classification is **single-pass per line** — each line is checked
  against all patterns in one pass. A line can only have one `classified_as` value
  (first match wins, priority order: panic > oops > kasan > ubsan > warn > normal).

- **D-09:** Baseline regex patterns (researcher confirms/extends):
  - `panic` — matches `r"Kernel panic"` (case-sensitive)
  - `oops` — matches `r"Oops:"` (case-sensitive)
  - `kasan` — matches `r"KASAN:"` (case-sensitive)
  - `ubsan` — matches `r"UBSAN:"` (case-sensitive)
  - `warn` — matches `r"WARNING:"` (case-sensitive)
  - `normal` — everything else

- **D-10:** `DmesgResult.oops_detected`, `panic_detected`, `warning_count`, and
  `error_count` are derived from the classified lines:
  - `oops_detected` = any line with `classified_as == "oops"`
  - `panic_detected` = any line with `classified_as == "panic"`
  - `warning_count` = count of lines with `classified_as == "warn"`
  - `error_count` = count of lines with `level` in `["err", "crit", "alert", "emerg"]`

### Module Structure
- **D-11:** Follow the design doc layout exactly:
  - `vmcluster_mcp/observe/dmesg.py` — SSH-based marker insertion, dmesg collection,
    line parsing (timestamp, level, facility, text extraction)
  - `vmcluster_mcp/observe/classifier.py` — Pattern-matching classifier; pure function,
    no SSH dependency
  - `vmcluster_mcp/observe/__init__.py` — Tool registrations (`dmesg_mark`, `dmesg_collect`)
    using `@mcp.tool()` + `@structured_tool_handler` pattern

- **D-12:** `vmcluster_mcp/server.py` import block gains `observe` alongside
  `artifact`, `cluster`, `exec`, `snapshot` — same lazy-import trigger pattern.

### Models (already defined in models.py)
- **D-13:** `DmesgLine` and `DmesgResult` are **already defined** in `models.py`
  (added during Phase 1 model scaffolding). No new models needed. Researcher
  should verify the existing definitions match the design doc exactly before planning.

### Carried Forward (from prior phases)
- **D-14:** `@structured_tool_handler` wraps every tool handler — no undecorated tools.
- **D-15:** Pydantic v2 models for all return types — `ToolResult[DmesgResult]` per node.
- **D-16:** `asyncio.gather` for parallel fan-out — no `ThreadPoolExecutor` needed
  (no libvirt calls in this phase).
- **D-17:** Best-effort per-node failures with full reporting — a failed `dmesg_collect`
  on one node does not abort others.
- **D-18:** Stateless server — no in-memory marker state. Markers are ephemeral
  strings returned to the caller; the server does not track them.

</decisions>

<canonical_refs>
## Canonical References

**Downstream agents MUST read these before planning or implementing.**

### Design Document
- `docs/vmcluster-mcp-design.md` — Primary design authority. Contains the exact
  tool signatures for `dmesg_mark` and `dmesg_collect`, the `DmesgResult`/`DmesgLine`
  model definitions, and the `observe/` directory layout (section on Kernel Observability
  and section 2.4 data models).

### Requirements
- `.planning/REQUIREMENTS.md` §Kernel Observability — OBS-01, OBS-02, OBS-03 acceptance
  criteria that this phase must satisfy.

### Prior Phase Context (patterns to follow)
- `.planning/phases/05-artifact-pipeline/05-CONTEXT.md` — Concurrency model (D-13),
  best-effort failure semantics (D-08/D-09), tool registration pattern (D-14/D-15).
- `.planning/phases/03-ssh-execution/03-CONTEXT.md` (if exists) — SSH fan-out pattern
  that `dmesg_collect` reuses.

### Existing Code
- `vmcluster_mcp/models.py` — `DmesgLine` and `DmesgResult` already defined; verify
  against design doc before planning.
- `vmcluster_mcp/exec/ssh.py` — `SSHConnectionManager.execute()` is the SSH primitive
  that `dmesg.py` will use for marker insertion and dmesg collection.
- `vmcluster_mcp/server.py` — Tool registration pattern; `observe` import must be
  added to the import block.
- `vmcluster_mcp/artifact/__init__.py` — Reference implementation of the tool
  registration + `@structured_tool_handler` pattern.

</canonical_refs>

<code_context>
## Existing Code Insights

### Reusable Assets
- `SSHConnectionManager` (`vmcluster_mcp/exec/ssh.py`): Provides `execute(ip, port, user, key_path, command)` — used directly for `echo > /dev/kmsg` and `dmesg -r` calls.
- `load_topology` (`vmcluster_mcp/topology/parser.py`): Resolves cluster name → node list with IPs/SSH credentials — same pattern as all prior phases.
- `DmesgLine`, `DmesgResult` (`vmcluster_mcp/models.py`): Already scaffolded; verify field types match design doc.
- `structured_tool_handler` + `mcp` (`vmcluster_mcp/server.py`): Decorator and server instance imported by all tool modules.

### Established Patterns
- **Tool registration**: `@mcp.tool()` + `@structured_tool_handler` on every async def in `__init__.py`
- **Fan-out**: `asyncio.gather(*[per_node_coroutine(node) for node in nodes])` → collect into `dict[str, Result]`
- **Topology resolution**: `load_topology(cluster_name, config.topology_dir)` at the top of each tool handler
- **Error contract**: Return `ToolResult(success=False, error=ErrorDetail(...))` for validation failures; let `@structured_tool_handler` catch unexpected exceptions.

### Integration Points
- `vmcluster_mcp/server.py` import block: add `observe` import
- `vmcluster_mcp/observe/__init__.py`: new file, registers `dmesg_mark` and `dmesg_collect`
- `vmcluster_mcp/observe/dmesg.py`: new file, SSH-based collection logic
- `vmcluster_mcp/observe/classifier.py`: new file, pure classifier

</code_context>

<specifics>
## Specific Ideas

- The design doc explicitly names `/dev/kmsg` as the marker insertion target — not
  `logger` or syslog. This is intentional: `/dev/kmsg` writes appear in dmesg with
  kernel timestamps, making them reliable time anchors.
- The `since_marker` parameter on `dmesg_collect` is optional — callers can collect
  all dmesg without a marker (useful for initial baseline capture).
- `dmesg_mark` returns `dict[str, str]` (node → marker string) per the design doc,
  not a `ToolResult`-wrapped type. Researcher should confirm whether this should be
  wrapped in `ToolResult` for consistency with the error contract (D-14 implies yes).

</specifics>

<deferred>
## Deferred Ideas

- Real-time dmesg streaming during test runs — deferred to OBS-V2-02.
- Per-node CPU/memory/disk metrics — deferred to OBS-V2-01.
- None from discussion — auto mode, no scope creep surfaced.

</deferred>

---

*Phase: 06-kernel-observability*
*Context gathered: 2026-03-23*
*Mode: --auto (all gray areas auto-selected, recommended defaults applied)*
