---
phase: 06-kernel-observability
plan: 02
type: execute
wave: 2
depends_on: ["06-01"]
files_modified:
  - vmcluster_mcp/observe/dmesg.py
  - vmcluster_mcp/observe/__init__.py
  - tests/test_dmesg_collect.py
autonomous: true
requirements:
  - OBS-02
  - OBS-03

must_haves:
  truths:
    - "dmesg_collect returns structured DmesgResult per node with parsed dmesg lines"
    - "Lines before the since_marker are excluded from the result"
    - "filter_level='warn+' includes only warning/err/crit/alert/emerg lines"
    - "filter_level='err+' includes only err/crit/alert/emerg lines"
    - "Each DmesgLine has classified_as set (oops/panic/warn/kasan/ubsan/normal)"
    - "DmesgResult.oops_detected/panic_detected/warning_count/error_count are derived from classified lines"
    - "Missing marker returns success=True with empty lines and a warning note"
    - "Collection runs concurrently across nodes via asyncio.gather"
  artifacts:
    - path: "vmcluster_mcp/observe/dmesg.py"
      provides: "collect_dmesg function with parsing, filtering, classification"
      exports: ["collect_dmesg", "parse_dmesg_line", "insert_marker"]
    - path: "vmcluster_mcp/observe/__init__.py"
      provides: "dmesg_collect tool registration"
      exports: ["dmesg_mark", "dmesg_collect"]
    - path: "tests/test_dmesg_collect.py"
      provides: "dmesg_collect tool tests"
  key_links:
    - from: "vmcluster_mcp/observe/__init__.py"
      to: "vmcluster_mcp/observe/dmesg.py"
      via: "import collect_dmesg"
      pattern: "from.*dmesg.*import.*collect_dmesg"
    - from: "vmcluster_mcp/observe/dmesg.py"
      to: "vmcluster_mcp/observe/classifier.py"
      via: "import classify_line"
      pattern: "from.*classifier.*import.*classify_line"
    - from: "vmcluster_mcp/observe/__init__.py"
      to: "vmcluster_mcp/exec/ssh.py"
      via: "SSHConnectionManager.execute for dmesg command"
      pattern: "_ssh_manager.execute"
---

<objective>
Implement the `dmesg_collect` MCP tool — the second and main kernel observability tool that collects, parses, filters, and classifies dmesg output from cluster nodes.

Purpose: Enables agents to retrieve structured kernel log data since a named marker, with log-level filtering and automatic classification of kernel events (oops, panics, KASAN, UBSAN, warnings). This is the primary observability interface for test workflows.

Output: `collect_dmesg()` in `observe/dmesg.py`, `dmesg_collect` tool in `observe/__init__.py`, comprehensive tests.
</objective>

<execution_context>
@~/.copilot/get-shit-done/workflows/execute-plan.md
@~/.copilot/get-shit-done/templates/summary.md
</execution_context>

<context>
@.planning/PROJECT.md
@.planning/ROADMAP.md
@.planning/STATE.md
@.planning/phases/06-kernel-observability/06-CONTEXT.md
@.planning/phases/06-kernel-observability/06-RESEARCH.md
@.planning/phases/06-kernel-observability/06-01-SUMMARY.md

<interfaces>
<!-- Key types and contracts from Plan 06-01 output + existing codebase -->

From vmcluster_mcp/observe/classifier.py (created in 06-01):
```python
from typing import Literal

ClassificationType = Literal["oops", "panic", "warn", "kasan", "ubsan", "normal"]

def classify_line(text: str) -> ClassificationType:
    """Single-pass classification: panic > oops > kasan > ubsan > warn > normal."""
```

From vmcluster_mcp/observe/dmesg.py (created in 06-01):
```python
def parse_dmesg_line(line: str) -> DmesgLine | None:
    """Parse a raw dmesg -r line into DmesgLine. Returns None for unparseable lines."""

async def insert_marker(ssh_manager, ip, port, user, key_path, marker) -> None:
    """Write marker string to /dev/kmsg via SSH."""
```

From vmcluster_mcp/models.py:
```python
class DmesgLine(BaseModel):
    timestamp: float
    level: Literal["debug", "info", "notice", "warning", "err", "crit", "alert", "emerg"]
    facility: str
    text: str
    classified_as: Literal["oops", "panic", "warn", "kasan", "ubsan", "normal"] | None = None

class DmesgResult(BaseModel):
    lines: list[DmesgLine]
    oops_detected: bool
    panic_detected: bool
    warning_count: int
    error_count: int

class ToolResult(BaseModel, Generic[T]):
    success: bool
    result: T | None = None
    error: ErrorDetail | None = None
```

From vmcluster_mcp/exec/ssh.py:
```python
class SSHConnectionManager:
    async def execute(self, ip: str, port: int, user: str, key_path: str, command: str, timeout_s: int = 30) -> ExecResult:
```

From vmcluster_mcp/observe/__init__.py (created in 06-01):
```python
from vmcluster_mcp.exec.ssh import SSHConnectionManager
from vmcluster_mcp.server import config, mcp, structured_tool_handler
from vmcluster_mcp.topology.parser import load_topology

_ssh_manager = SSHConnectionManager()

@mcp.tool()
@structured_tool_handler
async def dmesg_mark(cluster_name: str, nodes: list[str] | None = None) -> ToolResult[dict[str, str]]:
    ...
```

Level filtering thresholds (from 06-RESEARCH.md):
```python
LEVEL_VALUES = {"emerg": 0, "alert": 1, "crit": 2, "err": 3, "warning": 4, "notice": 5, "info": 6, "debug": 7}
FILTER_THRESHOLDS = {"all": 7, "warn+": 4, "err+": 3}
# Include line if LEVEL_VALUES[line.level] <= FILTER_THRESHOLDS[filter_level]
```
</interfaces>
</context>

<tasks>

<task type="auto" tdd="true">
  <name>Task 1: collect_dmesg function with parsing, filtering, and classification</name>
  <files>vmcluster_mcp/observe/dmesg.py, tests/test_dmesg_collect.py</files>
  <read_first>
    - vmcluster_mcp/observe/dmesg.py (current state from Plan 06-01 — parse_dmesg_line, insert_marker, LEVEL_NAMES, etc.)
    - vmcluster_mcp/observe/classifier.py (classify_line function)
    - vmcluster_mcp/models.py (DmesgLine, DmesgResult field definitions)
    - vmcluster_mcp/exec/ssh.py (SSHConnectionManager.execute signature — returns ExecResult)
    - .planning/phases/06-kernel-observability/06-CONTEXT.md (D-04 through D-07, D-10, D-17)
    - .planning/phases/06-kernel-observability/06-RESEARCH.md (sections 1, 2, 5, 6)
  </read_first>
  <behavior>
    - collect_dmesg with no since_marker: returns all parsed dmesg lines
    - collect_dmesg with since_marker="vmcluster-marker-abc": returns only lines after the marker line
    - collect_dmesg with since_marker not found: returns DmesgResult(lines=[], oops_detected=False, ...)
    - collect_dmesg with filter_level="warn+": only lines at warning/err/crit/alert/emerg
    - collect_dmesg with filter_level="err+": only lines at err/crit/alert/emerg
    - collect_dmesg with filter_level="all": all lines included
    - DmesgResult.oops_detected is True when any line has classified_as=="oops"
    - DmesgResult.panic_detected is True when any line has classified_as=="panic"
    - DmesgResult.warning_count counts lines with classified_as=="warn"
    - DmesgResult.error_count counts lines with level in ("err", "crit", "alert", "emerg")
    - ISO format fallback: if --time-format=iso fails (non-zero exit), retries with plain dmesg -r
    - Unparseable lines are silently skipped (not included in result)
  </behavior>
  <action>
    **Add to `vmcluster_mcp/observe/dmesg.py`:**

    Level filtering constants:
    ```python
    LEVEL_VALUES: dict[str, int] = {"emerg": 0, "alert": 1, "crit": 2, "err": 3, "warning": 4, "notice": 5, "info": 6, "debug": 7}
    FILTER_THRESHOLDS: dict[str, int] = {"all": 7, "warn+": 4, "err+": 3}
    ```

    `collect_dmesg` function:
    ```python
    async def collect_dmesg(
        ssh_manager: SSHConnectionManager,
        ip: str,
        port: int,
        user: str,
        key_path: str,
        since_marker: str | None = None,
        filter_level: str = "all",
    ) -> DmesgResult:
    ```

    Implementation:
    1. **Try ISO format first (D-04):** Run `dmesg -r --time-format=iso` via ssh_manager.execute()
    2. **Fallback detection:** If ExecResult.exit_code != 0, retry with `dmesg -r` (plain raw format)
    3. **Parse output:** Split stdout by newlines, call `parse_dmesg_line()` for each. Skip lines that return `None`.
    4. **Marker filtering (D-05):** If `since_marker` is provided, find the first line whose `text` contains the marker string. Discard that line and all lines before it. If marker is not found, return `DmesgResult(lines=[], oops_detected=False, panic_detected=False, warning_count=0, error_count=0)`.
    5. **Level filtering (D-06):** For each remaining line, include only if `LEVEL_VALUES[line.level] <= FILTER_THRESHOLDS[filter_level]`. Lines with `filter_level="all"` pass everything.
    6. **Build DmesgResult (D-10):**
       - `oops_detected = any(l.classified_as == "oops" for l in lines)`
       - `panic_detected = any(l.classified_as == "panic" for l in lines)`
       - `warning_count = sum(1 for l in lines if l.classified_as == "warn")`
       - `error_count = sum(1 for l in lines if l.level in ("err", "crit", "alert", "emerg"))`

    **Create `tests/test_dmesg_collect.py`:**

    Test collect_dmesg with mocked SSHConnectionManager:
    - Basic collection: mock SSH returns multi-line dmesg output, verify parsed DmesgLine list
    - Marker filtering: output contains marker line, verify only post-marker lines returned
    - Marker not found: verify empty DmesgResult returned
    - Level filtering warn+: mock output has info, warning, err lines — verify only warning+err returned
    - Level filtering err+: verify only err/crit/alert/emerg returned
    - Classification integration: output with "WARNING:", "KASAN:", "Oops:", "Kernel panic" lines — verify classified_as fields
    - DmesgResult summary: oops_detected, panic_detected, warning_count, error_count derived correctly
    - ISO fallback: first SSH call returns exit_code=1, second returns exit_code=0 with raw output — verify retry happened
    - Unparseable lines skipped: mix of valid and garbage lines, verify garbage not in result
  </action>
  <verify>
    <automated>cd /home/hornc/repos/vmcluster-mcp && python -m pytest tests/test_dmesg_collect.py -x -v</automated>
  </verify>
  <acceptance_criteria>
    - `vmcluster_mcp/observe/dmesg.py` contains `async def collect_dmesg(`
    - `vmcluster_mcp/observe/dmesg.py` contains `LEVEL_VALUES` and `FILTER_THRESHOLDS` dicts
    - collect_dmesg tries `dmesg -r --time-format=iso` first (grep for `--time-format=iso` in dmesg.py)
    - collect_dmesg falls back to `dmesg -r` on non-zero exit code
    - Marker filtering: lines before marker are excluded
    - Missing marker returns DmesgResult with empty lines list
    - `tests/test_dmesg_collect.py` exists with at least 8 test functions
    - `pytest tests/test_dmesg_collect.py` exits 0
  </acceptance_criteria>
  <done>collect_dmesg parses dmesg output, filters by marker and level, classifies lines, derives summary fields, handles ISO fallback — all tests pass</done>
</task>

<task type="auto">
  <name>Task 2: Wire dmesg_collect tool + integration tests</name>
  <files>vmcluster_mcp/observe/__init__.py, tests/test_dmesg_collect.py</files>
  <read_first>
    - vmcluster_mcp/observe/__init__.py (current state from Plan 06-01 — has dmesg_mark, need to add dmesg_collect)
    - vmcluster_mcp/observe/dmesg.py (collect_dmesg signature from Task 1)
    - vmcluster_mcp/exec/__init__.py (node_exec_all pattern for parallel fan-out with per-node error handling)
    - vmcluster_mcp/models.py (DmesgResult, ErrorDetail, ToolResult)
    - .planning/phases/06-kernel-observability/06-CONTEXT.md (D-05 warning on missing marker, D-07 concurrent gather, D-17 best-effort)
  </read_first>
  <action>
    **Add `dmesg_collect` tool to `vmcluster_mcp/observe/__init__.py`:**

    ```python
    @mcp.tool()
    @structured_tool_handler
    async def dmesg_collect(
        cluster_name: str,
        nodes: list[str] | None = None,
        since_marker: str | None = None,
        filter_level: str = "all",
    ) -> ToolResult[dict[str, DmesgResult]]:
    ```

    Implementation:
    1. Load topology: `topo = load_topology(cluster_name, config.topology_dir)`
    2. Resolve target nodes (same pattern as dmesg_mark): None → all nodes, otherwise validate names exist
    3. Validate `filter_level` is one of `"all"`, `"warn+"`, `"err+"` — return `ToolResult(success=False, error=ErrorDetail(code="INVALID_FILTER_LEVEL", ...))` if not
    4. Fan-out with `asyncio.gather(*[collect_dmesg(_ssh_manager, node.ip, 22, topo.ssh.user, topo.ssh.key_path, since_marker, filter_level) for node in target_nodes], return_exceptions=True)`
    5. Build result dict: `{node.name: dmesg_result}` for successful results
    6. Per-node error handling (D-17): if a gather result is an Exception, record it as a DmesgResult with empty lines or skip it and note in a separate errors dict. Simplest: on exception, create DmesgResult(lines=[], oops_detected=False, panic_detected=False, warning_count=0, error_count=0) and log the error. Alternatively, return partial results with success=True if at least one node succeeded.
    7. Return `ToolResult(success=True, result=results_dict)`

    **Add integration-style tests to `tests/test_dmesg_collect.py`:**

    Test the dmesg_collect TOOL (not just collect_dmesg function):
    - Tool returns `ToolResult[dict[str, DmesgResult]]` for a 2-node cluster
    - Tool with `nodes=["node1"]` only collects from node1
    - Tool with `since_marker` passes marker through to collect_dmesg
    - Tool with invalid `filter_level` returns `ToolResult(success=False)`
    - Tool with invalid node name returns `ToolResult(success=False)` with NODE_NOT_FOUND
    - Tool handles SSH failure on one node gracefully (D-17)
    - Tool with `filter_level="warn+"` returns only warning+ lines per node

    Import `dmesg_collect` from `vmcluster_mcp.observe` for tool-level tests.
  </action>
  <verify>
    <automated>cd /home/hornc/repos/vmcluster-mcp && python -m pytest tests/test_classifier.py tests/test_dmesg_mark.py tests/test_dmesg_collect.py -x -v</automated>
  </verify>
  <acceptance_criteria>
    - `vmcluster_mcp/observe/__init__.py` contains `@mcp.tool()` before `async def dmesg_collect(`
    - `vmcluster_mcp/observe/__init__.py` contains `@structured_tool_handler` before `async def dmesg_collect(`
    - `vmcluster_mcp/observe/__init__.py` imports `collect_dmesg` from `vmcluster_mcp.observe.dmesg`
    - `vmcluster_mcp/observe/__init__.py` imports `DmesgResult` from `vmcluster_mcp.models`
    - dmesg_collect tool uses `asyncio.gather` for parallel collection (grep for `asyncio.gather` in __init__.py)
    - `tests/test_dmesg_collect.py` has tool-level tests (testing `dmesg_collect` imported from observe)
    - `pytest tests/test_classifier.py tests/test_dmesg_mark.py tests/test_dmesg_collect.py` exits 0 — ALL phase 6 tests green
  </acceptance_criteria>
  <done>dmesg_collect MCP tool registered, collects dmesg concurrently across nodes, filters by marker and level, returns dict[str, DmesgResult] wrapped in ToolResult, handles per-node failures gracefully, all phase 6 tests pass</done>
</task>

</tasks>

<verification>
```bash
# All Phase 6 tests pass
cd /home/hornc/repos/vmcluster-mcp && python -m pytest tests/test_classifier.py tests/test_dmesg_mark.py tests/test_dmesg_collect.py -x -v

# Full test suite still passes (no regressions)
python -m pytest --tb=short -q

# Server imports cleanly with observe module
python -c "from vmcluster_mcp.server import mcp; print('server OK')"

# Both tools are registered
python -c "from vmcluster_mcp import observe; print('observe tools loaded')"
```
</verification>

<success_criteria>
1. `dmesg_collect` returns structured `DmesgResult` per node with parsed, filtered, classified lines
2. Marker-based filtering excludes all lines before `since_marker`
3. Level filtering (`"all"`, `"warn+"`, `"err+"`) correctly filters by syslog level
4. Classification fields (oops_detected, panic_detected, warning_count, error_count) are derived from lines
5. ISO timestamp fallback works (tries `--time-format=iso`, falls back to raw)
6. Concurrent collection across nodes via `asyncio.gather`
7. Per-node failures don't abort the entire operation
8. All phase 6 tests pass, full test suite has no regressions
</success_criteria>

<output>
After completion, create `.planning/phases/06-kernel-observability/06-02-SUMMARY.md`
</output>
