# Phase 6: Kernel Observability — Research

**Date:** 2026-03-23
**Phase:** 06-kernel-observability
**Requirements:** OBS-01, OBS-02, OBS-03

---

## 1. dmesg Raw Output Format (`dmesg -r`)

The `dmesg -r` (or `dmesg --raw`) command outputs kernel ring buffer lines with a raw syslog priority prefix and a kernel monotonic timestamp.

### Raw format (always available)

```
<priority>[whitespace*seconds.microseconds] message text
```

Examples:
```
<6>[    0.000000] Linux version 6.1.0 (root@build) ...
<4>[   12.345678] WARNING: CPU: 0 PID: 1234 at kernel/sched/core.c:1234
<1>[   12.345679] Kernel panic - not syncing: Fatal exception
<3>[   55.123456] KASAN: slab-out-of-bounds in some_function+0x42/0x100
```

**Parsing regex:** `r'^<(\d+)>\[\s*(\d+\.\d+)\]\s*(.*)'`
- Group 1: syslog priority (decimal integer)
- Group 2: monotonic timestamp (float seconds since boot)
- Group 3: message text

### ISO format (`--time-format=iso`, kernel 3.5+/util-linux 2.22+)

```
<priority>YYYY-MM-DDThh:mm:ss,microseconds+offset message text
```

Example:
```
<6>2024-01-15T10:30:00,000001+0000 Linux version 6.1.0
```

**Parsing regex:** `r'^<(\d+)>(\d{4}-\d{2}-\d{2}T[\d:,]+[+-]\d{4})\s+(.*)'`

**Fallback detection (D-04):** Run `dmesg -r --time-format=iso`. If exit code != 0 or stderr contains "unrecognized option" or "invalid option", the kernel/util-linux version is too old. Fall back to `dmesg -r` with monotonic timestamps. Detection can happen at collection time — no probe needed.

**DmesgLine.timestamp mapping:** Since the model field is `float`, monotonic timestamps map directly. For ISO format, convert using `datetime.fromisoformat()` with comma→period substitution, then `.timestamp()`. This approach is straightforward.

---

## 2. Syslog Priority Decoding

The `<N>` prefix is a combined syslog priority: `priority = facility * 8 + level`.

### Level extraction: `level = priority & 7`

| Value | Name    | DmesgLine.level |
|-------|---------|-----------------|
| 0     | emerg   | `"emerg"`       |
| 1     | alert   | `"alert"`       |
| 2     | crit    | `"crit"`        |
| 3     | err     | `"err"`         |
| 4     | warning | `"warning"`     |
| 5     | notice  | `"notice"`      |
| 6     | info    | `"info"`        |
| 7     | debug   | `"debug"`       |

### Facility extraction: `facility = priority >> 3`

| Value | Name   |
|-------|--------|
| 0     | kern   |
| 1     | user   |
| 2     | mail   |
| 3     | daemon |
| 4     | auth   |
| 5     | syslog |

For kernel dmesg output, facility is almost always `kern` (0). Markers written via `/dev/kmsg` show up as `user` (1) at level `info` (6), so priority = `14`.

---

## 3. /dev/kmsg Marker Insertion

Writing to `/dev/kmsg` inserts a line into the kernel ring buffer. Requires root (CAP_SYSLOG or CAP_SYS_ADMIN).

**Command:** `echo "vmcluster-marker-{uuid4}" > /dev/kmsg`

The marker appears in subsequent `dmesg` output as:
```
<14>[timestamp] vmcluster-marker-a1b2c3d4-e5f6-7890-abcd-ef1234567890
```

Priority 14 = facility=user(1), level=info(6). This is correct — the marker is a user-space informational message.

**Key behavior:** The written string appears verbatim in the dmesg text field. No prefix/suffix is added by the kernel. This means marker detection is a simple substring match: `if marker_string in line_text`.

**Per D-03:** One UUID per `dmesg_mark` call, shared across all nodes. Not per-node.

---

## 4. Kernel Event Classification Patterns

### Confirmed baseline patterns (D-09)

| Classification | Regex | Case | Notes |
|----------------|-------|------|-------|
| `panic` | `r"Kernel panic"` | Sensitive | Always preceded by "Kernel panic - not syncing:" |
| `oops` | `r"Oops:"` | Sensitive | Might also match "Oops: 0000 [#1]" variants |
| `kasan` | `r"KASAN:"` | Sensitive | KASAN report header |
| `ubsan` | `r"UBSAN:"` | Sensitive | UBSAN report header |
| `warn` | `r"WARNING:"` | Sensitive | Kernel WARNING macro output |

### Extended patterns (considered but NOT in v1 baseline per D-09)

These patterns appear in kernel logs but are secondary indicators (follow-up lines after an initial oops/panic):
- `r"BUG:"` — often precedes oops, could be separate classification
- `r"Call Trace:"` — stack trace follows oops/panic
- `r"RIP: \d{4}:"` — instruction pointer dump (x86)
- `r"general protection fault"` — another oops variant
- `r"unable to handle"` — page fault oops variant

**Decision:** Stick with D-09 baseline for v1. The 5 patterns cover the primary signatures. Follow-up lines (Call Trace, RIP, etc.) are captured as `normal` lines and included in the output — the agent can see them in context after the classified header line.

### Priority order (D-08, first match wins)

1. `panic` (highest priority — subsumes everything)
2. `oops`
3. `kasan`
4. `ubsan`
5. `warn`
6. `normal` (default)

A line matching both "Kernel panic" and "WARNING:" would be classified as `panic` since it's checked first. In practice, a single line rarely matches multiple patterns.

---

## 5. Log Level Filtering (D-06)

| filter_level | Included levels |
|-------------|-----------------|
| `"all"` | All 8 levels |
| `"warn+"` | warning, err, crit, alert, emerg (level values 0–4) |
| `"err+"` | err, crit, alert, emerg (level values 0–3) |

Implementation: After parsing `<N>` to extract level value, check against the filter threshold.

```python
LEVEL_VALUES = {"emerg": 0, "alert": 1, "crit": 2, "err": 3, "warning": 4, "notice": 5, "info": 6, "debug": 7}
FILTER_THRESHOLDS = {"all": 7, "warn+": 4, "err+": 3}

# Include line if level_value <= threshold
```

---

## 6. Concurrency and Error Handling

### Fan-out pattern (D-07, D-17)

Same as Phase 3/5: `asyncio.gather(*tasks, return_exceptions=True)` for parallel SSH execution across nodes. Failures on individual nodes are captured and reported — they do NOT abort the entire operation.

### SSHConnectionManager reuse

`observe/__init__.py` creates a module-level `_ssh_manager = SSHConnectionManager()` — same pattern as `exec/__init__.py` and `artifact/__init__.py`. All dmesg SSH commands go through this shared manager, benefiting from connection caching and transparent reconnect.

---

## 7. Existing Model Verification

`DmesgLine` and `DmesgResult` in `vmcluster_mcp/models.py` match the design doc exactly:

```python
class DmesgLine(BaseModel):
    timestamp: float
    level: Literal["debug", "info", "notice", "warning", "err", "crit", "alert", "emerg"]
    facility: str
    text: str
    classified_as: Literal["oops", "panic", "warn", "kasan", "ubsan", "normal"] | None = None

class DmesgResult(BaseModel):
    lines: list[DmesgLine]
    oops_detected: bool
    panic_detected: bool
    warning_count: int
    error_count: int
```

**Confirmed:** No model changes needed for Phase 6.

---

## 8. Module Layout (D-11)

```
vmcluster_mcp/observe/
├── __init__.py      — dmesg_mark + dmesg_collect tool handlers
├── classifier.py    — classify_line() pure function
└── dmesg.py         — insert_marker(), collect_dmesg(), parse_dmesg_line()
```

`server.py` gains `observe` in the import block alongside existing modules.

---

## Validation Architecture

### Required Tests

| Component | Test Focus | File |
|-----------|-----------|------|
| `classifier.py` | classify_line() for each pattern + priority ordering + normal default | `tests/test_classifier.py` |
| `dmesg.py` parse | parse_dmesg_line() for raw and ISO formats, invalid lines | `tests/test_dmesg_mark.py` |
| `dmesg.py` marker | insert_marker() SSH mock, UUID generation | `tests/test_dmesg_mark.py` |
| `dmesg_mark` tool | Tool-level integration with topology loading | `tests/test_dmesg_mark.py` |
| `dmesg_collect` tool | Collection, filtering, classification, marker-not-found | `tests/test_dmesg_collect.py` |

### Verification Commands

```bash
# All phase 6 tests
pytest tests/test_classifier.py tests/test_dmesg_mark.py tests/test_dmesg_collect.py -x -v

# Quick classifier-only
pytest tests/test_classifier.py -x -v
```

---

## RESEARCH COMPLETE

All D-01 through D-18 context decisions confirmed. Key findings:
1. `dmesg -r` parsing is well-defined with regex extraction of priority, timestamp, text
2. Syslog priority decodes cleanly to level (& 7) and facility (>> 3)
3. /dev/kmsg marker insertion is a simple echo, appears verbatim in dmesg
4. Baseline classifier patterns (5 categories + normal) are sufficient for v1
5. ISO timestamp fallback is detectable via exit code on `dmesg --time-format=iso`
6. No model changes needed — DmesgLine/DmesgResult already exist and match design doc
