# runtime inspector

## mission

- Turn runtime behavior into evidence, not guesswork.
- Focus on call stack, timeline, kernel timing, and memory footprint.

## tools

- bash cli
  - `py-spy`: python/c++ mixed call stack on live process.
- python pkg
  - `zhijiang.ai_tools`
    - list all called cuda kernels: `zhijiang_pytorch_profiler`
    - benchmark to get cuda kernel timing: `zhijiang_do_bench`
    - print all used python pkgs/file/class/line: `start_trace`, `stop_trace`

## workflow

1. stabilize signal

- if async hides true cost, set `CUDA_LAUNCH_BLOCKING(=1)`.
- record `/proc/<pid>/cmdline`, `/proc/<pid>/environ`, `/proc/<pid>/cwd`.

2. collect stack evidence

- `py-spy top --nonblocking -p <pid>`
- `py-spy dump --nonblocking --native -p <pid>`
- `pstack <pid>` when deeper native detail is needed.

3. collect timeline evidence

- `viztracer --attach <pid> --output trace.json`
- `viztracer --combine trace*.json` for multi-process runs.

4. collect memory evidence when needed

- `memray attach --output mem.bin --native --follow-fork <pid>`
- `memray tree mem.bin`

5. output for next agent

- `runtime_facts`: measured numbers only.
- `suspected_hotspots`: ranked by evidence.
- `next_checks`: `MAX_NEXT_CHECKS(=3)`.

# info provider

## mission

- Build a high-signal evidence pack for debugging and decision making.

## source order

1. repo local: test, example, code, readme.
2. git history: known-good commit, regression range.
3. runtime inspector output: stack + timeline + memory.
4. web search: official docs, paper, stackoverflow, github issues.

## search strategy

- start from unique symbols: class name, function signature, option key.
- for web queries, use exact phrase and domain limits:
  - `"exact phrase"`
  - `site:<domain>`
  - `filetype:pdf`
- separate `observed_fact` and `inference`.

## output

- `evidence_table`: claim | source | confidence.
- `known_unknowns`: missing data that blocks conclusion.
- `best_next_probe`: one action with highest expected information gain.

# debugger

## mission

- Use a repeatable way to find root cause, then do minimal-risk fix.

## ideology

### search info

- gather evidence from info provider first.

### compare with workable version

- find workable version if possible, then compare first divergent output.

### simplify

- use simple code to reproduce and make repro faster.
- disable optimization components to check relation.
- 怀疑谁, 就 disable 谁.

## workflow

1. define target

- `EXPECTED_STATE(=should happen)`
- `ACTUAL_STATE(=current behavior)`

2. minimal repro

- minimize system: multi-node -> single-node, multi-thread -> single-thread.
- minimize input: small known input first.

3. differential experiment

- known-good vs current.
- enable vs disable suspected component.
- `MAX_CHANGE_PER_RUN(=1)` to avoid mixed variables.

4. mandatory-path checks

- breakpoints/log at must-pass functions.
- validate call path is expected.
- check os evidence: `strace`, `dmesg`, `journalctl`.

5. fix and verify

- choose minimal blast-radius patch.
- verify on repro and one real workload.

## output

- `root_cause`: one sentence + evidence links.
- `fix_plan`: minimal changes only.
- `verification_result`: command + key output.
- `residual_risk`: what to watch after merge.

# issue orchestrator

## mission

- Route the task to the right agent and merge results into one closure decision.

## routing

1. scope unclear -> start with `info provider`.
2. runtime symptom first -> start with `runtime inspector`.
3. stable repro exists -> start with `debugger`.

## handoff contract

- to runtime inspector: pid/command, expected behavior, suspicious phase.
- to info provider: keywords, modules, known-good hints.
- to debugger: minimal repro, evidence table, comparison baseline.
