llmdebug | Evidence-First Debug Snapshot Capture for LLM-Assisted Debugging
Description: llmdebug captures structured failure-time snapshots to support evidence-first, reproducible LLM-assisted debugging workflows.

Skip to main content

Research Software | Open Source | MIT

# Evidence-First Snapshot Capture for LLM-Assisted Debugging

`llmdebug` captures structured execution evidence at failure time, including exception context, stack frames, and local state, and makes this evidence available through CLI, notebook, and MCP interfaces.

Problem

LLM debugging without runtime evidence is often underdetermined.

Contribution

Reproducible snapshots that prioritize crash-site signal over verbose traces.

$ pip install llmdebug[cli]
Reproducible Minimal Example

## Project Profile

Primary artifact
  `.llmdebug/latest.json`
Core interfaces
  Pytest plugin, CLI, Jupyter, MCP server
Safety controls
  PII redaction and production rate limiting
License
  MIT

## Abstract

Context. LLM-assisted debugging pipelines often receive insufficient runtime evidence, which limits diagnosis quality. Method. `llmdebug` captures structured snapshots at exception boundaries with crash-frame prioritization and local-state summaries. Output. The captured evidence is exposed through machine-readable and human-readable interfaces (CLI, notebook, MCP) to support iterative analysis. Scope. The project provides evidence transport and inspection; it does not provide formal guarantees of root-cause correctness.

## Key Contributions

This section summarizes the core capabilities that make snapshot-based debugging reproducible and inspectable.

### Default failure capture

Pytest failures can emit snapshots without additional project instrumentation.

### Structured, inspectable artifacts

JSON snapshots preserve exception context, frames, locals, and environment metadata.

### Consistent multi-surface access

The same evidence can be queried from terminal, notebook, production hooks, and MCP.

### Differential analysis support

Snapshot diffing enables run-to-run comparisons for regression diagnosis.

### Hypothesis support

A pattern-based hypothesis engine ranks common failure mechanisms for faster triage.

### Governance-aware controls

Redaction policies and rate limiting support safer operation in production contexts.

## Method Overview

This section outlines the failure-triggered pipeline from exception boundary to evidence consumption.

1. Step 1

   ### Exception boundary

       def test_transform():
           result = transform(data)
           assert result.shape == (100, 5)

2. Step 2

   ### Snapshot serialization

       {
         "exception": "ValueError: shape mismatch",
         "closest_frame": {
           "file": "pipeline.py",
           "line": 47
         }
       }

3. Step 3

   ### Evidence consumption

       $ llmdebug show
       $ llmdebug hypothesize
       $ llmdebug diff

## Capability Evidence

This section lists currently available capabilities with links to canonical documentation.

Evidence table: features available in the current public release.
Capability | Status | Documentation
--- | --- | ---
Pytest failures produce snapshots by default | Available | README: Quick Start
CLI inspection (`show`, `list`, `frames`, `diff`, `hypothesize`) | Available | README: CLI
Detail levels (`crash`, `full`, `context`) for evidence size control | Available | README: Detail Levels
Production hooks with rate limiting and redaction controls | Available | README: Production Hooks
MCP server with evidence tools and RCA state tools | Available | README: MCP Server

## Reproducible Minimal Example

This section provides a minimal procedure for capturing and inspecting a failure artifact.

terminal

    $ pip install llmdebug[cli]
    $ pytest
    $ llmdebug show

### Expected Artifact

After a test failure, the latest snapshot is written to `.llmdebug/latest.json`. Use `llmdebug show --detail crash|full|context` to control output scope.

### Reproducibility Checklist

For benchmarks or reports, record package version, model settings, random seed, and command-line invocation alongside each emitted snapshot.

## Interfaces (Shared Evidence Contract)

This section shows equivalent access patterns for the same snapshot evidence across integration surfaces.

shell

    # zero additional setup after installation
    $ pip install llmdebug
    $ pytest

    # failure artifact:
    #   .llmdebug/latest.json

    from llmdebug import debug_snapshot

    @debug_snapshot()
    def run_job(payload: list[int]) -> list[int]:
        return [transform(x) for x in payload]

    from llmdebug import snapshot_section

    with snapshot_section("feature_pipeline"):
        features = build_features(raw_data)

    %load_ext llmdebug
    %llmdebug
    %llmdebug hypothesize
    %llmdebug diff

    import llmdebug
    llmdebug.install_hooks()

    # captures:
    # sys.excepthook
    # threading.excepthook
    # sys.unraisablehook

    {
      "mcpServers": {
        "llmdebug": {
          "command": "uvx",
          "args": ["llmdebug[mcp]", "serve"]
        }
      }
    }

## Data Governance and Operational Safety

This section summarizes data handling safeguards and operational constraints for practical deployments.

### Redaction-aware capture

Built-in redaction controls help reduce accidental leakage of sensitive fields in stored snapshots.

### Production rate limiting

Exception hooks apply rate limits to avoid artifact floods during repeated failures.

### Local-first artifact handling

Snapshots are stored locally by default, enabling offline and air-gapped debugging workflows.

### No automatic causal guarantees

The system provides structured context for diagnosis but does not establish causal correctness.

## Limitations

This section states interpretation limits and known boundaries of the current implementation.

- Snapshots reflect observed failing executions; unexercised paths remain unobserved.
- Large objects may be summarized for compactness, which can omit low-level detail.
- Hypothesis ranking is heuristic and should be treated as triage support, not proof.
- Benchmark and statistical evaluation workflows live in `evals/` and are separate from this page.

## Citation and Resources

This section provides a software citation template and release-traceable project references.

BibTeX (software citation template)

    @software{vadasz2026llmdebug,
      author  = {Vadasz, Nicolas},
      title   = {llmdebug: Structured Debug Snapshots for LLM-Assisted Debugging},
      year    = {2026},
      url     = {https://github.com/nicholasvadasz/llmdebug},
      license = {MIT}
    }

### Project Links

Update citation year and metadata if you reference a specific release.

- GitHub repository
- PyPI package
- README documentation
- Research roadmap