# Agent-First Shared Memory Roadmap

> Focused follow-up to the broader research synthesis in
> `docs/research-improvement-roadmap.md`.
>
> **Date:** 2026-03-10
> **Status:** Draft
> **Primary motion:** Agent-first
> **Secondary differentiator:** Shared hosted debug memory

---

## Purpose & Goal

Define the next 30/60/90 days for `llmdebug` around a sharper product claim:

`llmdebug` is the local-first debug evidence and shared memory layer for coding
agents, with retrieval and online-learning loops optimized for patch proposals.

This document narrows the roadmap around four decisions:

1. **Primary user motion:** agent-first
2. **Memory direction:** shared hosted memory, not local-only memory
3. **Downstream task:** patch proposals remain the main evaluation target
4. **Retrieval/control policy:** evidence selection is driven by the online-learning
   approach, not fixed hand-authored routing as the long-term control plane

## Scientific Gap

There are strong tools for:

- pretty tracebacks and local variable display
- cloud observability with AI-assisted root-cause analysis
- generic agent memory
- agentic code repair on benchmarks

There is still a gap for a **typed, persistent, debugging-specific memory layer**
that connects:

- local failure capture
- structured RCA state
- shared retrieval across runs and projects
- patch proposal outcomes
- agent-facing MCP access

That gap is where `llmdebug` can differentiate.

## Hypothesis / Claim

If `llmdebug` evolves from a local snapshot tool into a **shared hosted debug
memory and retrieval layer**, then coding agents will produce better patch
proposals than they do with snapshot-only or traceback-only context, especially
on repeated failures, regressions, and similar historical bugs.

## Architectural Snapshot

### Center of Change

The center of this roadmap is the seam between:

- snapshot capture/schema
- failure identity and clustering
- RCA state/store
- MCP retrieval APIs
- patch proposal evals
- online-learning policy selection

### Proposed Components

1. **Local capture layer**
   - Existing source of truth for raw evidence
   - Responsible for redaction and schema normalization before export

2. **Hosted evidence index**
   - Stores redacted snapshot bundles, RCA events, patch proposals, eval outcomes,
     and cluster metadata

3. **Typed retrieval service**
   - Returns structured results for similar failures, prior successful patches,
     regression windows, and RCA history

4. **MCP gateway**
   - Makes the hosted memory useful to coding agents in Copilot, Cursor, Junie,
     Claude Code, and other MCP-capable clients

5. **Online-learning loop**
   - Learns which retrieval actions and evidence slices improve patch proposal
     outcomes

6. **Eval harness**
   - Measures whether hosted memory retrieval materially improves patch proposal
     quality, solve rate, token use, and time-to-fix

### Data Flow

1. Local/CI failure creates a snapshot bundle
2. Bundle is redacted and normalized
3. Failure identity and clustering features are computed
4. Artifact is ingested into hosted memory
5. Agent retrieves typed context through MCP
6. Agent produces a patch proposal
7. Test reruns and eval outcomes are written back into memory
8. Online-learning policy updates retrieval preferences

### Invariants

- Local capture remains the source of truth
- Hosted memory stores **typed debug artifacts**, not arbitrary chat history
- Redaction happens before upload
- Retrieval results must remain inspectable and attributable
- Patch proposal evaluation remains the main outcome signal

### Primary Risks

1. **Scope drift into generic agent memory**
   - Mitigation: keep storage and retrieval centered on failures, patches, RCA,
     and outcomes

2. **Privacy and trust concerns**
   - Mitigation: strict redaction defaults, tenancy boundaries, retention policy,
     and explicit upload controls

3. **Hosted complexity outrunning retrieval quality**
   - Mitigation: prove retrieval lift with evals before building a large UI/admin
     surface

4. **Competition with existing memory tools**
   - Mitigation: stay focused on debugging-specific typed evidence rather than
     general assistant memory

## Scope Clarifications

### What "Shared Hosted Memory" Means

A shared, multi-project debug memory service that indexes redacted `llmdebug`
artifacts and supports agent-ready retrieval over:

- similar failures
- prior successful patch proposals
- regression clusters
- suspect files and commits
- historical RCA transitions
- test rerun outcomes

### What It Does Not Mean

- generic note-taking or chat transcript memory
- a broad SaaS observability platform in the next 90 days
- replacing local snapshot capture
- replacing the online-learning approach with static heuristics

### Evidence Packs and Retrieval

Short-term evidence packs may exist as operational interfaces, but the strategic
control plane should be the online-learning approach. Retrieval decisions should
be treated as actions, with rewards derived from patch proposal quality and
validation outcomes.

## 30 / 60 / 90 Day Roadmap

## 30 Days

### Goal

Build the substrate for shared debug memory and make it usable from agents.

### Deliverables

1. **Hosted artifact schema**
   - Define typed hosted entities for snapshot bundle, RCA event, patch proposal,
     eval outcome, and failure cluster

2. **Redaction and tenancy contract**
   - Define upload boundary, namespace/project scoping, retention policy, and
     minimum hosted metadata

3. **Remote ingest path**
   - Support exporting local or CI-captured artifacts into a simple hosted index

4. **Agent retrieval primitives**
   - Add MCP retrieval for:
     - `related_failures`
     - `similar_patches`
     - `cluster_summary`
     - `rca_history`

5. **Learning telemetry**
   - Log retrieval actions, reward signals, patch proposal outcomes, and test
     rerun results in a policy-friendly format

### Exit Criteria

- An agent can ask whether a failure or similar patch has been seen before
- Hosted memory contains redacted, typed artifacts rather than ad hoc text blobs
- Retrieval action logs are available for online-learning experiments

### Suggested Metrics

- ingest success rate
- retrieval latency
- top-k related-failure usefulness
- patch proposal acceptance with vs. without retrieval

## 60 Days

### Goal

Turn hosted memory into a measurable advantage for patch proposal quality.

### Deliverables

1. **Cross-run clustering**
   - Cluster failures across commits, branches, and runs

2. **Patch-memory retrieval**
   - Rank prior patch proposals by failure similarity plus validation outcome

3. **Git-aware memory features**
   - Add suspect diff, regression window, and last-known-good context into the
     retrieval surface

4. **Hosted-memory evals**
   - Add experimental conditions such as:
     - `traceback_only`
     - `with_snapshot`
     - `with_hosted_memory`
     - `with_hosted_memory_plus_patch_history`

5. **Project/team controls**
   - Lightweight auth, namespace scoping, and opt-in sharing controls

### Exit Criteria

- Hosted memory retrieval shows measurable lift over snapshot-only baselines
- Similar historical failures and successful patches are retrievable in agent
  workflows
- Team/project isolation is explicit and testable

### Suggested Metrics

- patch proposal solve rate
- validation pass rate
- localization accuracy
- token usage
- time to validated patch

## 90 Days

### Goal

Close the loop between hosted memory, online learning, and agent workflows.

### Deliverables

1. **Policy-driven retrieval selection**
   - Let the online-learning system choose hosted retrieval actions and evidence
     combinations

2. **Memory write-back loop**
   - Persist successful and failed patch proposals plus rerun outcomes back into
     hosted memory

3. **Background-agent pilot**
   - Demonstrate one end-to-end MCP workflow:
     - diagnose
     - retrieve hosted memory
     - propose patch
     - validate

4. **Benchmark package**
   - Publish a compact result set showing the lift from hosted shared memory

### Exit Criteria

- The online-learning loop consumes hosted retrieval actions as first-class
  decisions
- Hosted memory improves at least one core patch-proposal metric over the
  snapshot-only baseline
- A small but credible end-to-end agent demo exists

### Suggested Metrics

- reward improvement over policy baseline
- patch proposal quality uplift
- validated fix rate
- retrieval utility by action type
- write-back coverage

## Work Packages

1. Define hosted artifact schemas and versioning
2. Design upload/redaction/tenancy contract
3. Implement hosted ingest path
4. Implement hosted retrieval API
5. Expose hosted retrieval via MCP
6. Add clustering and failure identity enrichment
7. Add patch-history retrieval and ranking
8. Add git-aware regression features
9. Extend eval conditions for hosted memory
10. Instrument online-learning signals and write-back
11. Pilot a background-agent retrieval workflow
12. Publish benchmark results and positioning docs

## What Not To Build Yet

- generic personal memory features
- full observability dashboards
- broad enterprise admin/billing
- autonomous repair beyond the existing patch-proposal-centric evaluation loop
- hand-built routing logic that conflicts with the online-learning direction

## Evaluation Plan

### Methodology

- Compare traceback-only, snapshot-only, and hosted-memory-assisted conditions
- Evaluate on patch proposal tasks with validation reruns
- Record both task outcome and retrieval/action decisions

### Variables

- retrieval source: local snapshot vs. hosted memory
- retrieval type: failure history vs. patch history vs. git-aware evidence
- policy choice: fixed baseline vs. online-learning policy

### Metrics

- patch proposal solve rate
- validated fix rate
- localization accuracy
- token cost
- wall-clock time
- retrieval usefulness
- reward signal quality

### Threats to Validity

- leakage from repeated benchmark instances
- poor redaction reducing hosted retrieval value
- hosted latency changing agent behavior
- improvements caused by prompt changes rather than memory quality

## Recommendation

The recommended next big bet is:

**Agent-first shared debug memory, with hosted typed retrieval and
online-learning-driven action selection.**

This is ambitious but achievable if the hosted layer stays narrow, typed, and
debugging-specific.

## Related Reading

- `docs/research-improvement-roadmap.md`
- `docs/adaptive_online_policy_design.md`
- `docs/adaptive_online_policy_proposals_2026-02-26.md`
- `docs/snapshot_retrieval_optimization_2026-02-25.md`
- `docs/evidence_first_protocol.md`
