# Litmus — Deterministic Agent Simulation Engine
## Complete Project Blueprint

**One-liner:** Open-source runtime for deterministic replay, fault injection, and reliability scoring of multi-step AI agent executions.

**Conviction:** 8.8/10 — backed by 121 podcast evidence rows, 50+ sources, 10 research batches, 8 methodology iterations, 120+ candidates killed.

---

# 1. THE PROBLEM

## 1.1 Why agents are broken

Traditional software is deterministic: same input produces same output produces same code path. You can write a unit test, run it 1000 times, and get the same result. The entire $500B+ software testing industry is built on this assumption.

AI agents violate this assumption. An agent is a loop: observe, think (LLM call), act (tool call), repeat. The "think" step is non-deterministic — the same prompt can produce different outputs every time. This means:

- **You cannot reproduce bugs.** A user reports "the agent booked the wrong flight." By the time you investigate, the agent would make different choices.
- **You cannot regression test.** You fix a bug, but how do you verify the fix works? Running the agent again produces a different execution path.
- **You cannot quantify reliability.** What is the failure rate of your agent? You literally cannot measure it without running it thousands of times in production.
- **You cannot do CI/CD for agents.** There is no equivalent of "run the test suite before deploy" because there IS no test suite that works for non-deterministic systems.

## 1.2 What exists today and why it is insufficient

### Observability tools (LangSmith, Langfuse, Braintrust, Arize)
- They LOG what happened — traces of LLM calls, tool calls, latencies, costs
- They CANNOT replay an execution. You can see the dashcam footage but you cannot re-run the scenario
- Analogy: A security camera shows you the robbery. It does not let you simulate "what if the alarm went off 30 seconds earlier?"

### Static benchmarks (SWE-bench, HumanEval, MMLU)
- Fixed datasets with known-correct answers
- Do not test real production workflows. Your agent does not solve GitHub issues — it books flights, processes expenses, handles customer tickets
- SWE-bench creators say unit-test verification is limiting and long-running agent tournaments are the future
- OpenAI Frontier Evals team says SWE-bench is saturated and new eval paradigms are needed

### Synthetic simulation (Maxim AI, Coval)
- Generate fake scenarios (synthetic personas, synthetic conversations) and test agents against them
- Problem: synthetic scenarios do not match your actual production traffic
- Sarah Guo (Conviction VC): "RL environments are a fad. Real-world logs beat synthetic clones every time."

### Antithesis ($182M from Jane Street)
- Deterministic simulation at the CPU/hypervisor level. Brilliant technology.
- Cannot intercept LLM API calls. Cannot understand agent semantics. Cannot inject LLM-specific faults.
- Built for traditional distributed systems, not for AI agents.

### Docker Cagent (Jan 2026)
- API-level record-and-replay for agent testing. Very early stage.
- No fault injection. No scoring. No environment synthesis. No RL bridge.
- Validates the market thesis but only addresses Layer 2 at the most basic level.

## 1.3 The gap

Nobody does the full stack: Record real production agent traces -> Replay them deterministically -> Inject faults to test resilience -> Score reliability quantitatively -> Synthesize new scenarios from real data -> Bridge to RL training.

That is Litmus. Each of those arrows is a layer.

---

# 2. THE SOLUTION — 6-LAYER ARCHITECTURE

## Architecture overview

```
Layer 6: RL Training Bridge          [$20-50K/mo]  [Year 2+]
Layer 5: Environment Synthesis       [$15K/mo]     [Year 2+]
Layer 4: Reliability Scoring         [$15K/mo]     [Month 6-9]
Layer 3: Fault Injection             [Included]    [Month 3]
Layer 2: Deterministic Replay        [$5K/mo]      [Month 2-3]
Layer 1: Trace Recorder              [Free/OSS]    [Month 1]
```

Each layer builds on the one below it. You cannot do fault injection without replay. You cannot do scoring without fault injection. This creates a natural upgrade path and lock-in.

## Value ladder

| Layer | What it does | Who pays | Why they pay | ACV |
|-------|-------------|----------|-------------|-----|
| 1 | Records every LLM/tool call | Nobody (free) | Distribution + data collection | $0 |
| 2 | Replays executions deterministically | AI platform eng | Debug production failures | $60K |
| 3 | Injects faults into replays | Same buyer + security | Test resilience before deploy | Included |
| 4 | Scores reliability across scenarios | VP Eng / CTO | Board deck, compliance, SOC 2 | $180K |
| 5 | Generates new scenarios from data | AI governance | Pre-deployment certification | $180K |
| 6 | Bridges to RL training | RL training teams | Continuous agent improvement | $240-600K |

---

# 3. LAYER 1: TRACE RECORDER (OSS)

## What it is
An open-source SDK/library that instruments agent frameworks to capture a complete, structured record of every external interaction during an agent execution.

## What it captures

For every LLM call:
- Timestamp (monotonic clock)
- Model identifier (e.g., claude-sonnet-4-20250514)
- Full request payload (system prompt, messages, tools schema, temperature, seed, max_tokens)
- Full response payload (content blocks, tool_use blocks, stop_reason, usage)
- Streaming chunks (if streaming was used)
- Latency (time to first token, total time)
- Request/response headers (for debugging provider issues)

For every tool call:
- Tool name (e.g., "search_database", "send_email", "read_file")
- Input arguments (full JSON payload)
- Output response (full JSON payload)
- HTTP status code (if API-based)
- Latency
- Whether the tool call was retried (and how many times)

For every agent decision:
- Which tools were available in the context
- Which tool the agent chose (and the reasoning if chain-of-thought is visible)
- The ordering of concurrent operations (happens-before relationships)
- State mutations (what changed in the agent's memory/context between steps)

## Trace format

```json
{
  "trace_id": "lt-abc123",
  "agent_id": "booking-agent-v2",
  "started_at": "2026-04-15T10:23:45.123Z",
  "completed_at": "2026-04-15T10:23:52.891Z",
  "status": "completed",
  "steps": [
    {
      "step_id": "s001",
      "type": "llm_call",
      "timestamp": "2026-04-15T10:23:45.200Z",
      "request": {
        "model": "claude-sonnet-4-20250514",
        "messages": [...],
        "tools": [...],
        "temperature": 0.7,
        "seed": null
      },
      "response": {
        "content": [...],
        "stop_reason": "tool_use",
        "usage": {"input_tokens": 1523, "output_tokens": 89}
      },
      "latency_ms": 1243
    },
    {
      "step_id": "s002",
      "type": "tool_call",
      "timestamp": "2026-04-15T10:23:46.500Z",
      "tool_name": "search_flights",
      "input": {"origin": "SFO", "destination": "JFK", "date": "2026-05-01"},
      "output": {"flights": [...]},
      "latency_ms": 890
    }
  ],
  "metadata": {
    "framework": "langchain",
    "framework_version": "0.3.1",
    "litmus_sdk_version": "0.1.0",
    "environment": "production"
  }
}
```

## Supported frameworks (priority order)

1. **Raw API calls** — OpenAI SDK, Anthropic SDK, direct HTTP. Instrument via monkey-patching or middleware.
2. **LangChain / LangGraph** — Hook into callback system. LangChain has the largest installed base.
3. **CrewAI** — Growing fast in multi-agent use cases.
4. **OpenAI Agents SDK** — New but backed by OpenAI, will grow.
5. **AutoGen** — Microsoft-backed, popular in enterprise.
6. **Custom frameworks** — Provide a decorator/context-manager API for any Python code.

## Installation

```bash
pip install litmus-trace
```

```python
from litmus import trace

@trace.record()
def my_agent(user_input: str):
    # Your agent code here — LangChain, raw API, anything
    response = client.messages.create(model="claude-sonnet-4-20250514", ...)
    tool_result = search_flights(response.tool_input)
    final = client.messages.create(...)
    return final

# Trace is automatically saved to ./traces/lt-{uuid}.json
```

## Why it is free

This is the distribution layer. Classic open-core model:
- Give away the data collector → maximize installs
- Each install generates traces → traces are only useful with paid replay engine
- OSS builds trust and community → community becomes pipeline for paid product
- GitHub stars and HN traction → social proof for enterprise sales

## Success metric for Layer 1
- 500+ GitHub stars in first 2 months
- 50+ weekly active installs (pip downloads)
- 10+ community contributors
- 5+ blog posts / tweets from users

---

# 4. LAYER 2: DETERMINISTIC REPLAY ($5K/mo)

## What it is
Given a trace file from Layer 1, Layer 2 replays the exact same agent execution — same LLM responses, same tool outputs, same timing — producing a bit-exact reproduction of what happened.

## How it works technically

### The core insight
The agent's code (your LangChain pipeline, your custom Python) is deterministic — it always follows the same logic. What is non-deterministic is:
1. LLM responses (different every call, even with same prompt)
2. Tool responses (external APIs return different data)
3. Timing (network latency varies)

If you record all three and play them back, the agent code runs identically.

### The proxy architecture

```
[Agent Code] ←→ [Litmus Proxy] ←→ [Real LLM APIs / Tool APIs]
                      |
                      ↓
               [Trace File]
```

**Record mode:** The proxy sits between the agent and the real world. Every LLM call and tool call passes through. The proxy forwards to the real API, records both request and response, and passes the response back to the agent.

**Replay mode:** The proxy intercepts every LLM call and tool call, but instead of forwarding to the real API, it looks up the corresponding recorded response from the trace file and returns that. The real APIs are never called.

### Technical challenges and solutions

**Challenge 1: Request matching**
In replay mode, the agent makes a call. How does the proxy know which recorded response to serve?
- **Simple approach:** Sequential matching — serve responses in the same order they were recorded. Works for single-threaded agents.
- **Better approach:** Content-based matching — hash the request payload and match it to the recorded request with the same hash. Works for concurrent agents.
- **Best approach:** Causal matching — use the step_id and happens-before ordering to match requests to their correct recorded responses, even when concurrency patterns differ slightly.

**Challenge 2: Streaming responses**
Many LLM APIs stream tokens one at a time. The trace must record the full stream (including timing between chunks) and replay it at the same cadence — otherwise the agent's streaming handler might behave differently.

**Challenge 3: Multi-model pipelines**
A real agent might call Claude for planning, GPT for code generation, and a local model for classification — all in the same execution. The proxy must handle all three providers simultaneously and route each to the correct recorded response.

**Challenge 4: Stateful tools**
If the agent reads from a database, writes to it, then reads again — the second read depends on the write. The proxy must understand this dependency chain and replay the state mutations in order.
- Solution: Record the full tool response including any state-bearing information. On replay, serve the recorded response regardless of actual database state.
- Limitation: If the agent code ALSO mutates local state between tool calls, and that local state affects the next request, the replay may diverge. This is detected and flagged.

### Replay fidelity verification
After every replay, Litmus compares the replayed execution to the original trace:
- Did the agent make the same sequence of calls?
- Did it make the same number of calls?
- Did it produce the same final output?
- If any divergence is detected, it is flagged with the exact step where divergence occurred.

### API

```python
from litmus import replay

# Load a production trace
trace = replay.load("./traces/lt-abc123.json")

# Replay it deterministically
result = replay.run(trace)

# Compare to original
diff = replay.compare(trace, result)
print(diff.diverged)       # False if bit-exact
print(diff.divergence_step) # None, or the step where it diverged
print(diff.final_output_match) # True/False
```

### CLI

```bash
# Replay a trace
litmus replay ./traces/lt-abc123.json

# Replay and diff against original
litmus replay --diff ./traces/lt-abc123.json

# Replay with verbose output (shows each step)
litmus replay --verbose ./traces/lt-abc123.json
```

## Who pays for this
- **Primary buyer:** AI platform engineers at Series B-D startups who ship agents to production
- **Title:** Staff Engineer, Principal Engineer, AI Platform Lead
- **Company size:** 50-500 employees, $10M-$100M ARR
- **Budget line:** Engineering tools / DevOps infrastructure
- **Pain they feel:** "Our agent broke in production last night and we have no way to reproduce the bug"

---

# 5. LAYER 3: FAULT INJECTION (included with Layer 2)

## What it is
Once you can replay deterministically, you can start asking "what if?" questions. Fault injection lets you modify a recorded trace and replay a variant where something went wrong.

## Types of faults

### LLM response faults
| Fault type | What it does | Why it matters |
|-----------|-------------|---------------|
| **Hallucination** | Replace a factual response with a plausible-but-wrong one | Does your agent detect and recover from hallucinated data? |
| **Refusal** | Replace response with "I cannot help with that" | Does your agent handle model refusals gracefully? |
| **Format change** | Return JSON when agent expects text, or vice versa | Does your agent parse responses defensively? |
| **Truncation** | Cut the response off mid-sentence | Does your agent handle incomplete responses? |
| **Slow response** | Add 30 seconds of latency | Does your agent timeout correctly? Does it retry? |
| **Model version shift** | Return a response characteristic of a different model version | Is your agent brittle to model upgrades? |
| **Token limit exceeded** | Return a max_tokens-truncated response | Does your agent handle context window overflow? |

### Tool response faults
| Fault type | What it does | Why it matters |
|-----------|-------------|---------------|
| **HTTP 500** | Return a server error | Does your agent retry? Fall back? Crash? |
| **HTTP 429 (rate limit)** | Return rate limit after N successful calls | Does your agent backoff correctly? |
| **Timeout** | Tool never responds | Does your agent have a timeout? What happens when it fires? |
| **Auth failure (401/403)** | Return authentication error | Does your agent re-auth? Alert a human? |
| **Stale data** | Return data from 24 hours ago instead of current | Does your agent validate freshness? |
| **Schema change** | Return response with different field names | Does your agent break on API version changes? |
| **Partial response** | Return only some of the expected data | Does your agent handle missing fields? |
| **Empty response** | Return empty/null | Does your agent check for empty? |

### Compound faults
| Fault type | What it does | Why it matters |
|-----------|-------------|---------------|
| **Cascading failure** | Tool A fails, causing tool B to get bad input | Multi-step error propagation |
| **Intermittent** | Tool fails 20% of the time randomly | Is the agent robust to flaky dependencies? |
| **Correlated** | Two tools fail simultaneously | How does the agent handle multiple failures at once? |
| **Progressive degradation** | Each subsequent call gets slightly worse | Does the agent detect quality decay? |

## API

```python
from litmus import replay, faults

trace = replay.load("./traces/lt-abc123.json")

# Inject a single fault
result = replay.run(trace, faults=[
    faults.llm_hallucinate(step="s003", severity="high"),
])

# Inject multiple faults
result = replay.run(trace, faults=[
    faults.tool_timeout(step="s005", timeout_ms=30000),
    faults.llm_refuse(step="s007"),
])

# Inject random faults (chaos mode)
result = replay.run(trace, faults=[
    faults.random_tool_failure(probability=0.1),
    faults.random_llm_hallucination(probability=0.05),
])

# Check what happened
print(result.status)          # "completed" | "crashed" | "hung" | "wrong_output"
print(result.recovery_actions) # What the agent did when it hit the fault
print(result.total_retries)    # How many retries the agent attempted
```

## Inspiration: MiniMax perturbation pipelines
MiniMax (Olive Song, CogRev Feb 2026) described their RL training approach: "Agent generalization is adaptation to perturbations across the entire operational space — tool information, system prompts, user prompts, chat templates, the environment, and tool responses."

They perturb at TRAIN time for robustness. We perturb at TEST time for reliability measurement. Same concept, complementary use.

---

# 6. LAYER 4: RELIABILITY SCORING ($15K/mo)

## What it is
Layer 3 generates hundreds of "what if" scenarios. Layer 4 automatically scores the agent's behavior across all of them, producing a single reliability number.

## The reliability score

```
Litmus Reliability Score: 87/100
├── Correctness: 92/100     (produces right output in normal conditions)
├── Resilience: 78/100      (recovers from faults gracefully)
├── Safety: 95/100          (never takes irreversible actions without confirmation)
├── Efficiency: 84/100      (reasonable token/API usage, no infinite loops)
└── Consistency: 86/100     (output variance across 100 replays is low)
```

## How each dimension is measured

### Correctness (does it produce the right answer?)
- Replay the original trace 100 times with NO faults
- Measure: does the final output match the original? How much does it vary?
- Requires the user to define "correct" — either via expected output or via a judge function

### Resilience (does it handle failures?)
- Run the full fault injection matrix from Layer 3
- For each fault: did the agent recover, degrade gracefully, or crash?
- Score = (recovered + degraded_gracefully) / total_faults

### Safety (does it avoid dangerous actions?)
- Scan every tool call for irreversible actions (delete, send, purchase, transfer)
- Verify: was human confirmation obtained before each irreversible action?
- Under fault injection: does the agent EVER take an irreversible action when confused?

### Efficiency (is it reasonable in resource usage?)
- Count total LLM calls, total tokens, total tool calls
- Compare to baseline (the original trace's usage)
- Flag: infinite loops, excessive retries, redundant calls

### Consistency (is the output stable?)
- Replay 100 times with small perturbations (slightly different phrasing, minor tool variations)
- Measure output variance using semantic similarity
- High consistency = agent is robust. Low consistency = agent is brittle.

## The compliance angle

This score is the number that goes into:
- **SOC 2 audits:** "We test our AI agents with Litmus and maintain a reliability score above 90%"
- **Board decks:** "Agent reliability improved from 72% to 91% this quarter"
- **Customer SLAs:** "Our agent is certified to Litmus Level 3 (85+ reliability)"
- **Insurance:** As AI liability frameworks emerge, quantified reliability reduces premiums
- **Regulatory filings:** EU AI Act requires risk assessment for high-risk AI systems

## The data that METR found is missing
Joel Becker (METR, Latent Space Feb 2026): "In 2026, agents are more autonomous, but developers were NOT measurably more productive (despite feeling so)."

The measurement gap exists because nobody has deterministic, quantitative, reproducible measurement of agent behavior. Layer 4 fills this gap.

---

# 7. LAYER 5: ENVIRONMENT SYNTHESIS ($15K/mo)

## What it is
Layers 1-4 work with recorded real traces. Layer 5 generates entirely NEW scenarios that have never happened — synthetic environments grounded in your real production data.

## How it works

### Step 1: Build a statistical model of your trace corpus
Analyze the 10,000+ traces you have collected via Layer 1:
- What distribution of tool responses do you see? (e.g., flight search returns 2-15 results, average 7)
- What patterns of LLM behavior occur? (e.g., model asks for clarification 30% of the time)
- What failure modes have been observed? (e.g., payment API fails 2% of the time with 429)
- What user input patterns exist? (e.g., 60% domestic flights, 30% international, 10% multi-city)

### Step 2: Generate novel but realistic scenarios
Using the statistical model, generate traces that are:
- **Within distribution:** Scenarios that COULD plausibly happen but HAVE NOT yet
- **Edge cases:** Scenarios at the tails of your distributions (a flight search returning 0 results, a 15-leg multi-city trip)
- **Combination attacks:** Faults that occur together (rate limit + stale data + user in non-English language)
- **Temporal patterns:** Scenarios that test time-dependent behavior (end-of-month, holiday pricing, timezone edge cases)

### Step 3: The data moat
Every customer's traces improve the synthesis engine:
- More traces → better statistical model → more realistic scenarios
- A customer with 100K traces gets dramatically better synthetic scenarios than one with 1K
- Cross-customer patterns (with privacy-preserving aggregation) improve the engine for everyone
- This is the compounding data advantage that makes Litmus harder to compete with over time

## Why not start with synthetic?
Maxim AI starts with synthetic personas and conversations. The problem: synthetic scenarios are only as good as your imagination. You test what you THOUGHT could go wrong.

Litmus starts with real data and THEN generates variations. You test what ACTUALLY happened plus plausible extensions. This is the "real logs beat synthetic clones" principle.

---

# 8. LAYER 6: RL TRAINING BRIDGE ($20-50K/mo)

## What it is
The infrastructure from Layers 1-5 — deterministic replay, fault injection, scoring, synthetic environments — becomes a training environment for reinforcement learning.

## The insight
To train an agent with RL, you need three things:
1. **An environment** the agent can interact with → Layers 2 + 3 (replay + fault injection)
2. **A reward signal** that says how well it did → Layer 4 (reliability scoring)
3. **Diverse episodes** to train on → Layer 5 (environment synthesis)

We already built the RL training loop without meaning to.

## How it works

```python
from litmus import rl

# Define the training environment from your real traces
env = rl.Environment(
    trace_corpus="./traces/",
    fault_profile=rl.FaultProfile.REALISTIC,
    synthesis_budget=10000,  # generate 10K synthetic scenarios
)

# Define the reward function using Layer 4 scoring
reward = rl.RewardFunction(
    correctness_weight=0.4,
    resilience_weight=0.3,
    safety_weight=0.2,
    efficiency_weight=0.1,
)

# Train
agent = rl.train(
    base_agent=my_current_agent,
    environment=env,
    reward=reward,
    episodes=50000,
    method="ppo",
)
```

## Who pays $20-50K/mo for this
- Teams building autonomous agents for high-stakes workflows
- Financial services (trading agents, fraud detection)
- Healthcare (clinical decision support)
- Autonomous operations (infrastructure management, incident response)
- They need agents that continuously improve from simulated experience

## Why this is the endgame moat
If Litmus becomes the environment where agents are TRAINED (not just tested), switching away means:
- Retraining from scratch (months of compute)
- Losing the entire simulated world your agent learned in
- Losing the reward function calibration
- Losing the statistical model of your production environment

This is deeper than testing lock-in — this is training lock-in.

---

# 9. GO-TO-MARKET STRATEGY

## Phase 1: OSS traction (Months 1-3)

### Target
AI engineers who build agents and feel the pain of untestable systems.

### Actions
1. **Ship Layer 1 on GitHub** — Python package, MIT license, clean README, demo video
2. **Launch on Hacker News** — "Show HN: Flight recorder for AI agents — record and deterministically replay any agent execution"
3. **Write 3 blog posts:**
   - "Why you cannot unit test AI agents (and what to do instead)"
   - "We replayed 10,000 production agent traces — here is what we found"
   - "Chaos engineering for AI agents: what Netflix taught us about reliability"
4. **Get listed on:** awesome-langchain, awesome-llm-tools, LangChain integrations page
5. **Engage early adopters:** Discord server, GitHub issues, weekly office hours

### Success criteria
- 500+ GitHub stars
- 50+ weekly active pip installs
- 5+ design partner conversations started
- 1+ blog post with 50K+ views

## Phase 2: First revenue (Months 3-6)

### Target
AI platform teams at Series B-D startups who are already in production with agents and have experienced agent failures they could not debug.

### Actions
1. **Launch Litmus Cloud** — hosted version of Layer 2 + 3 (replay + fault injection)
2. **Convert 5-8 design partners** from Phase 1 OSS users
3. **Run case studies** with early customers showing before/after debugging times
4. **Speak at AI Engineer Conference** (if timing works)
5. **Content: "State of Agent Reliability 2026"** report based on anonymized data from OSS traces

### Pricing
- $5K/mo per team (up to 50K traces/month)
- Free tier: 1K traces/month with replay (rate-limited)
- Design partner discount: 50% off first 6 months

### Success criteria
- 5-8 paying customers
- $300-480K ARR run rate
- NPS > 50 from design partners
- <10% monthly churn

## Phase 3: Enterprise (Months 6-12)

### Target
VP Engineering, CTO, AI Governance leads at Series C+ and public companies who need quantifiable agent reliability for compliance, board reporting, and customer SLAs.

### Actions
1. **Launch Layer 4** (reliability scoring) as enterprise add-on
2. **SOC 2 Type I certification** for Litmus Cloud
3. **Enterprise sales motion:** Outbound to companies with public AI agent announcements
4. **Compliance narrative:** "Litmus Reliability Score" as industry standard for agent quality
5. **Partnerships:** Integrate with existing CI/CD (GitHub Actions, GitLab CI, Jenkins)

### Pricing
- Layer 2+3: $5K/mo
- Layer 4: $15K/mo (includes 2+3)
- Enterprise: Custom pricing, annual contracts, dedicated support

### Success criteria
- 10-15 total customers
- $1M+ ARR
- 2+ enterprise logos (Fortune 500 or equivalent)
- Average ACV of $85K

## Phase 4: Platform moat (Month 12+)

### Target
RL training teams, AI safety teams, regulatory compliance teams.

### Actions
1. **Launch Layer 5** (environment synthesis)
2. **Launch Layer 6** (RL training bridge) in beta
3. **Publish research paper** on deterministic agent simulation methodology
4. **Industry standard push:** Propose "Litmus Score" as standard metric (like Lighthouse scores for web performance)
5. **API ecosystem:** Let third parties build on top of Litmus traces and scores

### Success criteria
- Data moat visibly compounding (customers with 100K+ traces see measurably better results)
- 2+ customers using Layer 6 for RL training
- Industry references to "Litmus Score" in blog posts, conference talks

---

# 10. COMPETITIVE LANDSCAPE (DETAILED)

## Direct competitors

### LangSmith (LangChain) — $25M funding
- **What they do:** Observability, tracing, prompt playground, dataset management, evaluation
- **Strength:** Largest installed base via LangChain ecosystem. Harrison Chase is actively branding around "context engineering"
- **Gap:** Cannot replay traces. Cannot inject faults. No reliability scoring. They LOG, we SIMULATE.
- **Threat level:** MEDIUM. Could add replay as a feature, but it would require fundamentally different infrastructure. Their architecture is built for logging, not for execution.
- **Our positioning vs them:** "LangSmith tells you what happened. Litmus tells you what WOULD happen."

### Langfuse — Acquired by ClickHouse (Jan 2026)
- **What they do:** Open-source LLM observability. Traces, evals, prompt management, cost tracking.
- **Strength:** OSS, popular with teams who do not want vendor lock-in
- **Gap:** Same as LangSmith — logs, does not replay. Acquisition by ClickHouse suggests focus on data/analytics, not simulation.
- **Threat level:** LOW-MEDIUM. Post-acquisition direction is unclear.

### Braintrust
- **What they do:** Eval scoring, dataset management, logging, human feedback collection
- **Strength:** Good eval framework. Strong with teams that do prompt engineering.
- **Gap:** No replay. No fault injection. Evals are synthetic/static, not from production traces.
- **Threat level:** LOW-MEDIUM. Different product philosophy (eval-first vs simulation-first).

### Antithesis — $182M from Jane Street
- **What they do:** Deterministic simulation at the hypervisor level. Record and replay entire VMs with bit-exact CPU-level determinism.
- **Strength:** Deepest technology. Proven on traditional distributed systems.
- **Gap:** CPU-level approach cannot intercept LLM API calls, cannot understand agent semantics, cannot inject LLM-specific faults. Would need a complete new product line for AI agents.
- **Threat level:** HIGH long-term, LOW short-term. They COULD build this but it would be a new product, not an extension. Their focus is on database testing, not AI agents.
- **Our positioning vs them:** "Antithesis simulates the hardware. Litmus simulates the agent."

### Docker Cagent — Docker internal project (Jan 2026)
- **What they do:** API-level record-and-replay for agent testing. Proxy-and-cassette model.
- **Strength:** Docker brand. Distribution via Docker ecosystem.
- **Gap:** Very early. No fault injection. No scoring. No environment synthesis. No RL bridge. API-level only (misses framework-level semantics).
- **Threat level:** MEDIUM. Validates market thesis. Could grow fast if Docker invests heavily.
- **Our positioning vs them:** "Cagent records API calls. Litmus understands agent behavior."

### Maxim AI
- **What they do:** Synthetic simulation and testing platform. AI-generated test personas, synthetic conversations.
- **Strength:** "Simulation-first" branding. Good for pre-deployment testing.
- **Gap:** Synthetic only — does not use real production traces. Sarah Guo's critique applies: synthetic environments do not match real-world patterns.
- **Threat level:** MEDIUM-HIGH. Closest to our vision, but opposite approach (synthetic-first vs real-trace-first).
- **Our positioning vs them:** "Maxim imagines what could go wrong. Litmus knows what actually went wrong and tests variations of reality."

## Adjacent players (not direct competitors, but in the space)

| Company | What they do | Funding | Overlap |
|---------|-------------|---------|---------|
| Patronus AI | AI safety red-teaming | Undisclosed | Safety scoring only (our L4 subset) |
| Galileo AI | Hallucination detection, quality metrics | Undisclosed | Quality metrics (our L4 subset) |
| Arize AI | ML/LLM observability, traces, evals | $62M | Observability (our L1 subset) |
| AgentOps | Agent observability and debugging | Undisclosed | Agent-specific observability |
| Weights & Biases | ML experiment tracking | Acquired by CoreWeave | Experiment tracking, not simulation |

## Companies that will NOT compete

| Company | Why not |
|---------|--------|
| OpenAI / Anthropic / Google | They are model providers, not testing infrastructure. They want agents to work, not to test them. Would be like AWS building your CI/CD for you. |
| GitHub / GitLab | Their CI/CD is for code, not for agents. Adding agent testing would be a new product line. Possible but unlikely near-term. |
| Datadog / New Relic | Infrastructure observability. They could extend to agent observability but not to simulation. Different core competency. |

---

# 11. TECHNICAL ARCHITECTURE

## System components

```
┌─────────────────────────────────────────────────────────────┐
│                     LITMUS CLOUD                            │
│                                                             │
│  ┌──────────┐  ┌──────────────┐  ┌─────────────────────┐   │
│  │ Trace    │  │ Replay       │  │ Scoring Engine       │   │
│  │ Ingestion│→ │ Engine       │→ │ (Correctness,        │   │
│  │ Service  │  │ (Proxy +     │  │  Resilience, Safety, │   │
│  │          │  │  Cassettes)  │  │  Efficiency,         │   │
│  │          │  │              │  │  Consistency)        │   │
│  └──────────┘  └──────────────┘  └─────────────────────┘   │
│       ↑              ↑                    ↓                 │
│  ┌──────────┐  ┌──────────────┐  ┌─────────────────────┐   │
│  │ Trace    │  │ Fault        │  │ Dashboard &          │   │
│  │ Storage  │  │ Injection    │  │ API                  │   │
│  │ (S3/R2)  │  │ Library      │  │ (Scores, Reports,    │   │
│  │          │  │              │  │  CI/CD Integration)  │   │
│  └──────────┘  └──────────────┘  └─────────────────────┘   │
│                                                             │
└─────────────────────────────────────────────────────────────┘
       ↑
┌──────────────┐
│ litmus-trace │  ← OSS SDK installed in customer's agent
│ (pip package)│
└──────────────┘
```

## Tech stack (recommended)

| Component | Technology | Why |
|-----------|-----------|-----|
| OSS SDK | Python | 90%+ of agent frameworks are Python |
| Proxy/Replay Engine | Rust or Go | Performance-critical: must replay at 10x real-time speed |
| Trace Storage | S3-compatible (R2, MinIO) | Cheap, scalable, customer can self-host |
| Scoring Engine | Python + NumPy | Complex scoring logic, rapid iteration |
| Dashboard | React + TypeScript | Standard SaaS frontend |
| API | FastAPI (Python) or Axum (Rust) | Depends on team preference |
| Database | PostgreSQL | Metadata, customer accounts, scores |
| Queue | Redis / NATS | Async replay job scheduling |
| CI/CD Integration | GitHub Actions, GitLab CI plugins | First-class integration with existing pipelines |

## Data flow

1. **Agent runs in production** → litmus-trace SDK captures every external call
2. **Trace is uploaded** to Litmus Cloud (or saved locally for self-hosted)
3. **User triggers replay** → Replay Engine loads trace, starts proxy, runs agent code in sandbox
4. **User triggers fault injection** → Fault Library mutates trace, Replay Engine replays variant
5. **Scoring Engine** runs N replays (normal + faulted), computes reliability score
6. **Dashboard** shows score breakdown, trend over time, comparison between agent versions
7. **CI/CD plugin** runs replay suite before deploy, blocks deploy if score drops below threshold

## Self-hosted option

Critical for enterprise customers who cannot send traces to a cloud service:
- Litmus Cloud is packaged as a Docker Compose / Helm chart
- All components run in customer's VPC
- Traces never leave their network
- Same features, customer manages infrastructure
- Pricing: same monthly fee + optional support contract

## Security considerations

- Traces contain the FULL payload of every LLM call and tool call — this includes potentially sensitive customer data, PII, and secrets
- Encryption at rest (AES-256) and in transit (TLS 1.3)
- Customer-managed encryption keys (BYOK) for enterprise
- SOC 2 Type I certification by Month 9, Type II by Month 18
- Data isolation: traces from different customers are never co-mingled
- Retention policies: customer controls how long traces are stored
- PII detection and auto-redaction option for traces before storage

---

# 12. 90-DAY BUILD PLAN

## Week 1-2: Foundation

### Deliverables
- [ ] Git repository set up (litmus-trace for OSS, litmus-cloud for paid)
- [ ] Python package structure for litmus-trace
- [ ] Basic trace format spec (JSON schema)
- [ ] Instrument ONE framework: raw Anthropic SDK calls
  - Monkey-patch `client.messages.create()` to capture request + response
  - Write to local JSON file
- [ ] Record 10 real agent traces from a demo agent you build
- [ ] README with installation instructions and demo

### Technical spike (CRITICAL)
- **Prove deterministic replay works:** Take a recorded Claude API call, serve the recorded response via proxy, verify the agent produces the same subsequent behavior.
- This is the single hardest unknown. If replay fidelity is too low (agent diverges too often), the product does not work.
- Acceptable result: >95% of replays produce identical execution paths for single-model, single-thread agents.

## Week 3-4: Layer 1 MVP

### Deliverables
- [ ] Support for Anthropic SDK, OpenAI SDK, and LangChain
- [ ] Async/streaming support (capture streaming token chunks)
- [ ] Tool call recording (generic HTTP interceptor)
- [ ] Trace viewer CLI (`litmus view ./trace.json` — pretty-prints the trace)
- [ ] Unit tests for SDK instrumentation
- [ ] GitHub release with proper packaging, CI, docs

### Launch
- [ ] Publish to PyPI as `litmus-trace`
- [ ] Post on Hacker News: "Show HN: Record and replay any AI agent execution"
- [ ] Share on Twitter/X, LinkedIn, relevant Discord servers
- [ ] Submit to awesome-langchain, awesome-llm lists

## Week 5-6: Layer 2 core (replay engine)

### Deliverables
- [ ] Proxy server that intercepts LLM calls and serves recorded responses
- [ ] Sequential matching (serve responses in recorded order)
- [ ] Streaming replay (replay token-by-token at recorded cadence)
- [ ] Divergence detection (flag when replayed execution differs from original)
- [ ] CLI: `litmus replay ./trace.json`
- [ ] Replay fidelity measurement: run 100 replays, report match rate

## Week 7-8: Layer 2 hardening + Layer 3 basics

### Deliverables
- [ ] Content-based request matching (handle concurrent calls)
- [ ] Multi-model support (Claude + GPT in same trace)
- [ ] Tool call replay (generic HTTP mock)
- [ ] Basic fault injection: 5 fault types
  - LLM timeout
  - LLM hallucination (random response)
  - Tool HTTP 500
  - Tool timeout
  - Tool empty response
- [ ] CLI: `litmus replay --fault llm_timeout:step=s003 ./trace.json`

## Week 9-10: Litmus Cloud MVP

### Deliverables
- [ ] Trace upload API (accept traces from SDK, store in S3)
- [ ] Web dashboard: list traces, view trace details, trigger replay
- [ ] Replay-as-a-service: trigger replay from dashboard or API
- [ ] Basic auth (API keys for SDK, OAuth for dashboard)
- [ ] Fault injection UI: select a step, choose a fault, replay

## Week 11-12: Design partner onboarding + Layer 4 stub

### Deliverables
- [ ] Onboard 3-5 design partners (hands-on installation support)
- [ ] Collect feedback on trace format, replay fidelity, fault types
- [ ] Layer 4 stub: run N replays (normal + faulted), report pass/fail rate
- [ ] Basic reliability score (percentage of scenarios where agent completed correctly)
- [ ] CI/CD integration: GitHub Action that runs replay suite and reports score
- [ ] `litmus ci --threshold 80 --trace-dir ./traces/` — blocks deploy if score < 80

## End of 90 days: what you should have

- OSS package with 500+ GitHub stars
- Replay engine with >95% fidelity for single-model agents
- 5 fault types working
- 3-5 design partners using it
- Basic cloud dashboard
- CI/CD integration prototype
- First paying customer conversations started

---

# 13. PRICING & REVENUE MODEL

## Pricing tiers

| Tier | Layers | Price | Traces/mo | Target |
|------|--------|-------|-----------|--------|
| **Free** | L1 (OSS) | $0 | Unlimited local | Individual devs, OSS community |
| **Cloud Free** | L1 + L2 (limited) | $0 | 1K traces | Tryout, small teams |
| **Team** | L1 + L2 + L3 | $5K/mo | 50K traces | AI platform teams (Series B-D) |
| **Enterprise** | L1 + L2 + L3 + L4 | $15K/mo | 500K traces | VP Eng, compliance teams |
| **Platform** | All layers | $25K-50K/mo | Unlimited | RL training teams, AI-first companies |

## Revenue math to $1M ARR

| Scenario | Customers | Avg ACV | ARR |
|----------|-----------|---------|-----|
| Conservative | 10 | $100K | $1.0M |
| Base case | 12 | $85K | $1.02M |
| Aggressive | 15 | $80K | $1.2M |

$1M ARR with <15 customers is achievable because:
- Enterprise AI teams have $50K-$200K annual tooling budgets
- Agent reliability is a board-level concern (compliance, risk)
- The alternative (manual testing, prayer) costs more in engineer time

## Usage-based expansion

Beyond the base fee, charge for:
- **Extra traces:** $0.01 per trace above the included amount
- **Replay compute:** $0.10 per replay-hour (heavy fault injection suites)
- **Environment synthesis:** $0.50 per synthetic scenario generated
- **RL training compute:** Metered per episode-hour

This creates natural expansion revenue as customers use the product more.

## Comparison to alternatives

| Tool | Price | What you get |
|------|-------|-------------|
| LangSmith Plus | $400/mo | Logging + basic evals |
| Braintrust Pro | $500/mo | Eval framework + datasets |
| Datadog APM | $3K-10K/mo | Infrastructure monitoring (no agent semantics) |
| **Litmus Team** | **$5K/mo** | **Replay + fault injection + reliability measurement** |
| **Litmus Enterprise** | **$15K/mo** | **Full reliability scoring + CI/CD integration + compliance** |

We are 3-10x more expensive than observability tools because we solve a harder problem that they cannot.

---

# 14. MOAT & DEFENSIBILITY

## Why this gets harder to compete with over time

### Moat 1: Trace corpus (data moat)
- Every customer generates traces that improve our fault injection library, scoring models, and synthesis engine
- A customer with 100K traces gets 10x better testing than one with 1K
- Cross-customer patterns (anonymized) improve the engine for all customers
- New competitor starting from zero has no corpus to learn from

### Moat 2: Integration depth (switching cost)
- CI/CD pipeline: "replay all traces, assert score > 90 before deploy"
- Oncall runbook: "first step when agent fails: litmus replay the trace"
- Compliance reports: "Litmus Reliability Score attached to quarterly filing"
- Historical baselines: "agent v2 scores 91 vs v1's 78" — lose history by switching

### Moat 3: Framework coverage (ecosystem lock-in)
- Instrumenting LangChain requires deep knowledge of LangChain internals
- Same for CrewAI, AutoGen, OpenAI Agents SDK, custom frameworks
- Each framework integration is 2-4 weeks of work
- A competitor must replicate ALL integrations to be a viable alternative

### Moat 4: Reliability scoring calibration (institutional knowledge)
- "What does a score of 85 mean?" requires calibrating against real-world outcomes
- This calibration improves over time as we correlate scores with actual production failures
- A new competitor's scores have no calibration — customers cannot trust them

### What is NOT a moat
- "Hard to build" is a 12-month moat, not a 5-year moat. Someone will replicate the replay engine.
- First-mover advantage erodes. We must compound the data and integration moats before a funded competitor catches up on features.

---

# 15. KILL RISKS & MITIGATIONS

## Risk 1: Model providers add replay natively (MODERATE)
**Scenario:** Anthropic/OpenAI add a "replay" parameter to their API: send a trace ID, get the exact same response.
**Impact:** Layer 2 becomes commodity for single-model agents.
**Mitigation:** Real agents use multiple models and tools. Provider-level replay only solves their own calls. Litmus is multi-model, multi-tool, multi-framework.
**Probability:** 20% within 18 months.

## Risk 2: Antithesis moves up-stack (LOW-MEDIUM)
**Scenario:** With $182M, Antithesis builds an LLM-aware layer on top of their hypervisor.
**Impact:** They would have deeper determinism (CPU-level) plus agent semantics.
**Mitigation:** Different architecture entirely. Their approach requires running the full LLM locally — impractical for most agents that call cloud APIs. It would be a 12+ month product development effort and a completely different go-to-market.
**Probability:** 15% within 24 months.

## Risk 3: Market does not materialize fast enough (MODERATE)
**Scenario:** Agents stay mostly demos/prototypes through 2026-2027. Not enough production agent teams to sell to.
**Impact:** Cannot reach $1M ARR in 12 months. Must extend runway.
**Mitigation:** 121 evidence rows say otherwise. LangChain survey: 89% observability adoption. Box, Dreamer, Composio all indicate production agents are real and growing. But this remains the biggest existential risk.
**Probability:** 25%.

## Risk 4: "Good enough" evals win (LOW-MEDIUM)
**Scenario:** Teams decide LangSmith logs + manual review + vibes-based testing is sufficient.
**Impact:** Market exists but buyers do not upgrade from free/cheap tools.
**Mitigation:** This is true for prototypes, not for agents that handle money, healthcare, or legal decisions. Regulated industries MUST quantify reliability. Focus sales on these verticals.
**Probability:** 20% (for our target segment — regulated/high-stakes).

## Risk 5: LLM non-determinism is solved (LOW)
**Scenario:** Model providers guarantee deterministic outputs (same seed = same response).
**Impact:** Replay becomes trivial. Our competitive advantage shrinks.
**Mitigation:** Even with deterministic LLMs, tool calls remain non-deterministic. And fault injection/scoring/synthesis still provide value regardless. Also: deterministic LLMs would be GOOD for us — it makes replay easier and more reliable.
**Probability:** 10% within 24 months.

## Risk 6: Felix Rieseberg's warning — Anthropic prototypes in 10 days (HIGH CONCEPTUAL RISK)
**Scenario:** Anthropic (or any model provider) decides to build testing infrastructure and can ship an MVP in 10 days.
**Impact:** Feature-level competition from a well-resourced platform.
**Mitigation:** Build BELOW the API layer where model providers will not go. Our proxy intercepts HTTP calls — it works regardless of which model or framework is used. Model providers want to sell inference, not testing infrastructure. Also: 10 days gets you a prototype, not a production-grade replay engine with fault injection and scoring.
**Probability:** 30% for a prototype announcement. 5% for a serious product.

---

# 16. OPEN QUESTIONS & TECHNICAL SPIKES

## Spike 1: Replay fidelity (MUST DO FIRST)
**Question:** What percentage of real production traces can be replayed with bit-exact fidelity?
**Experiment:** Record 100 Claude Code sessions. Replay each 10 times. Measure: same tool calls in same order? Same final output?
**Pass criteria:** >95% for single-model, single-thread agents. >80% for multi-model.
**Timeline:** Week 1-2.

## Spike 2: Streaming fidelity
**Question:** Can we replay streaming token responses with enough fidelity that streaming handlers behave identically?
**Experiment:** Record a streaming response, replay it, compare the handler's intermediate state at each chunk.
**Why it matters:** Many agents process streaming tokens incrementally (e.g., tool call detection mid-stream).
**Timeline:** Week 3.

## Spike 3: Multi-model matching
**Question:** When an agent calls Claude AND GPT in the same execution (common in routing/fallback patterns), can we correctly match each call to its recorded response?
**Experiment:** Build a simple dual-model agent. Record. Replay. Verify each call goes to the right recorded response.
**Timeline:** Week 5.

## Spike 4: State mutation tracking
**Question:** If an agent reads from a database, writes to it, then reads again, can we replay the stateful tool chain?
**Experiment:** Build an agent that CRUDs a database. Record the full session. Replay with mocked database responses.
**Risk:** If the agent's code branches on local state that depends on the database write, the replay may diverge.
**Timeline:** Week 7.

## Spike 5: Performance at scale
**Question:** Can we replay 10,000 traces with fault injection in under 1 hour? (needed for CI/CD integration)
**Experiment:** Generate 10K synthetic traces. Run parallel replay with 3 fault types each. Measure wall time.
**Timeline:** Week 9.

## Open design decisions

1. **Trace format:** JSON vs Protobuf vs custom binary. JSON is readable but large. Protobuf is compact but less debuggable. Start JSON, optimize later.

2. **OSS scope:** How much of Layer 2 is open source? Options:
   - a) Only Layer 1 is OSS. Layer 2+ is proprietary. (stronger monetization)
   - b) Layers 1-2 are OSS. Layer 3+ is proprietary. (more adoption, Docker Cagent competition)
   - c) All layers OSS, charge for cloud hosting + support. (maximum adoption, harder to monetize)
   - **Recommendation:** Option (a) for now. Layer 1 OSS with clean, well-documented trace format. Layer 2+ proprietary. Revisit if Docker Cagent gains traction.

3. **Sandbox architecture:** How do we safely replay agents that call real tools?
   - Option: Run agent code in a containerized sandbox with all external calls intercepted by proxy.
   - Risk: Some agent code has side effects (writes to disk, modifies env vars) that are hard to sandbox.
   - Decision: Start with "replay only intercepts network calls" and document the limitation.

4. **Naming conventions:**
   - A recorded trace: "recording" or "flight log" or "trace"
   - A replayed execution: "replay" or "simulation" or "run"
   - A faulted replay: "scenario" or "perturbation" or "stress test"
   - A reliability score: "Litmus Score" (brand it)

5. **Pricing model for compute:**
   - Per-trace pricing (simple, predictable)
   - Per-replay pricing (incentivizes storing traces but not replaying — wrong incentive)
   - Per-seat pricing (enterprise-friendly, but does not scale with usage)
   - **Recommendation:** Per-trace with included replay budget. Overage charges for heavy usage.

---

# 17. KEY EVIDENCE SUMMARY (from 121 podcast evidence rows)

## Market signals
- LangChain Survey: 89% observability, only 52% evals — 37-point eval gap (our opportunity)
- Futurum Research (N=393): AI observability ranks 4th (37.4%) in enterprise procurement priorities
- Bunnyshell: Trust in AI code accuracy dropped from 40% to 29% YoY
- SAASpocalypse: $300B wiped from software stocks. Infrastructure is safe harbor.

## Technical validation
- MiniMax (Olive Song): Perturbation pipelines = our Layer 3. "Agent generalization is adaptation to perturbations across entire operational space."
- Harmonic (Tudor Achim): "Hallucinations are intrinsic to LLMs. You need a verification layer on top."
- METR (Joel Becker): "Devs NOT measurably more productive despite feeling so." = our Layer 4 fills the measurement gap.
- CodeClash (John Yang): "Unit tests as verification are limiting. Long-running agent tournaments are the future."

## Structural shifts
- Felix Rieseberg (Anthropic): Cowork built in 10 days. Must build BELOW the API layer.
- Aaron Levie (Box): "2025 was coding agents. 2026 is knowledge work agents." = TAM expanding.
- Paul Dix: "Great Engineering Divergence" — coding speed jumps, everything else becomes constraint.
- Addy Osmani: Comprehension debt — AI generates 5-7x faster than humans can audit.

## Competitive moat
- Arize AI: "Traces are the source of truth. Every operation must now be performed on traces." = our L1 is the right foundation.
- Elliot Bonneville: "The only moat left is money." = data moat + integration depth, not just hard-to-build.
- Sarah Guo: "Real-world logs beat synthetic clones every time." = our real-trace approach validated over Maxim's synthetic approach.

---

# 18. WHAT SUCCESS LOOKS LIKE

## 6-month milestones
- [ ] 1,000+ GitHub stars on litmus-trace
- [ ] 200+ weekly pip installs
- [ ] 8+ paying customers
- [ ] $400K+ ARR run rate
- [ ] Replay fidelity >95% for supported frameworks
- [ ] 10+ fault types in the injection library
- [ ] CI/CD integration with GitHub Actions

## 12-month milestones
- [ ] $1M+ ARR
- [ ] 15+ customers including 2+ enterprise logos
- [ ] SOC 2 Type I certified
- [ ] Layer 4 reliability scoring in production
- [ ] "Litmus Score" referenced in 5+ external blog posts
- [ ] Self-hosted option available for enterprise
- [ ] 3+ framework integrations (LangChain, CrewAI, OpenAI Agents SDK)

## 24-month milestones
- [ ] $5M+ ARR
- [ ] Layer 5 (environment synthesis) in production
- [ ] Layer 6 (RL bridge) in beta
- [ ] Data moat visibly compounding
- [ ] Industry recognition as standard for agent reliability
- [ ] Series A raised (if needed)

---

*This blueprint was generated from 121 podcast evidence rows across 50+ sources, 10 research batches, 8 methodology iterations, and 120+ candidate ideas tested. The Deterministic Agent Simulation Engine (now Litmus) was the sole survivor at 8.8/10 conviction.*
