# OpenMedicine Agent — Architecture & Goals

> This document initializes the `openmedicine-agent` repo. It serves as both CLAUDE.md and AGENTS.md.
> Feed this file to Claude Code when starting the project.

## What This Is

An open-source medical reasoning agent that connects to the [OpenMedicine](https://github.com/open-medicine/openmedicine) MCP server to perform deterministic, auditable clinical calculations. Inspired by [OpenClaw](https://github.com/openclaw/openclaw)'s architecture (Gateway → Brain → Skills → Memory), adapted for clinical safety requirements.

**This is NOT a diagnostic tool.** It is a research platform for validating whether LLM-based agents can reliably orchestrate clinical tools (calculators, guidelines) without hallucinating medical data.

## Goals

1. **Validate LLM + MCP for clinical tasks** — Can agents correctly call deterministic tools instead of guessing at medical calculations?
2. **Benchmark multiple LLMs** — Compare Claude, GPT-4, Gemini, and open-source models on identical clinical cases
3. **Auditable reasoning** — Every decision traces back to a DOI-referenced tool output, never to LLM "knowledge"
4. **Support 3 active research studies** (PIBIC/TFG, Faculdade Evangélica Mackenzie do Paraná):
   - **Study 1:** eGFR-based renal dose adjustment (CKD-EPI → dose lookup)
   - **Study 2:** Automated Glasgow Coma Score from clinical narratives
   - **Study 3:** CHA₂DS₂-VASc calculation for AF stroke risk
5. **Reusable framework** — After the studies, this becomes a general-purpose medical agent scaffold

## Architecture

```
┌─────────────────────────────────────────────────────┐
│                    GATEWAY                          │
│  Case ingestion: synthetic patients, chat, API      │
│  Channels: CLI, API endpoint, (future: messaging)   │
└──────────────────────┬──────────────────────────────┘
                       │ patient case
                       ▼
┌─────────────────────────────────────────────────────┐
│                     BRAIN                           │
│  ReAct loop with clinical guardrails                │
│  Pluggable LLM: Claude │ GPT-4 │ Gemini │ Ollama   │
│                                                     │
│  1. Receive case → extract relevant variables       │
│  2. Decide which MCP tools to call                  │
│  3. Call tools via MCP client (MANDATORY, no guess)  │
│  4. Synthesize result from tool outputs             │
│  5. Log full chain-of-thought                       │
└──────────┬──────────────────────┬───────────────────┘
           │ MCP tool calls       │ persist
           ▼                      ▼
┌─────────────────────┐  ┌────────────────────────────┐
│       SKILLS        │  │          MEMORY             │
│  MCP Client ──────────►│  Session context            │
│  connects to        │  │  Chain-of-thought log       │
│  OpenMedicine       │  │  Audit trail (JSON-lines)   │
│  MCP Server         │  │  Patient state              │
│  (stdio transport)  │  └────────────────────────────┘
│                     │
│  Tools available:   │
│  - 92 calculators   │
│  - 43 guidelines    │
│  - search + execute │
└─────────────────────┘
           │ results
           ▼
┌─────────────────────────────────────────────────────┐
│                  EVALUATION                         │
│  Gold standard comparison (expert-validated cases)  │
│  Metrics: Accuracy, MAE, Cohen's Kappa              │
│  Failure analysis: categorize + clinical severity   │
│  Batch runner: N cases × M models = results matrix  │
└─────────────────────────────────────────────────────┘
```

## Core Principle: Mandatory Tool Use

The single most important design rule:

> **The agent MUST call OpenMedicine's deterministic MCP tools for any clinical calculation. It must NEVER compute scores, doses, or risk stratifications from its own parametric knowledge.**

This is enforced by:
1. System prompt instructs the LLM to always use tools
2. The evaluation harness checks that tool calls were made (protocol adherence rate)
3. Results without a tool-call trace are flagged as failures regardless of correctness

## Project Structure

```
openmedicine-agent/
├── CLAUDE.md                     # → symlink to AGENTS.md
├── AGENTS.md                     # Agent instructions (this file, trimmed)
├── pyproject.toml                # Python package config
├── src/
│   └── openmedicine_agent/
│       ├── __init__.py
│       │
│       ├── gateway/              # Case ingestion layer
│       │   ├── __init__.py
│       │   ├── cli.py            # Interactive CLI mode
│       │   ├── api.py            # REST API for batch processing
│       │   └── loader.py         # Load cases from JSON/CSV files
│       │
│       ├── brain/                # Clinical reasoning engine
│       │   ├── __init__.py
│       │   ├── react.py          # ReAct loop with clinical guardrails
│       │   ├── guardrails.py     # Safety rules (mandatory tool use, contraindication halt)
│       │   └── providers/        # LLM backends
│       │       ├── __init__.py
│       │       ├── base.py       # Abstract provider interface
│       │       ├── claude.py     # Anthropic Claude (primary)
│       │       ├── openai.py     # OpenAI GPT-4
│       │       ├── gemini.py     # Google Gemini
│       │       └── ollama.py     # Local models via Ollama
│       │
│       ├── skills/               # MCP tool integration
│       │   ├── __init__.py
│       │   ├── mcp_client.py     # Connects to OpenMedicine MCP server
│       │   └── tool_registry.py  # Discovers and caches available tools
│       │
│       ├── memory/               # Context and audit persistence
│       │   ├── __init__.py
│       │   ├── session.py        # Current encounter context window
│       │   ├── audit.py          # JSON-lines audit log (every tool call + result)
│       │   └── chain_of_thought.py  # Captures LLM reasoning trace
│       │
│       ├── workflows/            # Study-specific clinical workflows
│       │   ├── __init__.py
│       │   ├── renal_dosing.py   # Study 1: eGFR → CKD stage → dose adjustment
│       │   ├── gcs_assessment.py # Study 2: narrative → GCS components → score
│       │   └── chadsvasc.py      # Study 3: patient data → CHA₂DS₂-VASc → anticoag
│       │
│       ├── evaluation/           # Research evaluation harness
│       │   ├── __init__.py
│       │   ├── runner.py         # Batch: N cases × M models → results
│       │   ├── gold_standard.py  # Load expert-validated answers
│       │   ├── metrics.py        # Accuracy, MAE, Kappa, sensitivity, specificity
│       │   ├── failure_analysis.py  # Categorize errors + clinical significance
│       │   └── report.py         # Generate comparison tables and summaries
│       │
│       └── safety/               # Clinical safety layer
│           ├── __init__.py
│           ├── validators.py     # Validate tool outputs before synthesis
│           └── flags.py          # Contraindication halt, metric mismatch propagation
│
├── studies/                      # Per-study synthetic data and configs
│   ├── renal_dosing/
│   │   ├── synthetic_patients.json
│   │   ├── gold_standard.json
│   │   └── config.yaml           # Which tools, which LLMs, which metrics
│   ├── gcs_assessment/
│   │   ├── synthetic_patients.json
│   │   ├── gold_standard.json
│   │   └── config.yaml
│   └── chadsvasc/
│       ├── synthetic_patients.json
│       ├── gold_standard.json
│       └── config.yaml
│
├── tests/
│   ├── test_mcp_client.py
│   ├── test_react_loop.py
│   ├── test_guardrails.py
│   ├── test_evaluation.py
│   └── test_workflows.py
│
└── docs/
    └── plans/
```

## Tech Stack

| Component | Technology | Why |
|-----------|-----------|-----|
| Language | Python 3.10+ | Matches OpenMedicine |
| Package manager | uv | Matches OpenMedicine |
| LLM SDKs | anthropic, openai, google-genai | Multi-model benchmarking |
| MCP client | mcp (Python SDK) | Connects to OpenMedicine server |
| Data validation | Pydantic | Matches OpenMedicine patterns |
| CLI | typer or click | Simple interactive mode |
| API | FastAPI | Batch processing endpoint |
| Testing | pytest + Hypothesis | Matches OpenMedicine |
| Audit log format | JSON-lines (.jsonl) | Appendable, parseable, one record per line |

## MCP Client Connection

The agent connects to OpenMedicine's MCP server over stdio:

```python
# The MCP client spawns the OpenMedicine server as a subprocess
# and communicates via stdin/stdout using the MCP protocol.
#
# Available meta-tools:
#   search_clinical_calculators(query) → list of matching calculators
#   execute_clinical_calculator(name, params) → ClinicalResult
#   search_guidelines(query) → list of matching guidelines
#   retrieve_guideline(id, section) → guideline content
#
# The agent NEVER imports OpenMedicine directly.
# It ALWAYS goes through MCP. This is the real integration path.
```

## ReAct Loop with Clinical Guardrails

```
SYSTEM PROMPT:
  You are a clinical reasoning agent. You have access to medical
  calculators and guidelines via MCP tools.

  RULES:
  1. NEVER compute clinical scores, doses, or risk levels yourself.
     ALWAYS use the provided tools.
  2. Before recommending a dose, you MUST call the appropriate
     calculator. If no calculator exists, state that explicitly.
  3. If a tool returns "contraindicated", you MUST prominently flag
     this in your response. Do not downplay contraindications.
  4. If a tool returns a metric_mismatch_warning, include it in
     your response verbatim.
  5. Show your reasoning step by step. Each step must reference
     which tool you called and what it returned.

LOOP:
  1. THINK: What clinical question am I answering? What data do I have?
  2. ACT: Call the appropriate MCP tool with the patient's data
  3. OBSERVE: Read the tool's ClinicalResult (value, interpretation, evidence)
  4. REPEAT if more tools needed (e.g., CKD-EPI → then dose adjustment)
  5. SYNTHESIZE: Combine tool outputs into a clinical recommendation
     - Cite the DOI from each tool's evidence
     - Include any warnings or mismatch flags
```

## Workflow Examples

### Study 1: Renal Dose Adjustment

```
Input: Patient case with demographics + serum creatinine + medication list

Agent chain:
  1. Call calculate_ckd_epi(age, sex, creatinine) → eGFR
  2. For each medication needing renal adjustment:
     Call calculate_renal_dose_adjustment(drug, eGFR, "egfr") → adjusted dose
  3. If drug has metric_mismatch (label uses CrCl):
     Also call calculate_cockcroft_gault(age, sex, weight, creatinine) → CrCl
     Re-call calculate_renal_dose_adjustment(drug, CrCl, "crcl")
  4. Synthesize: list of drugs with adjusted doses, DOI references, warnings

Gold standard: Nephrologist-validated dose for each drug
Metrics: Accuracy (exact match), MAE (dose difference), Kappa (tier agreement)
```

### Study 2: Glasgow Coma Score

```
Input: Clinical narrative describing patient's eye/verbal/motor responses

Agent chain:
  1. Extract GCS components from narrative (eye, verbal, motor responses)
  2. Call calculate_gcs(eye, verbal, motor) → GCS score + interpretation
  3. If pediatric: call calculate_pediatric_gcs instead
  4. Optionally: call retrieve_guideline("btf_tbi_2016", "management")

Gold standard: Specialist-calculated GCS from same narrative
Metrics: Accuracy (exact score match), component-level accuracy, MAE
```

### Study 3: CHA₂DS₂-VASc

```
Input: Patient case with demographics + comorbidities + AF diagnosis

Agent chain:
  1. Extract risk factors from case (CHF, HTN, age, DM, stroke hx, vasc disease, sex)
  2. Call calculate_chadsvasc(extracted factors) → score + anticoag recommendation
  3. If score indicates anticoagulation:
     Call search_guidelines("atrial fibrillation anticoagulation")
     Call retrieve_guideline("acc_aha_af_2023", "anticoagulation")
  4. Synthesize: score + recommendation + guideline reference

Gold standard: Cardiologist-calculated score from same case
Metrics: Accuracy (exact score match), MAE, Kappa (risk tier agreement)
```

## Evaluation Harness

The evaluation system is first-class, not an afterthought:

```yaml
# studies/chadsvasc/config.yaml
study:
  name: "CHA2DS2-VASc Validation"
  workflow: "chadsvasc"

models:
  - provider: claude
    model: claude-sonnet-4-20250514
  - provider: openai
    model: gpt-4o
  - provider: gemini
    model: gemini-2.0-flash

data:
  synthetic_patients: "synthetic_patients.json"
  gold_standard: "gold_standard.json"

metrics:
  - accuracy          # Exact score match
  - mae               # Mean absolute error
  - kappa             # Cohen's Kappa (inter-rater reliability)
  - sensitivity       # Per risk tier
  - specificity       # Per risk tier
  - protocol_adherence # Did the agent call the right tools?
  - cot_quality       # Was reasoning auditable and correct?

failure_analysis:
  categories:
    - extraction_error     # Wrong data extracted from case
    - tool_call_error      # Wrong tool or wrong parameters
    - synthesis_error       # Correct tool output, wrong conclusion
    - hallucination         # Agent computed without tool call
    - contraindication_miss # Failed to flag contraindication
```

## Audit Log Format

Every agent run produces a JSON-lines audit file:

```jsonl
{"timestamp": "...", "event": "case_received", "case_id": "PT-001", "model": "claude-sonnet-4-20250514"}
{"timestamp": "...", "event": "think", "content": "Patient has AF, need CHA2DS2-VASc..."}
{"timestamp": "...", "event": "tool_call", "tool": "calculate_chadsvasc", "params": {"age": 72, ...}}
{"timestamp": "...", "event": "tool_result", "tool": "calculate_chadsvasc", "value": {"score": 4, ...}, "doi": "10.1161/..."}
{"timestamp": "...", "event": "think", "content": "Score is 4, anticoagulation indicated..."}
{"timestamp": "...", "event": "tool_call", "tool": "retrieve_guideline", "params": {"id": "acc_aha_af_2023", ...}}
{"timestamp": "...", "event": "synthesis", "recommendation": "...", "tool_calls_made": 2}
{"timestamp": "...", "event": "evaluation", "predicted": 4, "gold_standard": 4, "correct": true}
```

## Synthetic Patient Generator

Each study needs a generator that produces clinically plausible cases. Requirements:

1. **Statistically representative** — prevalence and comorbidity distributions match real populations
2. **Expert-validated** — nephrologists (Study 1), neurologists (Study 2), cardiologists (Study 3) review and validate
3. **Gold standard attached** — each case includes the expert-calculated correct answer
4. **Edge cases included** — boundary values, atypical presentations, contraindicated scenarios
5. **No real patient data** — fully synthetic, no ethics board required (per Resolucao CNS 510/2016)

Output format per case:
```json
{
  "case_id": "PT-001",
  "narrative": "72-year-old male with history of...",
  "structured_data": {
    "age": 72,
    "sex": "male",
    "serum_creatinine": 2.1,
    "medications": ["vancomycin", "metformin"],
    ...
  },
  "gold_standard": {
    "egfr": 28.5,
    "vancomycin_dose": "15-20 mg/kg IV q24-48h",
    "metformin_dose": "CONTRAINDICATED",
    "validated_by": "Dr. X (Nephrology)",
    "validation_date": "2025-09-15"
  }
}
```

## Build Order

### Phase 1: Foundation (Week 1-2)
1. Repo setup: pyproject.toml, uv, directory structure
2. MCP client: connect to OpenMedicine, list tools, execute one calculator
3. Minimal Brain: Claude provider + basic ReAct loop
4. CLI gateway: interactive mode, single case

### Phase 2: First Workflow (Week 3-4)
5. CHA₂DS₂-VASc workflow (simplest study, one calculator)
6. Audit log: JSON-lines trace of every tool call
7. Evaluation harness: load gold standard, compute accuracy + Kappa
8. 10 synthetic CHA₂DS₂-VASc cases for testing

### Phase 3: Multi-Model Benchmarking (Week 5-6)
9. Add OpenAI provider
10. Add Gemini provider
11. Batch runner: N cases × M models
12. Comparison report generator

### Phase 4: Remaining Workflows (Week 7-10)
13. GCS workflow (Study 2) — harder, needs narrative extraction
14. Renal dosing workflow (Study 1) — multi-step chain (eGFR → dose lookup)
15. Full synthetic patient generators (expert validation happens in parallel)
16. Failure analysis module

### Phase 5: Scale & Polish (Week 11-12)
17. API gateway (FastAPI) for batch processing
18. Guardrails hardening (contraindication halt, metric mismatch propagation)
19. Add Ollama provider for local models
20. Documentation and study-specific README per workflow

## Commands

```bash
uv sync                                          # Install dependencies
uv run agent chat                                 # Interactive CLI mode
uv run agent run studies/chadsvasc/config.yaml    # Batch evaluation
uv run agent report studies/chadsvasc/results/    # Generate comparison report
uv run python -m pytest -v                        # Run tests
```

## Dependencies on OpenMedicine

This agent requires OpenMedicine >= 0.4.0 installed and its MCP server runnable via:

```bash
uv run open-medicine-mcp
```

The agent spawns this process and communicates over stdio. It does NOT import OpenMedicine as a Python library.

## License

MIT (matches OpenMedicine and OpenClaw)
