# simboba

Lightweight eval tracking with LLM-as-judge. Users write Python scripts to run evals, results are stored as JSON files (git-trackable) and viewable in a web UI.

## Quick Start

```bash
pip install -e .
boba init        # Create boba-evals/ folder
boba magic       # Print AI prompt to configure evals
boba run         # Run evals (handles Docker automatically)
boba baseline    # Save run as baseline for regression detection
boba serve       # View results
```

## Project Structure

```
simboba/
├── simboba/
│   ├── __init__.py       # Exports Boba class
│   ├── boba.py           # Core Boba class (eval, run methods)
│   ├── cli.py            # Click CLI (init, magic, generate, run, baseline, serve, datasets, reset)
│   ├── config.py         # Configuration (.boba.yaml) and Docker exec
│   ├── storage.py        # JSON file storage operations
│   ├── schemas.py        # Pydantic models for validation
│   ├── server.py         # FastAPI REST API
│   ├── judge.py          # LLM judge implementation
│   ├── prompts/          # LLM prompts for generation and judging
│   ├── utils/            # LLM client utilities
│   ├── samples/          # Template files copied by `boba init`
│   │   ├── setup.py      # Test fixtures template
│   │   └── test.py  # Eval script template
│   └── static/           # Built React frontend (generated by frontend build)
├── frontend/             # React + TypeScript + Vite frontend source
│   ├── src/
│   │   ├── components/   # UI components (Layout, shadcn-style components)
│   │   ├── pages/        # Page components (Dashboard, Datasets, Runs, Settings)
│   │   ├── hooks/        # React hooks (useStore for state management)
│   │   ├── lib/          # Utilities and API client
│   │   └── types/        # TypeScript type definitions
│   ├── package.json
│   └── vite.config.ts
├── tests/
│   ├── conftest.py       # Pytest fixtures
│   └── test_core_flows.py
└── pyproject.toml
```

## Data Storage

All data is stored as JSON files in the `boba-evals/` directory:

```
boba-evals/
├── datasets/
│   └── {name}.json           # Dataset + all cases (git tracked)
├── baselines/
│   └── {name}.json           # Committed run results (git tracked)
├── runs/
│   └── {dataset}/
│       └── {timestamp}.json  # All runs (gitignored)
├── files/                    # Uploaded attachments (git tracked)
├── settings.json             # App settings
└── .gitignore                # Ignores runs/
```

## CLI vs Python API

The CLI and Python API work together:

- `boba run test.py` executes your Python script, which internally calls `boba.eval()` or `boba.run()`
- The CLI handles Docker detection, environment setup, and script execution
- Your script uses the Python API (`Boba` class) to actually run evaluations

```
┌─────────────────┐      ┌──────────────────┐      ┌─────────────────┐
│  boba run       │ ───> │  test.py         │ ───> │  Boba class     │
│  (CLI command)  │      │  (your script)   │      │  (Python API)   │
└─────────────────┘      └──────────────────┘      └─────────────────┘
```

## Architecture

### Core Class: Boba

```python
from simboba import Boba, AgentResponse

boba = Boba()

# Single eval - judges output against expected
result = boba.eval(
    input="Hello",
    output="Hi there!",
    expected="Should greet the user",
)
# Returns: {"passed": bool, "reasoning": str, "run_id": str}

# Dataset eval - runs agent against all cases in a dataset
result = boba.run(
    agent=my_agent_fn,  # Callable[[list[MessageInput]], str | AgentResponse]
    dataset="my-dataset",
)
# Returns: {"passed": int, "failed": int, "total": int, "score": float, "run_id": str, "regressions": list, "fixes": list}

# With deterministic metadata checking (boba.eval)
def check_citations(expected, actual):
    return expected.get("citations") == actual.get("citations")

result = boba.eval(
    input="What's in section 3?",
    output="Section 3 covers...",
    expected="Should cite the document",
    expected_metadata={"citations": [{"file": "doc.pdf", "page": 3}]},
    actual_metadata={"citations": [{"file": "doc.pdf", "page": 3}]},
    metadata_checker=check_citations,  # Deterministic check instead of LLM
)
```

### AgentResponse for Metadata with boba.run()

To return metadata from your agent when using `boba.run()`, use `AgentResponse`:

```python
from simboba import Boba, AgentResponse, MessageInput

boba = Boba()

# Agent receives full inputs list, returns output and metadata
def my_agent(inputs: list[MessageInput]) -> AgentResponse:
    # inputs is the full conversation history
    # Each input has: role, message, attachments (optional), metadata (optional)
    response = call_my_llm(inputs)
    return AgentResponse(
        output=response.text,
        metadata={
            "citations": [{"file": "doc.pdf", "page": 3}],
            "tool_calls": ["search", "summarize"],
        }
    )

# Metadata checker for deterministic validation
def check_citations(expected, actual):
    if not expected or not actual:
        return True
    return expected.get("citations") == actual.get("citations")

# Run with metadata checking
result = boba.run(
    agent=my_agent,
    dataset="my-dataset",
    metadata_checker=check_citations,
)
```

**Agent input:**
- `inputs: list[MessageInput]` - Full conversation history. Each `MessageInput` has `role`, `message`, and optional `attachments`/`metadata`

**Agent return types:**
- `str` - Simple response, no metadata
- `AgentResponse` - Response with `output` (str) and optional `metadata` (dict)

### Metadata Checking

Three modes for evaluating metadata (citations, tool_calls, etc.):

| Mode                   | How                                              | Behavior                                   |
| ---------------------- | ------------------------------------------------ | ------------------------------------------ |
| 1. No metadata         | Don't pass `expected_metadata`/`actual_metadata` | LLM judges output only                     |
| 2. LLM evaluation      | Pass metadata, no `metadata_checker`             | LLM judges output + metadata together      |
| 3. LLM + deterministic | Pass metadata AND `metadata_checker`             | LLM judges + your function checks metadata |

```python
# metadata_checker signature
def checker(expected_metadata: dict | None, actual_metadata: dict | None) -> bool:
    """Return True if metadata matches expectations."""
```

When `metadata_checker` is provided:

- LLM still sees metadata for context/reasoning
- Your function runs as an additional gate
- Case passes only if **both** checks pass
- Results include `expected_metadata`, `actual_metadata`, and `metadata_passed` fields

### Data Model

- **Dataset**: Named collection of eval cases (JSON file)
- **Case**: Test case with inputs, expected outcome, optional source reference
- **Run**: One execution with results (JSON file per run)
- **Baseline**: Committed run results for regression detection

### Dataset JSON Structure

```json
{
  "id": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
  "name": "customer-support",
  "description": "Customer support chatbot test cases",
  "cases": [
    {
      "id": "case-001",
      "name": "Order status inquiry",
      "inputs": [
        {"role": "user", "message": "What's the status of my order #12345?"}
      ],
      "expected_outcome": "Agent should look up the order and provide status",
      "expected_metadata": {
        "tool_calls": ["get_order_status"],
        "citations": []
      }
    },
    {
      "id": "case-002",
      "name": "Return policy question",
      "inputs": [
        {"role": "user", "message": "How do I return an item?"}
      ],
      "expected_outcome": "Agent should explain return policy clearly"
    }
  ],
  "created_at": "2024-01-15T10:30:00Z",
  "updated_at": "2024-01-15T10:30:00Z",
  "case_count": 2
}
```

### Case Structure

```python
{
    "id": "abc123",  # Stable ID for baseline comparison
    "name": "Test greeting",
    "inputs": [
        {
            "role": "user",
            "message": "...",
            "attachments": [{"file": "doc.pdf"}],
            "metadata": {...}  # optional - for tool_calls, citations, etc.
        }
    ],
    "expected_outcome": "Agent should...",
    "expected_metadata": {      # optional - expected citations, tool_calls, etc.
        "citations": [{"file": "doc.pdf", "page": 12}],
        "tool_calls": ["get_orders"]
    }
}
```

### Multi-turn Conversations

The `inputs` list supports multi-turn conversations with alternating `user` and `assistant` roles. The agent receives the full conversation history and should respond to the **last message**, using prior messages as context.

**Multi-turn case example:**

```json
{
  "name": "Follow-up question about order",
  "inputs": [
    {"role": "user", "message": "What's the status of my order #12345?"},
    {"role": "assistant", "message": "Your order #12345 was shipped yesterday and is expected to arrive by Friday."},
    {"role": "user", "message": "Can I change the delivery address?"}
  ],
  "expected_outcome": "Agent should explain how to change delivery address or whether it's still possible given the order has shipped"
}
```

**How it works:**

1. The agent function receives all three messages as `inputs: list[MessageInput]`
2. The agent should use the conversation history as context (e.g., pass to an LLM)
3. The agent returns a response to the last user message ("Can I change the delivery address?")
4. The judge evaluates whether that response meets `expected_outcome`

**Agent implementation for multi-turn:**

```python
def agent(inputs: list[MessageInput]) -> str:
    # Convert to your LLM's format (e.g., Anthropic messages)
    messages = [{"role": m.role, "content": m.message} for m in inputs]

    # Call your LLM with full history
    response = client.messages.create(
        model="claude-sonnet-4-20250514",
        messages=messages,
    )
    return response.content[0].text
```

**Single-turn vs multi-turn:**

| Type | `inputs` contains | Agent behavior |
|------|-------------------|----------------|
| Single-turn | One user message | Respond to that message |
| Multi-turn | User/assistant alternating, ends with user | Respond to last message with prior context |

### CLI Commands

| Command                | Description                                                           |
| ---------------------- | --------------------------------------------------------------------- |
| `boba init`            | Create `boba-evals/` folder with datasets/, baselines/, runs/, files/ |
| `boba init --docker`   | Quick setup for Docker Compose                                        |
| `boba init --local`    | Quick setup for local Python                                          |
| `boba magic`           | Print detailed AI prompt to configure eval scripts                    |
| `boba run [script]`    | Run eval script (default: `test.py`). Handles Docker automatically    |
| `boba baseline`        | Interactive - list recent runs, select one to save as baseline        |
| `boba serve`           | Start web UI at localhost:8787                                        |
| `boba datasets`        | List all datasets                                                     |
| `boba generate "desc"` | Generate dataset from description                                     |
| `boba reset`           | Clear run history (keeps datasets and baselines)                      |

### API Endpoints

| Endpoint                         | Method           | Description              |
| -------------------------------- | ---------------- | ------------------------ |
| `/api/datasets`                  | GET, POST        | List/create datasets     |
| `/api/datasets/{name}`           | GET, PUT, DELETE | CRUD for dataset         |
| `/api/datasets/{name}/export`    | GET              | Export dataset as JSON   |
| `/api/datasets/import`           | POST             | Import dataset from JSON |
| `/api/cases`                     | GET, POST        | List/create cases        |
| `/api/cases/{dataset}/{id}`      | GET, PUT, DELETE | CRUD for single case     |
| `/api/cases/bulk`                | POST             | Bulk create cases        |
| `/api/runs`                      | GET              | List runs                |
| `/api/runs/{dataset}/{filename}` | GET, DELETE      | Get/delete run           |
| `/api/baselines`                 | GET              | List baselines           |
| `/api/baselines/{dataset}`       | GET              | Get baseline for dataset |
| `/api/settings`                  | GET, PUT         | Get/update settings      |
| `/api/files/upload`              | POST             | Upload file              |
| `/api/files/{filename}`          | GET              | Get file                 |

### Regression Detection

After running evals, the system compares results to the baseline:

- **Regression**: Case was passing in baseline, now failing
- **Fix**: Case was failing in baseline, now passing

Workflow:

1. `boba run` - Execute evals, compare to baseline, report regressions
2. `boba baseline` - Save current run as new baseline
3. Commit baseline to git for tracking

### Docker Integration

When configured for Docker (`boba init --docker`), commands auto-exec into the container:

```yaml
# boba-evals/.boba.yaml
runtime: docker-compose
service: api
```

Set `BOBA_NO_DOCKER=1` to bypass.

## Quickstart Example

Copy-paste example with all files shown together.

**1. Dataset** — `boba-evals/datasets/my-first-eval.json`:

```json
{
  "name": "my-first-eval",
  "cases": [
    {
      "name": "Basic greeting",
      "inputs": [{"role": "user", "message": "Hello"}],
      "expected_outcome": "Friendly greeting response"
    }
  ]
}
```

**2. Eval script** — `boba-evals/test.py`:

```python
from simboba import Boba, MessageInput

boba = Boba()

def agent(inputs: list[MessageInput]) -> str:
    # inputs is the full conversation history
    # For simple cases, just get the last message:
    # message = inputs[-1].message
    return "Hi there! How can I help?"

if __name__ == "__main__":
    result = boba.run(agent, dataset="my-first-eval")
    print(f"{result['passed']}/{result['total']} passed")
```

**3. Run**:

```bash
ANTHROPIC_API_KEY=sk-ant-... boba run
```

**4. Output**:

```
  + Basic greeting
Results: 1/1 passed (100.0%)
```

Note: `id`, `created_at`, `updated_at`, `case_count` are auto-generated if omitted.

## Development

### Code Style

- **Imports**: All imports must be at the top of the file, not inline within functions
- **Type hints**: Use type hints for function signatures
- **Docstrings**: Use docstrings for public functions and classes

### Running Tests

```bash
pytest tests/ -v
```

### Test Coverage

- `TestBoba`: `eval()`, `run()` methods
- `TestDatasetManagement`: Dataset CRUD
- `TestCaseManagement`: Case CRUD
- `TestRunsAPI`: List/delete runs
- `TestBaselines`: Baseline endpoints
- `TestSettings`: Settings endpoints
- `TestJudge`: Simple keyword judge
- `TestUIServing`: Health, index endpoints

### Key Files for Common Changes

| Task              | Files                                 |
| ----------------- | ------------------------------------- |
| Change Boba API   | `boba.py`, `__init__.py`              |
| Add CLI command   | `cli.py`                              |
| Add API endpoint  | `server.py`                           |
| Change data model | `storage.py`, `schemas.py`            |
| Update templates  | `samples/setup.py`, `samples/test.py` |
| Change judging    | `judge.py`, `prompts/judge.py`        |
| Update UI         | `frontend/src/` (React + TypeScript)  |

### Adding New Features

1. Update schemas in `schemas.py` if needed
2. Add storage operations in `storage.py`
3. Add API endpoint in `server.py`
4. Update `boba.py` if it affects the Python API
5. Update CLI in `cli.py` if needed
6. Update UI in `frontend/src/` and rebuild with `npm run build`
7. Add tests in `test_core_flows.py`
8. Update README.md and CLAUDE.md

### Frontend Development

The web UI is built with React + TypeScript + Vite + Tailwind CSS.

```bash
cd frontend

# Development (hot reload, proxies /api to localhost:8787)
npm run dev

# Production build (outputs to simboba/static/)
npm run build
```

Key frontend files:

- `src/lib/api.ts` - Type-safe API client
- `src/hooks/useStore.tsx` - Global state management (Context + useReducer)
- `src/pages/` - Page components (Dashboard, Datasets, Runs, Settings)
- `src/components/ui/` - Reusable UI components (shadcn-style)

## Design System

### Colors (Tailwind Zinc + Taro accent)

| Token              | Usage           |
| ------------------ | --------------- |
| `--zinc-50`        | Page background |
| `--zinc-200`       | Borders         |
| `--zinc-900`       | Primary text    |
| `--taro` (#8B7BA5) | Primary accent  |
| `--green-500`      | Pass states     |
| `--red-500`        | Fail states     |

### Principles

- Sharp corners (max 4px border-radius)
- Minimal - zinc grays, single taro accent
- Monospace for data (scores, counts, timestamps)

## Judge Configuration

The LLM judge uses Claude by default. Set `ANTHROPIC_API_KEY` environment variable.

```python
# Judge function signature
def judge(inputs, expected, actual) -> tuple[bool, str]:
    """Returns (passed, reasoning)"""
```

A simple keyword-matching fallback (`create_simple_judge()`) is used when no API key is available.
