# simboba

Lightweight eval tracking with LLM-as-judge. Users write Python scripts to run evals, results are stored as JSON files (git-trackable) and viewable in a web UI.

## Quick Start

```bash
pip install -e .
boba init        # Create boba-evals/ folder
boba magic       # Print AI prompt to configure evals
boba run         # Run evals (handles Docker automatically)
boba baseline    # Save run as baseline for regression detection
boba serve       # View results
```

## Project Structure

```
simboba/
├── simboba/
│   ├── __init__.py       # Exports Boba class
│   ├── boba.py           # Core Boba class (eval, run methods)
│   ├── cli.py            # Click CLI (init, magic, generate, run, baseline, serve, datasets, reset)
│   ├── config.py         # Configuration (.boba.yaml) and Docker exec
│   ├── storage.py        # JSON file storage operations
│   ├── schemas.py        # Pydantic models for validation
│   ├── server.py         # FastAPI REST API
│   ├── judge.py          # LLM judge implementation
│   ├── prompts/          # LLM prompts for generation and judging
│   ├── utils/            # LLM client utilities
│   ├── samples/          # Template files copied by `boba init`
│   │   ├── setup.py      # Test fixtures template
│   │   └── test_chat.py  # Eval script template
│   └── static/           # Built React frontend (generated by frontend build)
├── frontend/             # React + TypeScript + Vite frontend source
│   ├── src/
│   │   ├── components/   # UI components (Layout, shadcn-style components)
│   │   ├── pages/        # Page components (Dashboard, Datasets, Runs, Settings)
│   │   ├── hooks/        # React hooks (useStore for state management)
│   │   ├── lib/          # Utilities and API client
│   │   └── types/        # TypeScript type definitions
│   ├── package.json
│   └── vite.config.ts
├── tests/
│   ├── conftest.py       # Pytest fixtures
│   └── test_core_flows.py
└── pyproject.toml
```

## Data Storage

All data is stored as JSON files in the `boba-evals/` directory:

```
boba-evals/
├── datasets/
│   └── {name}.json           # Dataset + all cases (git tracked)
├── baselines/
│   └── {name}.json           # Committed run results (git tracked)
├── runs/
│   └── {dataset}/
│       └── {timestamp}.json  # All runs (gitignored)
├── files/                    # Uploaded attachments (git tracked)
├── settings.json             # App settings
└── .gitignore                # Ignores runs/
```

## Architecture

### Core Class: Boba

```python
from simboba import Boba

boba = Boba()

# Single eval - judges output against expected
result = boba.eval(
    input="Hello",
    output="Hi there!",
    expected="Should greet the user",
)
# Returns: {"passed": bool, "reasoning": str, "run_id": str}

# Dataset eval - runs agent against all cases in a dataset
result = boba.run(
    agent=my_agent_fn,  # Callable[[str], str]
    dataset="my-dataset",
)
# Returns: {"passed": int, "failed": int, "total": int, "score": float, "run_id": str, "regressions": list, "fixes": list}

# With deterministic metadata checking
def check_citations(expected, actual):
    return expected.get("citations") == actual.get("citations")

result = boba.eval(
    input="What's in section 3?",
    output="Section 3 covers...",
    expected="Should cite the document",
    expected_metadata={"citations": [{"file": "doc.pdf", "page": 3}]},
    actual_metadata={"citations": [{"file": "doc.pdf", "page": 3}]},
    metadata_checker=check_citations,  # Deterministic check instead of LLM
)
```

### Metadata Checking

Three modes for evaluating metadata (citations, tool_calls, etc.):

| Mode | How | Behavior |
|------|-----|----------|
| 1. No metadata | Don't pass `expected_metadata`/`actual_metadata` | LLM judges output only |
| 2. LLM evaluation | Pass metadata, no `metadata_checker` | LLM judges output + metadata together |
| 3. LLM + deterministic | Pass metadata AND `metadata_checker` | LLM judges + your function checks metadata |

```python
# metadata_checker signature
def checker(expected_metadata: dict | None, actual_metadata: dict | None) -> bool:
    """Return True if metadata matches expectations."""
```

When `metadata_checker` is provided:
- LLM still sees metadata for context/reasoning
- Your function runs as an additional gate
- Case passes only if **both** checks pass
- Results include `expected_metadata`, `actual_metadata`, and `metadata_passed` fields

### Data Model

- **Dataset**: Named collection of eval cases (JSON file)
- **Case**: Test case with inputs, expected outcome, optional source reference
- **Run**: One execution with results (JSON file per run)
- **Baseline**: Committed run results for regression detection

### Case Structure

```python
{
    "id": "abc123",  # Stable ID for baseline comparison
    "name": "Test greeting",
    "inputs": [
        {
            "role": "user",
            "message": "...",
            "attachments": [{"file": "doc.pdf"}],
            "metadata": {...}  # optional - for tool_calls, citations, etc.
        }
    ],
    "expected_outcome": "Agent should...",
    "expected_metadata": {      # optional - expected citations, tool_calls, etc.
        "citations": [{"file": "doc.pdf", "page": 12}],
        "tool_calls": ["get_orders"]
    }
}
```

### CLI Commands

| Command | Description |
|---------|-------------|
| `boba init` | Create `boba-evals/` folder with datasets/, baselines/, runs/, files/ |
| `boba init --docker` | Quick setup for Docker Compose |
| `boba init --local` | Quick setup for local Python |
| `boba magic` | Print detailed AI prompt to configure eval scripts |
| `boba run [script]` | Run eval script (default: `test_chat.py`). Handles Docker automatically |
| `boba baseline` | Interactive - list recent runs, select one to save as baseline |
| `boba serve` | Start web UI at localhost:8787 |
| `boba datasets` | List all datasets |
| `boba generate "desc"` | Generate dataset from description |
| `boba reset` | Clear run history (keeps datasets and baselines) |

### API Endpoints

| Endpoint | Method | Description |
|----------|--------|-------------|
| `/api/datasets` | GET, POST | List/create datasets |
| `/api/datasets/{name}` | GET, PUT, DELETE | CRUD for dataset |
| `/api/datasets/{name}/export` | GET | Export dataset as JSON |
| `/api/datasets/import` | POST | Import dataset from JSON |
| `/api/cases` | GET, POST | List/create cases |
| `/api/cases/{dataset}/{id}` | GET, PUT, DELETE | CRUD for single case |
| `/api/cases/bulk` | POST | Bulk create cases |
| `/api/runs` | GET | List runs |
| `/api/runs/{dataset}/{filename}` | GET, DELETE | Get/delete run |
| `/api/baselines` | GET | List baselines |
| `/api/baselines/{dataset}` | GET | Get baseline for dataset |
| `/api/settings` | GET, PUT | Get/update settings |
| `/api/files/upload` | POST | Upload file |
| `/api/files/{filename}` | GET | Get file |

### Regression Detection

After running evals, the system compares results to the baseline:

- **Regression**: Case was passing in baseline, now failing
- **Fix**: Case was failing in baseline, now passing

Workflow:
1. `boba run` - Execute evals, compare to baseline, report regressions
2. `boba baseline` - Save current run as new baseline
3. Commit baseline to git for tracking

### Docker Integration

When configured for Docker (`boba init --docker`), commands auto-exec into the container:

```yaml
# boba-evals/.boba.yaml
runtime: docker-compose
service: api
```

Set `BOBA_NO_DOCKER=1` to bypass.

## Development

### Code Style

- **Imports**: All imports must be at the top of the file, not inline within functions
- **Type hints**: Use type hints for function signatures
- **Docstrings**: Use docstrings for public functions and classes

### Running Tests

```bash
pytest tests/ -v
```

### Test Coverage

- `TestBoba`: `eval()`, `run()` methods
- `TestDatasetManagement`: Dataset CRUD
- `TestCaseManagement`: Case CRUD
- `TestRunsAPI`: List/delete runs
- `TestBaselines`: Baseline endpoints
- `TestSettings`: Settings endpoints
- `TestJudge`: Simple keyword judge
- `TestUIServing`: Health, index endpoints

### Key Files for Common Changes

| Task | Files |
|------|-------|
| Change Boba API | `boba.py`, `__init__.py` |
| Add CLI command | `cli.py` |
| Add API endpoint | `server.py` |
| Change data model | `storage.py`, `schemas.py` |
| Update templates | `samples/setup.py`, `samples/test_chat.py` |
| Change judging | `judge.py`, `prompts/judge.py` |
| Update UI | `frontend/src/` (React + TypeScript) |

### Adding New Features

1. Update schemas in `schemas.py` if needed
2. Add storage operations in `storage.py`
3. Add API endpoint in `server.py`
4. Update `boba.py` if it affects the Python API
5. Update CLI in `cli.py` if needed
6. Update UI in `frontend/src/` and rebuild with `npm run build`
7. Add tests in `test_core_flows.py`
8. Update README.md and CLAUDE.md

### Frontend Development

The web UI is built with React + TypeScript + Vite + Tailwind CSS.

```bash
cd frontend

# Development (hot reload, proxies /api to localhost:8787)
npm run dev

# Production build (outputs to simboba/static/)
npm run build
```

Key frontend files:
- `src/lib/api.ts` - Type-safe API client
- `src/hooks/useStore.tsx` - Global state management (Context + useReducer)
- `src/pages/` - Page components (Dashboard, Datasets, Runs, Settings)
- `src/components/ui/` - Reusable UI components (shadcn-style)

## Design System

### Colors (Tailwind Zinc + Taro accent)

| Token | Usage |
|-------|-------|
| `--zinc-50` | Page background |
| `--zinc-200` | Borders |
| `--zinc-900` | Primary text |
| `--taro` (#8B7BA5) | Primary accent |
| `--green-500` | Pass states |
| `--red-500` | Fail states |

### Principles

- Sharp corners (max 4px border-radius)
- Minimal - zinc grays, single taro accent
- Monospace for data (scores, counts, timestamps)

## Judge Configuration

The LLM judge uses Claude by default. Set `ANTHROPIC_API_KEY` environment variable.

```python
# Judge function signature
def judge(inputs, expected, actual) -> tuple[bool, str]:
    """Returns (passed, reasoning)"""
```

A simple keyword-matching fallback (`create_simple_judge()`) is used when no API key is available.
