---
name: bandito
description: Analyze and debug LLM application traces. Use when investigating quality issues, debugging failures, analyzing cost/latency tradeoffs, or when the user asks about LLM app performance. Requires `bandito` CLI installed.
allowed-tools: Bash(bandito *)
---

# Bandito Skill

Bandito is an observability platform for LLM applications. It instruments traces, runs LLM-as-judge evaluations, and provides analysis surfaces for cost/quality/latency optimization.

## When to Use This Skill

Invoke this skill when:
- User mentions "debug", "quality", "failures", "errors", or "issues" with their LLM app
- User asks about LLM cost, latency, or performance analysis
- User wants to understand what's happening in their LLM application
- You encounter trace data and need to analyze it

## Available Commands

Run these via `bandito <command>`:

| Command | Purpose |
|---------|---------|
| `bandito observe projects` | List all projects with trace data |
| `bandito observe traces --project NAME` | List recent traces for a project |
| `bandito observe trace <id> --project NAME` | Inspect a single trace in detail |
| `bandito diagnostics --project NAME` | System health check (no LLM needed) |
| `bandito analyze tradeoffs --project NAME` | Cost/quality/latency optimization |
| `bandito analyze drift --project NAME` | Detect changes over time |
| `bandito analyze coverage --project NAME` | Find eval coverage gaps |
| `bandito judge overview --project NAME` | Check judge calibration state |
| `bandito judge run --project NAME --dataset NAME` | Run judge on a dataset |
| `bandito dataset create <name> --project NAME [--filters]` | Create dataset from traces |
| `bandito dataset list --project NAME` | List datasets for a project |
| `bandito dataset inspect <name> --project NAME` | Inspect dataset contents |
| `bandito dataset compare <name> --project NAME` | Compare dataset vs production |
| `bandito dataset regression <name> --project NAME` | Run diagnostics on dataset |
| `bandito dataset export <name> --project NAME --format FMT --output FILE` | Export dataset |
| `bandito tui --project NAME` | Interactive grading workbench |

## Common Patterns

### Debug a Failure

```bash
# 1. See recent traces with error status
bandito observe traces --project <name> --last 20

# 2. Inspect the failing trace
bandito observe trace <trace-id> --project <name>

# 3. Run diagnostics to find patterns
bandito diagnostics --project <name> --days 7
```

### Analyze Quality Issues

```bash
# 1. Check current quality baseline
bandito diagnostics --project <name>

# 2. Find where quality is low
bandito analyze coverage --project <name>

# 3. Understand cost/quality tradeoffs
bandito analyze tradeoffs --project <name>
```

### Set Up Automated Grading

```bash
# 1. Calibrate the judge (creates rubric + calibrates)
bandito judge calibrate --project <name>

# 2. Run judge on new traces
bandito judge run --project <name>

# 3. Check calibration state
bandito judge overview --project <name>
```

## Trace Structure

Traces contain:
- `input`: The user prompt
- `output`: The LLM response  
- `spans`: Nested operations (LLM calls, tool usage, retrievals)
- `status`: success | error | incomplete
- `metadata`: user_id, session_id, tags
- `metrics`: tokens, cost, latency

## Evaluation Data

Evaluations attach to traces:
- `human_score`: 0 (fail) or 1 (pass) from manual grading
- `human_comment`: Notes from grader
- `judge_score`: 0 or 1 from LLM-as-judge
- `judge_critique`: Judge's reasoning

## Datasets

Datasets are curated collections of traces for regression testing, comparison, and offline evaluation.

```bash
# Create dataset from traces matching criteria
bandito dataset create golden-set --project my-bot --human-score-eq 1
bandito dataset create problem-cases --project my-bot --judge-score-lt 0.5

# List and inspect
bandito dataset list --project my-bot
bandito dataset inspect golden-set --project my-bot

# Compare dataset vs production
bandito dataset compare golden-set --project my-bot

# Run diagnostics on dataset
bandito dataset regression golden-set --project my-bot

# Export for CI/testing
bandito dataset export golden-set --project my-bot --format pytest --output test.py

# Run judge on dataset (pre-deploy validation)
bandito judge run --project my-bot --dataset golden-set
```

### TUI: Add Trace to Dataset
In the grading TUI (`bandito tui --project NAME`), press `a` to add the current trace to a dataset.

### Dataset Filters (for `create`):
- `--human-score-eq 0|1` - Filter by human score
- `--judge-score-eq 0|1` - Filter by judge score
- `--judge-score-lt N` - Filter by judge score less than N
- `--limit N` - Max traces to include

## Environment

- **CLI**: `bandito` (Python SDK)
- **Config**: `~/.bandito/config.toml`
  - `[agent]` section enables agent-aware features:
    - `enabled = true` (default) — AI features available
    - `mcp_enabled = true` — Exposes Bandito as MCP server for agent tool access
- **Storage**: Local (default), S3, PostgreSQL, or Langfuse
- **Backend**: Requires `BANDITO_API_KEY` for judge/analyze commands

## First Steps

When working with a user's LLM application:

1. Ask for the project name or check if they have traces in `~/.bandito/traces/`
2. Run `bandito observe projects` to see available projects
3. Use `bandito diagnostics` for a quick health check (no API key needed)
4. Escalate to `bandito analyze *` commands for deeper analysis (requires API key)

If the user doesn't have Bandito set up yet, guide them to install:
```bash
pip install bandito
bandito signup  # Free account
```
