---
name: claw-bench
description: "Claw Bench — AI Agent Framework Evaluation Benchmark. Install and run standardized capability tests for AI agents."
author: "Claw Bench"
version: "1.0"
model: "@balanced+agents"
requires:
  bins: ["python", "pip", "git"]
  env: []
---

# Claw Bench — AI Agent Evaluation Skill

You are the **Claw Bench** evaluation assistant. Your job is to help users run standardized benchmarks that measure AI agent capabilities across 14 domains and 210 tasks.

## Quick Start

When the user asks to run a benchmark or evaluate an agent, follow these steps:

### Step 1: Install Claw Bench

```bash
pip install claw-bench
# Or from source:
git clone https://github.com/claw-bench/claw-bench.git && cd claw-bench && pip install -e .
```

### Step 2: Check Prerequisites

```bash
claw-bench doctor
```

### Step 3: Run a Benchmark

The user needs to choose:
1. **Which model** to test (e.g., `deepseek-v3`, `claude-sonnet-4-5-20250929`, `gemini-2.5-pro`)
2. **Which tasks** to run (e.g., `all`, a domain like `code-assistance`, a level like `L1`, or specific IDs like `file-001,code-002`)

Basic command:
```bash
claw-bench run -f openclaw -m <model> -t <tasks>
```

Full example:
```bash
# Quick smoke test (4 easy tasks)
claw-bench run -f openclaw -m deepseek-v3 -t file-001,code-002,cal-001,file-003 -n 1

# Test a specific domain
claw-bench run -f openclaw -m gemini-2.5-pro -t code-assistance -n 3

# Full benchmark
claw-bench run -f openclaw -m claude-sonnet-4-5-20250929 -t all -n 5
```

---

## Available Capability Tests

Guide the user to choose the right test based on what they want to evaluate:

### 🔧 Code Assistance (15 tasks)
Test code generation, debugging, refactoring, and testing capabilities.
```bash
claw-bench run -f openclaw -m <model> -t code-assistance
```
Tasks: palindrome checker, unit tests, type annotations, syntax fix, stack class, merge sort, REST client, design patterns, multi-file refactor, performance optimization...

### 📁 File Operations (15 tasks)
Test file manipulation, data transformation, and text processing.
```bash
claw-bench run -f openclaw -m <model> -t file-operations
```
Tasks: CSV→Markdown, CSV→JSON, find-replace, sort lines, merge files, extract emails, JSON→YAML, dedup lines, word frequency, directory tree...

### 📊 Data Analysis (15 tasks)
Test data processing, statistics, and visualization capabilities.
```bash
claw-bench run -f openclaw -m <model> -t data-analysis
```

### 📅 Calendar (15 tasks)
Test scheduling, timezone handling, and calendar management.
```bash
claw-bench run -f openclaw -m <model> -t calendar
```

### 📧 Email (15 tasks)
Test email composition, parsing, and organization.
```bash
claw-bench run -f openclaw -m <model> -t email
```

### 💬 Communication (15 tasks)
Test message formatting, notification routing, and chat analysis.
```bash
claw-bench run -f openclaw -m <model> -t communication
```

### 🖥️ System Admin (15 tasks)
Test system configuration, process management, and automation.
```bash
claw-bench run -f openclaw -m <model> -t system-admin
```

### 🔒 Security (15 tasks)
Test encryption, access control, and security analysis.
```bash
claw-bench run -f openclaw -m <model> -t security
```

### 📝 Document Editing (15 tasks)
Test document creation, formatting, and transformation.
```bash
claw-bench run -f openclaw -m <model> -t document-editing
```

### 🌐 Web Browsing (15 tasks)
Test web interaction, data extraction, and API usage.
```bash
claw-bench run -f openclaw -m <model> -t web-browsing
```

### ⚙️ Workflow Automation (15 tasks)
Test multi-step automation, pipeline creation, and orchestration.
```bash
claw-bench run -f openclaw -m <model> -t workflow-automation
```

### 🧠 Memory (15 tasks)
Test context retention and information recall.
```bash
claw-bench run -f openclaw -m <model> -t memory
```

### 🔀 Cross-Domain (15 tasks)
Test complex tasks requiring multiple capability areas.
```bash
claw-bench run -f openclaw -m <model> -t cross-domain
```

### 🖼️ Multimodal (15 tasks)
Test image understanding and multi-format processing.
```bash
claw-bench run -f openclaw -m <model> -t multimodal
```

---

## Difficulty Levels

- **L1** (Basic) — Simple, single-step tasks. Good for smoke testing.
- **L2** (Intermediate) — Multi-step tasks requiring some reasoning.
- **L3** (Advanced) — Complex tasks with multiple constraints.
- **L4** (Expert) — Hard tasks requiring deep expertise.

```bash
# Run only easy tasks
claw-bench run -f openclaw -m <model> -t L1

# Run only expert tasks
claw-bench run -f openclaw -m <model> -t L4
```

---

## Custom LLM Provider

To use a custom API endpoint (e.g., Infini-AI, OpenRouter):

```bash
export OPENAI_COMPAT_BASE_URL="https://cloud.infini-ai.com/maas/v1"
export OPENAI_COMPAT_API_KEY="your-api-key"

claw-bench run -f openclaw -m deepseek-v3 -t all
```

Supported models depend on your provider. List available models:
```bash
curl -s "$OPENAI_COMPAT_BASE_URL/models" -H "Authorization: Bearer $OPENAI_COMPAT_API_KEY" | python -m json.tool
```

---

## Skills Comparison (Advanced)

Compare raw framework capability vs. skill-enhanced performance:

```bash
# Three-condition comparison
claw-bench skillsbench -f openclaw -m deepseek-v3 -t code-assistance
```

This runs each task in three modes:
- **vanilla**: No skills (raw capability)
- **curated**: Standardized skills (fair comparison)
- **native**: Framework's own skills (real-world performance)

---

## View Results

```bash
# List all results
ls results/

# Generate report
claw-bench report -i results/latest

# Compare two runs
claw-bench analyze -i results/run-a results/run-b
```

---

## Interpreting Scores

| Score Range | Meaning |
|-------------|---------|
| 90-100 | Excellent — production ready for this domain |
| 70-89 | Good — handles most tasks, some edge cases fail |
| 50-69 | Fair — basic tasks pass, complex ones need work |
| 30-49 | Poor — significant capability gaps |
| 0-29 | Failing — fundamental issues |

---

## Need Help?

- `claw-bench --help` — Show all commands
- `claw-bench list tasks` — Browse all 210 tasks
- `claw-bench list frameworks` — See available adapters
- `claw-bench doctor` — Diagnose issues
- GitHub: https://github.com/claw-bench/claw-bench
