# LM Evaluation Harness - Deep Dive

> **TL;DR**: EleutherAI's lm-evaluation-harness (10,600+ stars) is THE standard framework for evaluating language models. It powers Hugging Face's Open LLM Leaderboard and includes 60+ academic benchmarks with 200+ tasks. Supports HuggingFace, vLLM, OpenAI, and more. If you've seen an LLM benchmark score, it was probably computed with this tool.

---

## What Is LM Evaluation Harness?

**lm-evaluation-harness** is a unified framework for evaluating language models across academic benchmarks. It eliminates the need to re-implement benchmarks individually, ensuring reproducible and comparable results.

### The Problem It Solves

```
┌─────────────────────────────────────────────────────────────────────┐
│                    THE EVALUATION PROBLEM                            │
├─────────────────────────────────────────────────────────────────────┤
│                                                                     │
│   WITHOUT LM-EVAL:                                                  │
│   ├── Each benchmark has its own codebase                          │
│   ├── Subtle implementation differences affect scores              │
│   ├── Hard to reproduce results from papers                        │
│   ├── Need to install dozens of libraries                          │
│   └── No standardized prompts or metrics                           │
│                                                                     │
│   WITH LM-EVAL:                                                     │
│   ├── One framework for all benchmarks                             │
│   ├── Standardized implementations                                  │
│   ├── Reproducible results (matches papers)                        │
│   ├── Single installation                                          │
│   └── Publicly available prompts                                   │
│                                                                     │
└─────────────────────────────────────────────────────────────────────┘
```

---

## Quick Stats

| Metric | Value |
|--------|-------|
| **GitHub Stars** | 10,600+ |
| **Forks** | 2,800+ |
| **Benchmarks** | 60+ academic benchmarks |
| **Tasks** | 200+ (with subtasks/variants) |
| **Current Version** | v0.4.9.1 |
| **Created By** | EleutherAI |

---

## Key Impact

**This is the backbone of Hugging Face's Open LLM Leaderboard** - the most widely referenced LLM ranking system.

When you see LLM benchmark scores like:
- "GPT-4 scores 86% on MMLU"
- "Model X beats Model Y on HellaSwag"
- "This model achieves SOTA on ARC"

Those numbers were likely computed with lm-evaluation-harness.

---

## Supported Benchmarks

### Major Benchmarks

| Benchmark | What It Tests |
|-----------|---------------|
| **MMLU** | Multitask accuracy (57 subjects) |
| **HellaSwag** | Commonsense inference |
| **TruthfulQA** | Tendency to reproduce falsehoods |
| **ARC** | Grade-school science reasoning |
| **GSM8K** | Mathematical reasoning |
| **HumanEval** | Code generation |
| **LAMBADA** | Language modeling |
| **IFEval** | Instruction following |
| **GPQA** | Graduate-level questions |
| **SQuAD** | Reading comprehension |

### Benchmark Discovery

```bash
# List all available tasks
lm_eval --tasks list
```

---

## Supported Model Types

| Backend | Support | Features |
|---------|---------|----------|
| **HuggingFace** | Local | Multi-GPU, quantized, GGUF |
| **vLLM** | Local | Fast inference, batching |
| **SGLang** | Local | Tensor parallelism |
| **NVIDIA NeMo** | Local | Pipeline parallelism |
| **OpenAI API** | Commercial | GPT models |
| **Anthropic API** | Commercial | Claude models |
| **Ollama** | Local | Any open model |

---

## Installation & Usage

### Installation

```bash
# Clone and install
git clone --depth 1 https://github.com/EleutherAI/lm-evaluation-harness
cd lm-evaluation-harness
pip install -e .

# Or with specific backends
pip install "lm_eval[hf,vllm,api]"
```

### Basic Usage

```bash
# Evaluate a HuggingFace model
lm_eval --model hf \
  --model_args pretrained=EleutherAI/gpt-j-6B \
  --tasks hellaswag \
  --device cuda:0 \
  --batch_size 8
```

### Reproduce Open LLM Leaderboard

```bash
lm_eval --model_args="pretrained=<model>,revision=<rev>,dtype=<dtype>" \
  --tasks=leaderboard \
  --batch_size=auto \
  --output_path=results
```

---

## How It Works

```
┌─────────────────────────────────────────────────────────────────────┐
│                    EVALUATION PIPELINE                               │
├─────────────────────────────────────────────────────────────────────┤
│                                                                     │
│   1. LOAD MODEL                                                     │
│      └── HuggingFace, vLLM, API, etc.                              │
│                              ↓                                      │
│   2. LOAD TASK                                                      │
│      └── YAML config defining prompts, metrics, data               │
│                              ↓                                      │
│   3. GENERATE PROMPTS                                               │
│      └── Jinja2 templates → actual prompts                         │
│                              ↓                                      │
│   4. RUN INFERENCE                                                  │
│      └── Batched inference with configurable size                  │
│                              ↓                                      │
│   5. COMPUTE METRICS                                                │
│      └── Accuracy, F1, ROUGE, perplexity, etc.                    │
│                              ↓                                      │
│   6. AGGREGATE RESULTS                                              │
│      └── JSON output with reproducibility info                     │
│                                                                     │
└─────────────────────────────────────────────────────────────────────┘
```

---

## Advanced Features

### Multi-GPU Evaluation

```bash
# With vLLM tensor parallelism
lm_eval --model vllm \
  --model_args pretrained=<model>,tensor_parallel_size=4 \
  --tasks lambada_openai \
  --batch_size auto
```

### Chat Templates

```bash
lm_eval --model hf \
  --model_args pretrained=<model> \
  --tasks <task> \
  --apply_chat_template \
  --system_instruction "You are a helpful assistant"
```

### Experiment Tracking

```bash
# Weights & Biases logging
lm_eval --model hf \
  --model_args pretrained=<model> \
  --tasks mmlu \
  --wandb_args project=my-project
```

---

## Adding Custom Tasks

Tasks are defined in YAML:

```yaml
# Example task config
task: my_custom_task
dataset_path: my_dataset
doc_to_text: "{{passage}}\nQuestion: {{question}}?\nAnswer:"
doc_to_target: "{{answer}}"
metric_list:
  - metric: exact_match
```

### Process

1. Fork the repository
2. Create YAML config in `lm_eval/tasks/`
3. Test with provided utilities
4. Submit pull request

---

## Who Is EleutherAI?

**EleutherAI** is a non-profit AI research institute founded in 2020.

| Fact | Detail |
|------|--------|
| **Founded** | July 2020 |
| **Mission** | Democratize AI research |
| **Key Projects** | GPT-J, GPT-NeoX, The Pile, Pythia |
| **Downloads** | 70M+ model downloads |
| **Publications** | 130+ in top venues |
| **Philosophy** | AI decisions shouldn't be restricted to profit-driven companies |

---

## Comparison with Alternatives

| Tool | Focus | Best For |
|------|-------|----------|
| **lm-evaluation-harness** | Academic benchmarks | Research, leaderboards |
| **HELM** | Holistic evaluation | Robustness, fairness |
| **Promptfoo** | App testing | Development, CI/CD |
| **DeepEval** | Output validation | Safety, local testing |
| **Chatbot Arena** | Human preference | Model comparison |

---

## Recent Developments

### v0.4.x Features

| Feature | Description |
|---------|-------------|
| **Chat Templates** | Apply model's chat format |
| **CoT Support** | Chain-of-thought evaluation |
| **Multimodal** | Text+image (prototype) |
| **SGLang** | Fast batch processing |
| **Ascend NPU** | Additional hardware support |

---

## How This Connects to Our Other Deep Dives

| Deep Dive | Connection |
|-----------|------------|
| **SWE-bench** | lm-eval can run SWE-bench-like tasks |
| **Promptfoo** | Different focus (benchmarks vs app testing) |
| **Awesome LLMOps** | Listed as evaluation tool |
| **OpenHands** | Uses lm-eval for agent evaluation |

---

## Key Links

| Resource | URL |
|----------|-----|
| GitHub | https://github.com/EleutherAI/lm-evaluation-harness |
| Documentation | In repository |
| EleutherAI | https://www.eleuther.ai/ |
| Open LLM Leaderboard | https://huggingface.co/spaces/open-llm-leaderboard |

---

## Bottom Line

lm-evaluation-harness is the **industry standard for LLM benchmarking**.

**Why it matters**:
- Powers the Open LLM Leaderboard
- 60+ benchmarks, 200+ tasks
- Ensures reproducible comparisons
- Used by everyone (NVIDIA, Cohere, BigScience, etc.)

**The key insight**: Without standardized evaluation, every model comparison is suspect. lm-evaluation-harness provides the common ground that makes LLM research reproducible and comparable.

For our repository, this is the foundation that all benchmark numbers are built on. When we see "X% on SWE-bench" or "Y% on MMLU", this framework is likely computing those numbers.

---

*Document created: January 2026*
*Sources: [GitHub - EleutherAI/lm-evaluation-harness](https://github.com/EleutherAI/lm-evaluation-harness), [EleutherAI](https://www.eleuther.ai/), [Open LLM Leaderboard](https://huggingface.co/spaces/open-llm-leaderboard)*
