# PromptGrad

[中文文档](README_zh.md)

Lightweight Prompt auto-optimization engine for Agent Pipelines.

PromptGrad automates the prompt engineering loop by applying a training-loop metaphor: **Executor** (forward pass) → **Evaluator** (loss computation) → **Optimizer** (parameter update), iterating until your prompt reaches a target quality score.

## How It Works

```
┌──────────────┐      ┌──────────────┐      ┌──────────────┐
│   Executor   │─────▶│  Evaluator   │─────▶│  Optimizer   │
│  (Forward)   │      │   (Loss)     │      │  (Update)    │
└──────────────┘      └──────────────┘      └──────────────┘
       ▲                                           │
       └──────────── updated prompt ───────────────┘
```

1. **Executor** — Runs your prompt + test cases against the target LLM (concurrent via litellm)
2. **Evaluator** — Scores outputs using Rule-based checks, Reference-based comparison, and LLM-as-Judge
3. **Optimizer** — Analyzes failure patterns, generates an improved prompt (Targeted strategy), and runs regression protection

The loop stops when: target score is reached, max iterations hit, or no improvement for N rounds.

## Delivery

PromptGrad runs as a **Claude Code Skill**. The host model (Claude) acts as the controller — performing LLM-as-Judge evaluation, failure pattern clustering, and prompt optimization inline — while the Python engine handles deterministic computation and external LLM calls.

## Quick Start

```bash
# Install
python3 -m venv .venv
source .venv/bin/activate
pip install -e ".[dev]"

# Run tests
pytest

# CLI (invoked by the Skill via Bash)
python -m promptgrad execute --config .promptgrad/config.yaml
python -m promptgrad evaluate --results .promptgrad/tmp/execution_results.json --config .promptgrad/config.yaml
python -m promptgrad regression-check --config .promptgrad/config.yaml --new-prompt-version 2 --prev-pass-rate 0.8
```

## Project Structure

```
promptgrad/
├── models.py          # Data models (TestCase, ExecutionResult, CaseScore, EvalResult)
├── config.py          # Config loading/saving (.promptgrad/config.yaml)
├── llm.py             # litellm wrapper
├── loader.py          # Dataset import (JSONL/Excel → YAML)
├── executor.py        # Prompt rendering + concurrent LLM execution
├── evaluator.py       # Rule-based & Reference-based evaluation
├── optimizer.py       # Regression protection checker
├── orchestrator.py    # Loop orchestration + exit conditions
├── history.py         # Optimization history tracking
└── __main__.py        # CLI entry point
SKILL.md               # Claude Code Skill definition
```

## Runtime Output

All artifacts are stored in `.promptgrad/` within the user's project:

```
.promptgrad/
├── config.yaml        # Project config
├── test_cases/        # Test cases (YAML)
├── history.yaml       # Optimization history
└── prompts/           # Prompt snapshots (v1.yaml, v2.yaml, ...)
```

## Evaluation Methods

| Method | How | Runs In |
|--------|-----|---------|
| Rule-based | max_length, min_length, regex_match, no_prefix | Python |
| Reference-based | exact_match, fuzzy (difflib), semantic | Python / Host model |
| LLM-as-Judge | Multi-dimension rubric scoring + critique | Host model |

## Tech Stack

- Python 3.10+
- [litellm](https://github.com/BerriAI/litellm) — multi-model LLM calls
- PyYAML — config/data handling
- openpyxl — Excel import

## License

MIT
