# token-diet

Compress prompts to use fewer tokens without losing meaning.

`token-diet` is a CLI tool that takes any prompt — from a file, stdin, or a pipe
— and rewrites it to be as compact as possible while preserving the original
intent. It works offline using battle-tested rewrite rules, or via the
Claude API (claude-haiku-4-5-20251001) for intelligent AI-powered compression.

## Quick install

```bash
pip install token-diet
```

Or from source:

```bash
git clone https://github.com/your-handle/token-diet
cd token-diet
pip install -e .
```

## Usage

```bash
# From a file
token-diet prompt.txt

# From stdin
echo "Please summarize the following text for me..." | token-diet

# With Claude API key (intelligent compression)
token-diet prompt.txt --api-key sk-ant-...

# Or set the env var once
export ANTHROPIC_API_KEY=sk-ant-...
token-diet prompt.txt

# Compression levels
token-diet prompt.txt --level aggressive   # max compression, may lose nuance
token-diet prompt.txt --level balanced     # default, preserves intent
token-diet prompt.txt --level gentle       # minimal, safest changes only

# Show a diff between original and compressed
token-diet prompt.txt --diff

# Output ONLY the compressed prompt (great for piping)
token-diet prompt.txt --quiet | pbcopy
token-diet prompt.txt --quiet > compressed.txt
```

## Example output

```
╭──────────────────────────────────────────╮
│            TOKEN DIET REPORT             │
╰──────────────────────────────────────────╯

Mode: balanced compression (rule-based / offline)

────────────────────── BEFORE ──────────────────────
│ Please take the following document and create a   │
│ comprehensive summary that covers all of the main │
│ points and key details. I would like you to make  │
│ sure to include every important fact...           │

────────────────────── AFTER ───────────────────────
│ Take the following document and create a thorough │
│ summary covering all main points and key details. │
│ Include every important fact...                   │

────────────────────── Results ─────────────────────
  Original:   847 tokens  (~3,388 chars)
  Compressed: 234 tokens  (~936 chars)
  Saved:      613 tokens  (72.4% reduction)
  Est. savings: $0.0031/call  (at GPT-4o pricing $5/1M tokens)

────────────────────── Techniques applied ──────────
  • Removed filler phrases (8)
  • Shortened verbose constructions (12)
  • Collapsed redundant instructions (3)
```

## How it works

### Offline (no API key)

The rule engine applies linguistically safe substitutions in three tiers:

| Level      | What it removes |
|------------|-----------------|
| `gentle`   | Politeness openers ("Please", "Could you please", "I would like you to"), obvious verbose forms ("in order to", "due to the fact that") |
| `balanced` | Everything in `gentle`, plus hedging adverbs ("very", "quite", "basically"), discourse markers ("of course", "clearly", "in fact"), and more verbose constructions |
| `aggressive` | Everything above, plus restructuring question openings ("Can you help me…"), discourse section headers, and more |

### With API key

Sends your prompt to `claude-haiku-4-5-20251001` with a meta-prompt asking it
to rewrite for minimum tokens. Falls back gracefully to rule-based compression
on any API error.

### Token counting

Uses a simple `4 chars ≈ 1 token` heuristic by default.
Install `tiktoken` for accurate OpenAI tokeniser counts:

```bash
pip install "token-diet[tiktoken]"
```

## Options

| Flag | Default | Description |
|------|---------|-------------|
| `--api-key KEY` | `$ANTHROPIC_API_KEY` | Anthropic API key for Claude-powered compression |
| `--level` | `balanced` | Compression level: `gentle`, `balanced`, `aggressive` |
| `--diff` | off | Show unified diff between original and compressed |
| `--quiet` / `-q` | off | Output only the compressed text (no formatting) |
| `--version` | | Show version and exit |
| `--help` / `-h` | | Show help and exit |

## Cost estimate

The report shows an estimated cost saving per API call based on GPT-4o input
pricing ($5 / 1M tokens). This is illustrative — actual savings depend on
which model you use and your actual call volume.

## Development

```bash
# Clone and install in dev mode
git clone https://github.com/your-handle/token-diet
cd token-diet
python -m venv venv && source venv/bin/activate
pip install -e ".[dev]"

# Run tests
pytest -v

# Run tests with coverage
pytest --cov=src --cov-report=term-missing

# Lint and format
black src/ tests/
isort src/ tests/
flake8 src/ tests/
mypy src/
```

## License

MIT
