# 🔭 tokenviz

> **Real-time LLM context window visualizer** — know exactly how much of your context you're burning.

[![PyPI](https://img.shields.io/pypi/v/tokenviz?color=58a6ff&style=flat-square)](https://pypi.org/project/tokenviz)
[![Python 3.9+](https://img.shields.io/badge/python-3.9+-58a6ff?style=flat-square)](https://python.org)
[![Zero dependencies](https://img.shields.io/badge/deps-zero%20required-3fb950?style=flat-square)](#installation)
[![License: MIT](https://img.shields.io/badge/license-MIT-7d8590?style=flat-square)](LICENSE)

```
  tokenviz · gpt-4o

  47,832 tokens  / 128,000

  ████████████████████░░░░░░░░░░░░░░░░░░░░
  37.4% used  ·  80,168 remaining

  chars     214,559
  words      37,201
  lines       3,847
  tok/word     1.29
```

**Works with every major LLM.** GPT-4o, Claude, Gemini 1M, Llama, DeepSeek, Mistral — 20+ models.

**Zero required dependencies.** Built-in BPE estimator (±5%) ships alone. Drop in `tiktoken` for exact OpenAI counts.

---

## Why?

You're writing a long prompt. Is it 30k tokens or 90k? You're building a RAG pipeline. How much context is left for the answer? You're debugging why the model truncated — was it the context limit?

**tokenviz answers these questions instantly**, with a CLI you can pipe into any workflow and a web UI you can paste anything into.

---

## Install

```bash
pip install tokenviz
```

Zero required dependencies. For **exact** OpenAI counts:

```bash
pip install tokenviz[exact]   # adds tiktoken
```

Or run without installing:

```bash
npx tokenviz count file.txt   # via npx (Node.js)
```

---

## Usage

### CLI

```bash
# Analyze a file
tokenviz count prompt.txt

# Pipe from stdin
cat conversation.json | tokenviz count

# Specify model
tokenviz count system_prompt.md --model claude-sonnet-4

# Inline text
tokenviz count --text "$(cat *.py)" --model gpt-4o

# JSON output (for scripting)
tokenviz count prompt.txt --json

# List all models + limits
tokenviz models
```

### Web UI

```bash
tokenviz serve
# → http://localhost:7432
```

Paste anything — code, documents, conversations, system prompts. Token count and density heatmap update as you type.

### Python API

```python
from tokenviz import count_tokens, analyze_text

# Simple count
result = count_tokens("Your text here", model="gpt-4o")
print(result["count"])    # 7
print(result["method"])   # "tiktoken" or "estimate"

# Full analysis
data = analyze_text(open("big_prompt.txt").read(), model="claude-sonnet-4")
print(f"{data['tokens']:,} / {data['limit']:,} ({data['percent']:.1f}%)")
print(f"Remaining: {data['remaining']:,}")

if data["warning"]:
    print("⚠️  Over 80% full!")
```

---

## Supported Models

| Provider   | Models                                    | Context    |
|------------|-------------------------------------------|------------|
| OpenAI     | gpt-4o, gpt-4o-mini, o1, o3-mini, ...    | 8k–200k    |
| Anthropic  | claude-sonnet-4, claude-opus-4, ...       | 200k       |
| Google     | gemini-2.5-pro, gemini-2.0-flash, ...     | 1M         |
| Meta       | llama-3.1-70b, llama-3.1-405b            | 128k       |
| DeepSeek   | deepseek-v3, deepseek-r1                 | 128k       |
| Mistral    | mistral-large, mixtral-8x22b             | 65k–128k   |

Missing a model? [Open an issue](../../issues/new) or add it in one line — it's just a dict.

---

## Token counting accuracy

| Mode               | Accuracy | Requires  |
|--------------------|----------|-----------|
| `tiktoken` (exact) | 100%     | `pip install tokenviz[exact]` |
| Built-in BPE est.  | ±5%      | nothing   |

The built-in estimator handles English, code, JSON, and mixed content well. It uses the same pre-tokenization heuristics as BPE encoders, without the vocabulary lookup.

---

## Scripting examples

```bash
# Alert if prompt is > 80% of context
TOKEN_PCT=$(tokenviz count prompt.txt --json | python3 -c "import sys,json; print(json.load(sys.stdin)['percent'])")
if (( $(echo "$TOKEN_PCT > 80" | bc -l) )); then echo "WARNING: context over 80%"; fi

# Count tokens across all Python files in a project
find . -name "*.py" | xargs cat | tokenviz count --model gpt-4o

# Pre-commit hook: warn if any staged file is > 50k tokens
git diff --cached --name-only | xargs -I{} sh -c 'tokenviz count "{}" --json | python3 -c "import sys,json; d=json.load(sys.stdin); print(f\"{d[\"tokens\"]:,} tokens in {}\") if d[\"tokens\"] > 50000 else None"'
```

---

## Contributing

```bash
git clone https://github.com/yourusername/tokenviz
cd tokenviz
python tokenviz.py count --text "hello world"
```

The codebase is intentionally small (~500 lines total). Contributions welcome:

- 🆕 Add new models (one-line dict entry in `server.py`)
- 🌐 Improve the web UI (`public/index.html`)
- 🧮 Better tokenizer heuristics (`fast_token_estimate()`)
- 📦 npm / npx wrapper
- 🔌 Editor extensions (VS Code, Neovim, Zed)

---

## Roadmap

- [ ] VS Code extension
- [ ] Neovim plugin  
- [ ] Conversation history analysis (import ChatGPT/Claude exports)
- [ ] Token budget planner (split text into N-token chunks)
- [ ] Live token streaming (watch a prompt being built)
- [ ] GitHub Action for CI token budgets

---

## License

MIT — do whatever you want with it.

---

<p align="center">
  Made for developers who are tired of guessing their token usage
  <br><br>
  <a href="../../stargazers">⭐ Star this repo if it saved you from a truncated context</a>
</p>
