ThriftLM — Semantic Cache for LLMs

v0.1.5 — now on PyPI

# ThriftLM

Semantic cache layer for LLM apps. Exact-match in microseconds. Near-match in milliseconds. PII scrubbed before anything touches your database.

Get started GitHub
73.5%
Hit rate @ 0.82
1ms
Semantic lookup
1200×
Faster than LLM
0PII
Stored in plain text
Architecture

## Three tiers.
One call.

Every query cascades through three layers before ever hitting your LLM provider. Most never make it past the second.

📥
Query
Raw input
→
⚡
Redis
Exact hash ~0.5ms
→
🧮
LocalIndex
Numpy cosine ~1ms
→
🤖
LLM
Miss only
→
🔒
PII Scrub
Then store
✓ HIT
~1ms
Redis or LocalIndex match. No LLM billed. No wait.
↻ MISS
Full latency
LLM called once. Response PII-scrubbed and stored. All future hits are free.
Quickstart

## Up in
sixty seconds.

Your existing LLM function stays exactly as-is. One wrapper replaces every direct call.

bash

    pip install thriftlm
    # For the local dashboard + API server:
    pip install thriftlm[api]

python

    from thriftlm import SemanticCache
    import openai

    # Initialize once per process — bulk-loads embeddings into local numpy index
    cache = SemanticCache(threshold=0.82, api_key="your-key")

    def call_llm(query: str) -> str:
        res = openai.chat.completions.create(
            model="gpt-4o-mini",
            messages=[{"role": "user", "content": query}]
        )
        return res.choices[0].message.content

    # Drop-in. Handles cache check + LLM fallback automatically.
    response = cache.get_or_call("Explain semantic caching", call_llm)

    # Near-duplicate query → instant hit, no LLM called
    response2 = cache.get_or_call("What is semantic caching?", call_llm)

.env

    SUPABASE_URL=https://your-project.supabase.co
    SUPABASE_KEY=your-anon-key
    REDIS_URL=redis://localhost:6379
    OPENAI_API_KEY=sk-...

01

#### Supabase + pgvector

Create a project at supabase.com and run `supabase/setup.sql` to create the HNSW-indexed cache table.

02

#### Redis

Run `docker compose up -d` for local Redis, or point `REDIS_URL` at Upstash for a managed option.

03

#### Wrap your LLM call

Replace your direct LLM call with `cache.get_or_call()`. That is the entire integration.

04

#### View metrics

Run `thriftlm serve --api-key your-key`. Opens a live dashboard at `localhost:8000` — bundled in the package, nothing extra to deploy.

bash

    # See live hit rate, tokens saved, and top cached queries
    thriftlm serve --api-key your-key
    # → ThriftLM dashboard → http://localhost:8000
    # → Opens in browser automatically

What's included

## Everything.
Nothing extra.

Open-source, self-hostable, no vendor lock-in. Your data stays on your infrastructure.

⚡
Redis Exact Cache
Embedding hash stored in Redis. Sub-millisecond exact matches with zero Supabase round-trips.
🧮
Local Numpy Index
All embeddings loaded into a float32 matrix at startup. Cosine similarity via matmul — no network latency ever.
🔒
PII Scrubbing
Presidio + spaCy strips names, emails, and phone numbers from every LLM response before it touches the database.
🗄️
Supabase HNSW
pgvector with HNSW index for accurate approximate nearest-neighbor search at any scale.
📊
thriftlm serve
Built-in metrics dashboard. Run `thriftlm serve --api-key sc_xxx` for live hit rates, tokens saved, and top queries — bundled in pip, nothing to deploy.
🔑
Multi-tenant
Each API key is an isolated cache namespace with its own LocalIndex. Ship it as a service.
QQP Benchmark

## Real numbers.
No cherry-picking.

200 duplicate question pairs from the Quora Question Pairs dataset. question1 stored, question2 used for lookup. Threshold controls the precision/recall tradeoff.

Threshold | Hit Rate | Hits / 200 |
--- | --- | --- | ---
0.70 | 92.5% | 185 |
0.75 | 86.0% | 172 |
0.80 | 78.0% | 156 |
0.82 ← default | 73.5% | 147 |
0.85 | 62.5% | 125 |
0.90 | 40.0% | 80 |

all-MiniLM-L6-v2 · mean sim=0.859 · HNSW index (Supabase pgvector)

Roadmap

## Shipped.
What's next.

V1 is the cache primitive. V2 is where it gets interesting.

shipped

#### V1 — Semantic Cache Core

Redis → LocalIndex → HNSW three-tier pipeline. Presidio PII scrubbing. Multi-tenant FastAPI. `thriftlm serve` bundled dashboard CLI. `pip install thriftlm`.

next

#### V2 — Agentic Plan Caching

Cache entire multi-step agent plans, not just responses. When intent repeats in a loop, skip re-planning entirely. Built for Claude Code SDK and long-running agent workflows.

later

#### ClawHub Distribution

Publish as an OpenClaw skill — drop-in for any Claude Code loop via a single slash command.

later

#### Cost Analytics

Per-model token pricing table. Real-time dollar savings tracked on the dashboard.

Get started

## Your LLM calls are
expensive and slow.

Three lines of Python. Open-source. Your data never leaves your own infrastructure.

pip install thriftlm