# Entroly

[![PyPI](https://img.shields.io/pypi/v/entroly)](https://pypi.org/project/entroly/)
[![CI](https://github.com/juyterman1000/entroly/actions/workflows/ci.yml/badge.svg)](https://github.com/juyterman1000/entroly/actions/workflows/ci.yml)
[![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](LICENSE)
[![Rust](https://img.shields.io/badge/core-Rust%20%2B%20PyO3-orange)](entroly-core/)

**Information-theoretic context optimization for AI coding agents.**

Every AI coding tool manages context with dumb FIFO truncation — stuffing tokens until the window is full, then cutting from the top. Entroly applies mathematics to select the **optimal** context subset.

```bash
pip install entroly[native]
```

## Zero-Friction Setup

```bash
pip install entroly[native]
cd your-project
entroly init        # auto-detects Cursor / VS Code / Windsurf / Claude Desktop
                    # writes the correct MCP config in one command
# Restart your AI tool — done.
```

`entroly init` detects your project type and AI tool, generates the right `mcp.json`, and confirms how many files it will auto-index on first run. The MCP server then automatically indexes your codebase (via `git ls-files`) when it starts — no manual `remember_fragment` calls needed.

```bash
entroly serve       # start MCP server with auto-indexing
entroly dashboard   # show live ROI metrics (cost saved, latency, compression)
```

## Architecture

Hybrid Rust + Python: CPU-intensive math (knapsack DP, entropy, SimHash, LSH, dependency graph) runs in Rust via PyO3 for 50-100x speedup. MCP protocol and orchestration run in Python via FastMCP. The Rust extension (`entroly-core`) is required — install with `pip install entroly[native]`.

## What It Does

An MCP server that sits between your AI coding tool and the LLM, providing:

| Engine | What it does | How it works |
|--------|-------------|--------------|
| **Knapsack Optimizer** | Selects mathematically optimal context subset | 0/1 Knapsack DP with budget quantization (N ≤ 2000), greedy fallback (N > 2000) |
| **Entropy Scorer** | Measures information density per fragment | Shannon entropy (40%) + boilerplate detection (30%) + cross-fragment multi-scale n-gram redundancy (30%) |
| **SimHash Dedup** | Catches near-duplicate content in O(1) | 64-bit SimHash fingerprints with 4-band LSH bucketing, Hamming threshold = 3 |
| **Multi-Probe LSH Index** | Sub-linear semantic recall over 100K+ fragments | 12-table LSH with 10-bit sampling + 3-neighbor multi-probe queries |
| **Dependency Graph** | Pulls in related code fragments together | Symbol table + auto-linking (imports, type refs, function calls) + two-pass knapsack refinement |
| **Predictive Pre-fetch** | Pre-loads context before the agent asks | Static import analysis + test file inference + learned co-access patterns |
| **Checkpoint & Resume** | Crash recovery for multi-step tasks | Gzipped JSON state serialization (~100 KB per checkpoint) |
| **Feedback Loop** | Learns which context leads to good outputs | Wilson score lower-bound confidence intervals (same formula as Reddit ranking) |
| **Context Ordering** | Orders fragments for optimal LLM attention | Pinned → criticality level → dependency count → relevance score |
| **Guardrails** | Auto-pins safety-critical files, classifies tasks | Criticality levels (Safety/Critical/Important/Normal) + task-aware budget multipliers |
| **PRISM Optimizer** | Adapts scoring weights to the codebase | Anisotropic spectral optimization via Jacobi eigendecomposition on 4×4 covariance matrix |
| **Provenance Chain** | Detects hallucination risk in selected context | Tracks source verification + confidence scoring per fragment |

## Setup

### Cursor

```json
{
  "mcpServers": {
    "entroly": {
      "command": "entroly",
      "args": ["serve"]
    }
  }
}
```

### Claude Code

```bash
claude mcp add entroly -- entroly serve
```

### Cline / Any MCP Client

```json
{
  "entroly": {
    "command": "entroly",
    "args": ["serve"]
  }
}
```

> **Tip:** Run `entroly init` to have this generated automatically for your tool.

## MCP Tools

### `remember_fragment`
Store context with auto-dedup, entropy scoring, dependency linking, and criticality detection.

```
remember_fragment(content="def process_payment(...)...", source="payments.py", token_count=45)
→ {"status": "ingested", "entropy_score": 0.82}

remember_fragment(content="def process_payment(...)...")  # same content
→ {"status": "duplicate", "duplicate_of": "a1b2c3", "tokens_saved": 45}
```

### `optimize_context`
Select the optimal context subset for a token budget. Includes dependency boosting, ε-greedy exploration, context sufficiency scoring, and provenance metadata.

```
optimize_context(token_budget=128000, query="fix payment bug")
→ {
    "selected_fragments": [...],
    "optimization_stats": {"method": "exact_dp", "budget_utilization": 0.73},
    "tokens_saved_this_call": 42000,
    "sufficiency": 0.91,
    "hallucination_risk": "low"
  }
```

### `recall_relevant`
Sub-linear semantic recall via multi-probe LSH. Falls back to brute-force scan on cold start.

```
recall_relevant(query="database connection pooling", top_k=5)
→ [{"fragment_id": "...", "relevance": 0.87, "content": "..."}]
```

### `record_outcome`
Feed the Wilson score feedback loop. Adjusts fragment scoring multipliers in the range [0.5, 2.0].

```
record_outcome(fragment_ids="a1b2c3,d4e5f6", success=true)
→ {"status": "recorded", "outcome": "success", "fragment_ids": ["a1b2c3", "d4e5f6"]}
```

### `explain_context`
Per-fragment scoring breakdown with sufficiency analysis and exploration swap log.

```
explain_context()
→ {
    "fragments": [{"id": "...", "recency": 0.9, "frequency": 0.3, "semantic": 0.7, "entropy": 0.8}],
    "sufficiency": 0.91,
    "exploration_swaps": 1
  }
```

### `checkpoint_state` / `resume_state`
Save and restore full session state — fragments, dedup index, co-access patterns, feedback scores.

```
checkpoint_state(task_description="Refactoring auth module", current_step="Step 5/8")
→ {"status": "checkpoint_saved", "fragments_saved": 47}

resume_state()
→ {"status": "resumed", "restored_fragments": 47, "metadata": {"step": "Step 5/8"}}
```

### `prefetch_related`
Predict and pre-load likely-needed context using import analysis, test file inference, and co-access history.

```
prefetch_related(file_path="src/payments.py", source_content="from utils import...")
→ [{"path": "src/utils.py", "reason": "import", "confidence": 0.70}]
```

### `get_stats`
Session statistics and cost savings.

```
get_stats()
→ {
    "session":  { "total_fragments": 142, "total_tokens_tracked": 384000, "avg_entropy": 0.84 },
    "savings":  { "total_tokens_saved": 284000, "estimated_cost_saved_usd": 0.85 },
    "performance": { "avg_optimize_us": 320, "context_compression": 0.39 },
    "memory":   { "total_kb": 612, "naive_cost_per_call_usd": 0.0115, "optimized_cost_per_call_usd": 0.0044 },
    "dedup":    { "duplicates_detected": 12 },
    "context_efficiency": { "context_efficiency": 0.88 }
  }
```

### `entroly_dashboard`
Full product-owner dashboard with live ROI from the running session. Shows real numbers — no mocks or synthetic data.

```
entroly_dashboard()
→ {
    "money": {
      "cost_per_call_without_entroly": "$0.0115",
      "cost_per_call_with_entroly":    "$0.0044",
      "savings_pct": "62%",
      "insight": "Each optimize call costs $0.0044 instead of $0.0115. Over 38 calls, that's $0.27 saved."
    },
    "performance": {
      "avg_optimize_latency": "320µs (0.32ms)",
      "vs_api_roundtrip":     "6250x faster than a typical API call"
    },
    "bloat_prevention": {
      "context_compression": "39.00%",
      "memory_footprint":    "612 KB",
      "duplicates_caught":   "12"
    },
    "selection_quality": { "information_density": "0.8840 bits/token" },
    "safety":            { "persistent_index": "active" },
    "last_optimization": { "context_sufficiency": "91%", "selected": 14, "excluded": 128 }
  }
```

## The Math

### Multi-Dimensional Relevance Scoring

Each fragment is scored across four dimensions:

```
r(f) = (w_rec · recency + w_freq · frequency + w_sem · semantic + w_ent · entropy)
       / (w_rec + w_freq + w_sem + w_ent)
       × feedback_multiplier
```

Default weights: recency 0.30, frequency 0.25, semantic 0.25, entropy 0.20.

- **Recency**: Ebbinghaus forgetting curve — `exp(-ln(2) × Δt / half_life)`, half_life = 15 turns
- **Frequency**: Normalized access count (spaced repetition boost)
- **Semantic similarity**: SimHash Hamming distance to query, normalized to [0, 1]
- **Information density**: Shannon entropy + boilerplate + redundancy (see below)

### Knapsack Context Selection

Context selection is the 0/1 Knapsack Problem:

```
Maximize:   Σ r(fᵢ) · x(fᵢ)     for selected fragments
Subject to: Σ c(fᵢ) · x(fᵢ) ≤ B  (token budget)
```

**Two strategies** based on fragment count:
- **N ≤ 2000**: Exact DP with budget quantization into 1000 bins — O(N × 1000)
- **N > 2000**: Greedy density sort — O(N log N), Dantzig 0.5-optimality guarantee

Pinned fragments (safety-critical files, config files) are always included; remaining budget is allocated via DP/greedy.

### Shannon Entropy Scoring

Three components combined:

```
score = 0.40 × normalized_entropy + 0.30 × (1 - boilerplate_ratio) + 0.30 × (1 - redundancy)
```

- **Shannon entropy** (40%): `H = -Σ p(char) · log₂(p(char))`, normalized by 6.0 bits/char. Stack-allocated 256-byte histogram, single O(n) pass.
- **Boilerplate detection** (30%): Pattern matching for imports, pass, dunder methods, closing delimiters.
- **Cross-fragment redundancy** (30%): Multi-scale n-gram overlap with adaptive weights by fragment length — bigram-heavy for short fragments (<20 words), 4-gram-heavy for long fragments (>100 words). Parallelized with rayon.

### SimHash Deduplication

64-bit fingerprints from word trigrams hashed via MD5:
- Hamming distance ≤ 3 → near-duplicate
- 4-band LSH bucketing for O(1) candidate lookup
- Separate 12-table multi-probe LSH index for semantic recall (~3 μs over 100K fragments)

### Dependency Graph

Auto-linking via source analysis:
- **Imports** (strength 1.0): Python `from X import Y`, Rust `use`, JS `import`
- **Type references** (0.9): Type annotations, isinstance checks
- **Function calls** (0.7): General identifier usage matching against symbol table
- **Same module** (0.5): Co-located definitions

Two-pass knapsack refinement: initial selection → boost dependencies of selected fragments → re-optimize.

### Task-Aware Budget Multipliers

```
Bug tracing / debugging     → 1.5× budget
Exploration / understanding → 1.3× budget
Refactoring / code review   → 1.0× budget
Testing                     → 0.8× budget
Code generation             → 0.7× budget
Documentation               → 0.6× budget
```

### PRISM Spectral Optimizer

Tracks a 4×4 covariance matrix over scoring dimensions [recency, frequency, semantic, entropy] with EMA updates (β=0.95). Jacobi eigendecomposition finds principal axes. Anisotropic spectral gain dampens noisy dimensions and amplifies clean signals — automatic learning rate adaptation without hyperparameter tuning.

## Configuration

```python
EntrolyConfig(
    default_token_budget=128_000,     # GPT-4 Turbo equivalent
    max_fragments=10_000,             # session fragment cap
    weight_recency=0.30,              # scoring weights (sum to 1.0)
    weight_frequency=0.25,
    weight_semantic_sim=0.25,
    weight_entropy=0.20,
    decay_half_life_turns=15,         # Ebbinghaus half-life
    min_relevance_threshold=0.05,     # auto-evict below this
    dedup_similarity_threshold=0.92,
    prefetch_depth=2,
    max_prefetch_fragments=10,
    auto_checkpoint_interval=5,       # checkpoint every N tool calls
)
```

## References

- Shannon (1948) — Information Theory
- Charikar (2002) — SimHash
- Ebbinghaus (1885) — Forgetting Curve
- Dantzig (1957) — Greedy Knapsack Approximation
- Wilson (1927) — Score Confidence Intervals
- ICPC (arXiv 2025) — In-context Prompt Compression
- Proximity (arXiv 2025) — LSH-bucketed Semantic Caching
- RCC (ICLR 2025) — Recurrent Context Compression
- ILRe (arXiv 2025) — Intermediate Layer Retrieval
- Agentic Plan Caching (arXiv 2025)

## Part of the Ebbiforge Ecosystem

Entroly integrates with [hippocampus-sharp-memory](https://pypi.org/project/hippocampus-sharp-memory/) for persistent memory and [Ebbiforge](https://pypi.org/project/ebbiforge/) for TF embeddings and RL weight learning. Both are optional — Entroly works standalone.

## License

MIT
