<p align="center">
    <img src="https://github.com/user-attachments/assets/735141f8-56ff-40ce-8a4e-013dbecfe299" alt="SteadyText Logo" height=320 width=480 />
</p>

# SteadyText

*Deterministic text generation and embeddings with zero configuration*

[![](https://img.shields.io/pypi/v/steadytext.svg)](https://pypi.org/project/steadytext/)
[![](https://img.shields.io/pypi/pyversions/steadytext.svg)](https://pypi.org/project/steadytext/)
[![](https://img.shields.io/badge/License-MIT-blue.svg)](LICENSE)

> [!IMPORTANT]
> **Version 2025.8.17 - Date-Based Versioning**
> 
> SteadyText has transitioned from semantic versioning to **date-based versioning** (yyyy.mm.dd format).
> 
> **Why this change?** The rapid pace of AI model improvements and feature additions made traditional semantic versioning impractical. With models evolving weekly and new capabilities being added frequently, date-based versioning provides clearer insight into release recency and better aligns with our continuous improvement philosophy.
> 
> This applies to both the Python package and the PostgreSQL extension (pg_steadytext). Existing installations can upgrade normally using pip or PostgreSQL extension commands.

**Same input → same output. Every time.**
No more flaky tests, unpredictable CLI tools, or inconsistent docs. SteadyText makes AI outputs as reliable as hash functions.

Ever had an AI test fail randomly? Or a CLI tool give different answers each run? SteadyText makes AI outputs reproducible - perfect for testing, tooling, and anywhere you need consistent results.

> [!TIP]
> ✨ _Powered by open-source AI workflows from [**Julep**](https://julep.ai)._ ✨

---

## 🚀 Quick Start

### Installing with UV (Recommended - 10-100x faster)

```bash
# Install UV package manager
curl -LsSf https://astral.sh/uv/install.sh | sh

# Install SteadyText
uv add steadytext
```

### Installing with pip

```bash
pip install steadytext
```

### Installing from Source (for development or llama-cpp-python build issues)

Due to the specific build requirements for the inference-sh fork of llama-cpp-python, you may need to install from source:

```bash
# Clone the repository
git clone https://github.com/julep-ai/steadytext.git
cd steadytext

# Set required environment variables
export FORCE_CMAKE=1
export CMAKE_ARGS="-DLLAVA_BUILD=OFF -DGGML_ACCELERATE=OFF -DGGML_BLAS=OFF -DGGML_CUDA=OFF -DGGML_BUILD_TESTS=OFF -DGGML_BUILD_EXAMPLES=OFF"

# Install with UV (recommended)
uv sync

# Or install with pip
pip install -e .
```

```python
import steadytext

# Deterministic text generation (uses daemon by default)
code = steadytext.generate("implement binary search in Python")
assert "def binary_search" in code  # Always passes!

# Choose model size (small=4B, large=30B)
quick = steadytext.generate("summarize this", size="small")  # Fast
detailed = steadytext.generate("explain in detail", size="large")  # Higher quality

# Generate with controlled randomness (temperature > 0)
creative = steadytext.generate("write a poem", temperature=0.8)
more_creative = steadytext.generate("write a poem", temperature=1.2)

# Streaming (also deterministic)
for token in steadytext.generate_iter("explain quantum computing"):
    print(token, end="", flush=True)

# Deterministic embeddings (uses Jina v4 by default)
vec = steadytext.embed("What is machine learning?")  # 1024-dimensional embedding
# Note: Jina v4 outputs 2048 dimensions but we truncate to 1024 for backward compatibility

# Explicit daemon usage (ensures connection)
from steadytext.daemon import use_daemon
with use_daemon():
    code = steadytext.generate("implement quicksort")
    embedding = steadytext.embed("machine learning")

# Model switching (v2.0.0+)
fast_response = steadytext.generate("Quick task", size="small")  # Qwen3-4B (default)
quality_response = steadytext.generate("Complex analysis", size="large")  # Qwen3-30B

# Size-based selection (v2025.8.16+)
small = steadytext.generate("Simple task", size="small")      # Qwen3-4B (default)
large = steadytext.generate("Complex task", size="large")    # Qwen3-30B
```

_Or,_

```bash
echo "hello" | uvx steadytext
```

---

## 📜 License Notice

The default generation models (Qwen3 family) are MIT licensed and open source. These models provide high-quality text generation with full commercial usage rights.

**Note:** The library supports various models including Gemma models which have their own [Terms of Use](https://ai.google.dev/gemma/terms).

---

## 🐘 PostgreSQL Extension

Transform your PostgreSQL database into an AI-powered system with **pg_steadytext** - the production-ready PostgreSQL extension that brings deterministic AI directly to your SQL queries.

### Key Features

- **Native SQL Functions**: Generate text and embeddings using simple SQL commands
- **Async Processing**: Non-blocking AI operations with queue-based background workers  
- **AI Summarization**: Enhanced aggregate functions (`steadytext_summarize`) with remote model support and TimescaleDB compatibility (v2025.8.17)
- **Structured Generation**: Generate JSON, regex-constrained text, and multiple-choice outputs
- **pgvector Integration**: Seamless compatibility for similarity search and vector operations
- **Built-in Caching**: PostgreSQL-based frecency cache that mirrors SteadyText's performance
- **Schema Qualification**: Full support for TimescaleDB continuous aggregates and custom schemas (v2025.8.17)

### Quick Example

```sql
-- Generate text
SELECT steadytext_generate('Write a product description for wireless headphones');

-- Create embeddings for similarity search
SELECT steadytext_embed('machine learning') <-> steadytext_embed('artificial intelligence');

-- AI-powered summarization (v2025.8.17+)
SELECT steadytext_summarize(content) AS summary
FROM documents
WHERE created_at > NOW() - INTERVAL '1 day'
GROUP BY category;

-- Or use short alias
SELECT st_summarize(content, 
    jsonb_build_object('max_facts', 10, 'model', 'openai:gpt-4o-mini', 'unsafe_mode', true)) 
FROM documents;

-- Structured JSON generation
SELECT steadytext_generate_json(
    'Create a user profile',
    '{"type": "object", "properties": {"name": {"type": "string"}, "age": {"type": "integer"}}}'::jsonb
);
```

📚 **[Full PostgreSQL Extension Documentation →](pg_steadytext/)**

---

## 🔧 How It Works

SteadyText achieves determinism via:

* **Customizable seeds:** Control determinism with a `seed` parameter, while still defaulting to `42`.
* **Greedy decoding:** Always chooses highest-probability token
* **Frecency cache:** LRU cache with frequency counting—popular prompts stay cached longer
* **Quantized models:** 8-bit quantization ensures identical results across platforms
* **Model switching:** Dynamically switch between models while maintaining determinism (v1.0.0+)
* **Daemon architecture:** Persistent model serving eliminates loading overhead (v1.2.0+)

This means `generate("hello")` returns the exact same 512 tokens on any machine, every single time.

## 🌐 Ecosystem

SteadyText is more than just a library. It's a full ecosystem for deterministic AI:

- **Python Library**: The core `steadytext` library for programmatic use in your applications.
- **Command-Line Interface (CLI)**: A powerful `st` command to use SteadyText from your shell for scripting and automation.
- **PostgreSQL Extension (pg_steadytext)**: Production-ready extension with async processing, AI summarization, and structured generation for SQL-native AI operations.
- **Zsh Plugin**: Supercharge your shell with AI-powered command suggestions and history search.
- **Cloudflare Worker**: Deploy SteadyText to the edge with a Cloudflare Worker for distributed, low-latency applications.

### ⚡ Daemon Architecture (Optional but Recommended)

SteadyText includes a daemon mode that provides significantly better performance:

* **Persistent serving:** Models stay loaded in memory between requests
* **Zero loading overhead:** Skip the 2-3 second model loading time on each call
* **Explicit startup required:** Start with `st daemon start` for best performance
* **Automatic fallback:** Gracefully falls back to direct model loading if daemon not running
* **Centralized caching:** Consistent cache behavior between daemon and direct access

```bash
# Start the daemon for better performance
st daemon start

# Now all operations will use the daemon
echo "Hello world" | st  # Much faster with daemon running!
```

```python
# Python automatically uses daemon if available
text = steadytext.generate("Hello world")  # Fast if daemon is running

# Explicit daemon usage (ensures connection)
from steadytext.daemon import use_daemon
with use_daemon():
    text = steadytext.generate("Hello world")
    embedding = steadytext.embed("Some text")

# Disable daemon globally
import os
os.environ["STEADYTEXT_DISABLE_DAEMON"] = "1"
text = steadytext.generate("Hello world")  # Direct model loading
```

---

## Fun Goofy Sample You Can Try

They will (literally) **always** do this...

```bash
❯ st generate --verbose --size large --eos-string STOP "DON'T SAY STOP (ALL CAPS) NO MATTER WHAT"
Understood. I will not use the word "%
```

> [!NOTE]
> This is by far the single best explanation of recursion in existence.

```bash
$> echo "explain recursion in pig latin" | st --verbose --size large

Okay, let's explain recursion in Pig Latin!  It's a bit tricky to do *in* Pig Latin, but I'll try to explain the concept and then give a Pig Latin-flavored analogy.

**What is Recursion? (In English)**

Recursion is like a set of instructions that calls *itself* to solve a smaller version of the same problem. Think of it like Russian nesting dolls (Matryoshka dolls). Each doll contains a smaller version of itself.

Here's the breakdown:

1. **Base Case:**  This is the *stopping point*.  It's the simplest version of the problem that you can solve directly, *without* calling the function again.  Without a base case, the recursion would go on forever (like an infinite loop!).

2. **Recursive Step:** This is where the function calls *itself*, but with a slightly modified (smaller) version of the original problem.  Each call gets closer to the base case.

**Example (in English):**

Let's say you want to calculate the factorial of a number (e.g., 5! = 5 * 4 * 3 * 2 * 1 = 120).  You can do this recursively:

* **Base Case:** If the number is 1, the factorial is 1.
* **Recursive Step:**  If the number is greater than 1, the factorial is the number multiplied by the factorial of the number minus 1.

**Pig Latin Analogy (Trying to explain it *using* Pig Latin):**

Okay, this is where it gets fun (and a little silly)!  Let's say we want to count the number of "ay" sounds in a word.

Here's how we could *imagine* a recursive Pig Latin function to do this:

\```piglatin
"Ehay-ay"  ay-ay-ay-ay-ay-ay-ay-ay-ay-ay-ay-ay-ay-ay-ay-ay-ay-ay-ay-ay-ay-ay-ay-ay-ay-ay-ay-ay-ay-ay-ay-ay-ay-ay-ay-ay-ay-ay-ay-ay-ay-ay-ay-ay-ay-ay-ay-ay-ay-ay-ay-ay-ay-ay-ay-%
```

---

## ✨ Structured Generation (v2.4.1+)

SteadyText now supports structured generation using llama.cpp's native grammar support, allowing you to force the model's output to conform to a specific format.

- **JSON Generation**: Generate JSON that validates against a schema or Pydantic model.
- **Regex Matching**: Constrain output to a regular expression.
- **Multiple Choice**: Force the output to be one of a list of choices.

### Python API

```python
import steadytext
from pydantic import BaseModel

# JSON generation with a Pydantic model
class User(BaseModel):
    name: str
    email: str

user_json = steadytext.generate(
    "Create a user: name John Doe, email john.doe@example.com",
    schema=User,
    temperature=0.0  # Fully deterministic (default)
# Output contains: <json-output>{"name": "John Doe", "email": "john.doe@example.com"}</json-output>

# Regex-constrained generation
phone = steadytext.generate("My number is ", regex=r"\(\d{3}\) \d{3}-\d{4}")
# Output: (123) 456-7890

# Multiple choice
response = steadytext.generate("Is this useful?", choices=["Yes", "No"])
# Output: Yes
```

### CLI Support

```bash
# JSON generation with schema
echo "Create a person" | st --schema '{"type": "object", "properties": {"name": {"type": "string"}, "age": {"type": "integer"}}}' --wait

# JSON from schema file
echo "Generate user data" | st --schema user_schema.json --wait

# Regex pattern matching
echo "My phone is" | st --regex '\d{3}-\d{3}-\d{4}' --wait

# Multiple choice selection
echo "Is Python good?" | st --choices "yes,no,maybe" --wait
```

📚 **[Learn more in the Structured Generation Guide](docs/structured-generation.md)**

---

## ⚠️ Unsafe Mode: Remote Models (Experimental)

SteadyText now supports remote AI models (OpenAI, Cerebras, VoyageAI, Jina) with **best-effort determinism** via seed parameters. This feature is explicitly marked as "unsafe" because remote models cannot guarantee reproducibility.

**New in v2025.8.17+:**
- **GPT-5 Series Support**: Full support for OpenAI's GPT-5 reasoning models (gpt-5-mini, gpt-5-pro, etc.) with automatic temperature adjustment
- **Custom Provider Options**: Pass provider-specific parameters directly to remote APIs via the `options` parameter

### Why Use Unsafe Mode?

- Access to larger, more capable models
- Prototype before switching to local models
- Use when true determinism isn't critical

### Quick Example

```bash
# Install OpenAI client (required for OpenAI models)
pip install openai
# or
pip install steadytext[unsafe]

# Enable unsafe mode
export STEADYTEXT_UNSAFE_MODE=true

# Use OpenAI for generation
echo "Explain quantum computing" | st --unsafe-mode --model openai:gpt-4o-mini

# Use GPT-5 reasoning models (temperature automatically set to 1.0)
echo "Solve this problem" | st --unsafe-mode --model openai:gpt-5-mini
echo "Complex reasoning" | st --unsafe-mode --model openai:gpt-5-pro

# Pass custom options to providers (v2025.8.17+)
echo "Creative writing" | st --unsafe-mode --model openai:gpt-4o-mini --options '{"top_p": 0.95, "presence_penalty": 0.5}'

# Use Cerebras for fast generation
echo "Write Python code" | st --unsafe-mode --model cerebras:llama3.1-8b

# Use VoyageAI for embeddings
echo "Advanced search query" | st embed --unsafe-mode --model voyageai:voyage-3

# Use Jina for multilingual embeddings
echo "你好世界" | st embed --unsafe-mode --model jina:jina-embeddings-v3

# List available remote models
st unsafe list-models
```

### Python API

```python
import os
import steadytext

# Enable unsafe mode
os.environ["STEADYTEXT_UNSAFE_MODE"] = "true"

# Generate with OpenAI (requires OPENAI_API_KEY)
text = steadytext.generate(
    "Explain AI", 
    model="openai:gpt-4o-mini",
    seed=42  # Best-effort determinism only
)

# Use GPT-5 reasoning models (temperature automatically adjusted to 1.0)
text = steadytext.generate(
    "Complex reasoning task",
    model="openai:gpt-5-mini",
    unsafe_mode=True
)

# Pass custom provider options (v2025.8.17+)
text = steadytext.generate(
    "Creative writing",
    model="openai:gpt-4o-mini",
    unsafe_mode=True,
    options={"top_p": 0.95, "presence_penalty": 0.5}
)
```

### Supported Remote Providers

| Provider | Generation | Embeddings | Determinism | API Key |
|----------|------------|------------|-------------|---------|
| **OpenAI** | ✅ gpt-4o, gpt-4o-mini<br>✅ gpt-5-mini, gpt-5-pro (reasoning) | ✅ text-embedding-3-* | Best-effort (seed param) | `OPENAI_API_KEY` |
| **Cerebras** | ✅ Llama models | ❌ | Best-effort (seed param) | `CEREBRAS_API_KEY` |
| **VoyageAI** | ❌ | ✅ voyage-3, voyage-large-2 | No seed support | `VOYAGE_API_KEY` |
| **Jina AI** | ❌ | ✅ jina-embeddings-v3/v2 | No seed support | `JINA_API_KEY` |

⚠️ **WARNING**: Remote models may produce different outputs despite using the same seed. Use local GGUF models for guaranteed determinism.

📚 **[Learn more in the Unsafe Mode Guide](docs/unsafe-mode.md)**

---

## 📦 Installation & Models

Install stable release:

```bash
pip install steadytext
```

#### Models

**Default models (v2025.8.17+)**:

* Generation (Small): `Qwen3-4B-Instruct` (3.9GB) - High-quality 4B parameter model
* Generation (Large): `Qwen3-30B-A3B-Instruct` (12GB) - Advanced 30B parameter model with A3B architecture  
* Embeddings: `Jina-v4-Text-Retrieval` (1.2GB) - State-of-the-art 2048-dim embeddings (truncated to 1024)
* Reranking: `Qwen3-Reranker-4B` (3.5GB) - Document reranking model

**Dynamic model switching (v1.0.0+):**

Switch between different models at runtime:

```python
# Use built-in model registry
text = steadytext.generate("Hello", size="large")  # Uses Qwen3-30B

# Use size parameter for model selection
text = steadytext.generate("Hello", size="small")  # Uses Qwen3-4B (default)
text = steadytext.generate("Hello", size="large")  # Uses Qwen3-30B

# Or specify custom models
text = steadytext.generate(
    "Hello",
    model_repo="unsloth/Qwen3-4B-Instruct-2507-GGUF",
    model_filename="Qwen3-4B-Instruct-2507-UD-Q6_K_XL.gguf"
)
```

Available models: Qwen3 models in 4B and 30B variants

Size shortcuts: `mini` (270M, CI/testing), `small` (4B, default), `large` (30B)

> Each model produces deterministic outputs. The default model remains fixed per major version.

## 🚀 Mini Models for CI/Testing

SteadyText includes support for "mini" models - extremely small models (~10x smaller) designed for fast CI testing and development environments where speed matters more than quality.

**🎯 Tests use mini models by default!** Running `poe test` or `pytest` via the project's configuration automatically uses mini models for faster testing.

### Mini Model Sizes
- **Generation**: Gemma-3-270M (~97MB) - Tiny but functional text generation (mini mode only)
- **Embedding**: BGE-large-en-v1.5 (~130MB) - Produces compatible 1024-dim embeddings
- **Reranking**: BGE-reranker-base (~300MB) - Basic reranking capabilities

### Usage in Testing

**Default test commands (use mini models automatically):**
```bash
# These commands use mini models by default (configured in pyproject.toml)
uv run poe test           # Run tests with mini models
uv run poe test-cov       # Run tests with coverage using mini models

# To test with full models instead:
uv run poe test-full      # Run tests with regular models
```

### Usage in CI

**Environment Variable (Recommended for CI):**
```bash
# Enable mini models globally
export STEADYTEXT_USE_MINI_MODELS=true
export STEADYTEXT_ALLOW_MODEL_DOWNLOADS=true

# All operations will now use mini models
st "Generate some text"  # Uses gemma-mini-270m
st embed "Create embedding"  # Uses bge-embedding-mini
st rerank "query" "doc1" "doc2"  # Uses bge-reranker-mini
```

**CLI with --size flag:**
```bash
# Use mini models for specific commands
st --size mini "Quick test generation"
st embed --size mini "Test embedding"
st rerank --size mini "query" "document"

# Start daemon with mini models
st daemon start --size mini
```

**Python API:**
```python
# Use mini models programmatically
text = steadytext.generate("Test prompt", size="mini")
```

### CI Workflow Example

```yaml
name: Test with Mini Models
on: [push, pull_request]

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - uses: actions/setup-python@v4
        with:
          python-version: '3.11'
      
      - name: Install dependencies
        run: pip install -e .
      
      - name: Run tests with mini models
        env:
          STEADYTEXT_USE_MINI_MODELS: true
          STEADYTEXT_ALLOW_MODEL_DOWNLOADS: true
        run: |
          # Quick smoke test
          echo "Test" | st --size mini
          
          # Run full test suite
          pytest tests/
```

### Benefits
- ✅ ~10x faster model downloads (530MB total vs 5GB+)
- ✅ Faster test execution
- ✅ Lower memory usage
- ✅ Full API compatibility
- ✅ Deterministic outputs maintained

> **Note**: Mini models trade quality for speed. Use them for testing and CI, not production.

## Version History

| Version | Key Features                                                                                                                            | Default Generation Model                               | Default Embedding Model                                | Default Reranking Model | Python Versions |
| :------ | :-------------------------------------------------------------------------------------------------------------------------------------- | :----------------------------------------------------- | :----------------------------------------------------- | :---------------------- | :-------------- |
| **2025.8.x** | - **Date-Based Versioning**: Switched to yyyy.mm.dd format.<br>- **Qwen3 Models**: Using Qwen3-4B/30B for generation.<br>- **Jina v4 Embeddings**: State-of-the-art retrieval embeddings.<br>- **Enhanced PostgreSQL Extension**: AI summarization with remote model support.<br>- **Document Reranking**: Reranking functionality with `Qwen3-Reranker-4B` model. | `unsloth/Qwen3-4B-Instruct-2507-GGUF` (Qwen3-4B) / `Qwen3-30B-A3B` (large) | `jinaai/jina-embeddings-v4-text-retrieval-GGUF` (Jina v4) | `QuantFactory/Qwen3-Reranker-4B-GGUF` | `>=3.10, <3.14` |
| **1.x** | - **Model Switching**: Added support for switching models via environment variables.<br>- **Centralized Cache**: Unified cache system.<br>- **CLI Improvements**: Streaming by default, quiet output. | `Qwen/Qwen3-1.7B-GGUF` (Qwen3-1.7B-Q8_0.gguf) | `Qwen/Qwen3-Embedding-0.6B-GGUF` (Qwen3-Embedding-0.6B-Q8_0.gguf) | - | `>=3.10, <3.14` |
| **1.0-1.2** | - **Model Switching**: Added support for switching models via environment variables and a model registry.<br>- **Qwen3 Models**: Switched to `qwen3-1.7b` for generation.<br>- **Indexing**: Added support for FAISS indexing. | `Qwen/Qwen3-1.7B-GGUF` (Qwen3-1.7B-Q8_0.gguf) | `Qwen/Qwen3-Embedding-0.6B-GGUF` (Qwen3-Embedding-0.6B-Q8_0.gguf) | - | `>=3.10, <3.14` |
| **0.x** | - **Initial Release**: Deterministic text generation and embedding.                                                                      | `Qwen/Qwen1.5-0.5B-Chat-GGUF` (qwen1_5-0_5b-chat-q4_k_m.gguf) | `Qwen/Qwen1.5-0.5B-Chat-GGUF` (qwen1_5-0_5b-chat-q8_0.gguf) | - | `>=3.10`        |

### Breaking Changes in v2025.8.x

* **Date-based versioning:** Switched from semantic versioning to yyyy.mm.dd format
* **Qwen3 models:** Using Qwen3-4B and Qwen3-30B models for generation
* **Enhanced PostgreSQL functions:** Renamed ai_* functions to steadytext_* with st_* aliases
* **Thinking mode removed:** `thinking_mode` parameter and `--think` flag have been deprecated
* **Default output:** Default max tokens is 512

### Breaking Changes in v2.3.0+

* **Document Reranking:** Added reranking functionality with the Qwen3-Reranker-4B model
* **Reranking API:** New `steadytext.rerank()` function and `st rerank` CLI command

### Other Notable Changes

* **Daemon enabled by default:** Use `STEADYTEXT_DISABLE_DAEMON=1` to opt-out
* **Streaming by default:** CLI streams output by default, use `--wait` to disable
* **Quiet by default:** CLI is quiet by default, use `--verbose` for informational output
* **Centralized caching:** Cache system now shared between daemon and direct access
* **New CLI syntax:** Use `echo "prompt" | st` instead of `st generate "prompt"`

---

## ⚡ Performance

SteadyText delivers deterministic AI with production-ready performance:

* **Text Generation**: 21.4 generations/sec (46.7ms latency)
* **Embeddings**: 104-599 embeddings/sec (single to batch-50)
* **Cache Speedup**: 48x faster for repeated prompts
* **Memory**: ~1.4GB models, 150-200MB runtime
* **100% Deterministic**: Same output every time, verified across 100+ test runs
* **Accuracy**: 69.4% similarity for related texts, correct ordering maintained

📊 **[Full benchmarks →](docs/benchmarks.md)**

---

## 🎯 Examples

Use SteadyText in tests or CLI tools for consistent, reproducible results:

```python
# Testing with reliable assertions
def test_ai_function():
    result = my_ai_function("test input")
    expected = steadytext.generate("expected output for 'test input'")
    assert result == expected  # No flakes!

# CLI tools with consistent outputs
import click

@click.command()
def ai_tool(prompt):
    print(steadytext.generate(prompt))
```

📂 **[More examples →](examples/)**

---

## 🖥️ CLI Usage

### Daemon Management

```bash
# Daemon commands
st daemon start                    # Start daemon in background
st daemon start --foreground       # Run daemon in foreground
st daemon status                   # Check daemon status
st daemon status --json            # JSON status output
st daemon stop                     # Stop daemon gracefully
st daemon stop --force             # Force stop daemon
st daemon restart                  # Restart daemon

# Daemon configuration
st daemon start --host 127.0.0.1 --port 5678  # Custom host/port
```

### Text Generation

```bash
# Generate text (streams by default, uses daemon if running)
echo "write a hello world function" | st

# Generate with temperature for creativity
echo "write a poem" | st --temperature 0.8

# Disable streaming (wait for complete output)
echo "write a function" | st --wait

# Enable verbose output
echo "explain recursion" | st --verbose

# JSON output with metadata
echo "hello world" | st --json

# Get log probabilities
echo "predict next word" | st --logprobs
```

### Model Management

```bash
# List available models
st models list

# Download models
st models download --size small
st models download --model gemma-3n-4b
st models download --all

# Delete models
st models delete --size small
st models delete --model gemma-3n-4b
st models delete --all --force

# Preload models
st models preload
```

### Other Operations

```bash
# Get embeddings (local model - deterministic)
echo "machine learning" | st embed

# Remote embeddings (requires unsafe mode)
echo "machine learning" | st embed --model openai:text-embedding-3-small --unsafe-mode
echo "document text" | st embed --model voyageai:voyage-3-lite --unsafe-mode --json

# Document reranking (v2.3.0+)
st rerank "what is Python?" document1.txt document2.txt document3.txt
st rerank "search query" --file documents.txt --top-k 5 --json

# Vector operations
st vector similarity "cat" "dog"
st vector search "Python" candidate1.txt candidate2.txt candidate3.txt

# Create and search FAISS indices
st index create *.txt --output docs.faiss
st index search docs.faiss "how to install" --top-k 5

# Generate with automatic context from index
echo "what is the configuration?" | st --index-file docs.faiss

# Disable daemon for specific command
STEADYTEXT_DISABLE_DAEMON=1 echo "hello" | st

# Preload models
st models --preload
```

---

## 📋 When to Use SteadyText

✅ **Perfect for:**

* Testing AI features (reliable asserts)
* Deterministic CLI tooling
* Reproducible documentation & demos
* Offline/dev/staging environments
* Semantic caching and embedding search
* Vector similarity comparisons
* Document retrieval & RAG applications

❌ **Not ideal for:**

* Creative or conversational tasks
* Latest knowledge queries
* Large-scale chatbot deployments

---

## 🔍 API Overview

```python
# Text generation (uses daemon by default)
steadytext.generate(prompt: str, seed: int = 42, temperature: float = 0.0) -> str
steadytext.generate(prompt, return_logprobs=True, seed: int = 42, temperature: float = 0.0)


# Streaming generation
steadytext.generate_iter(prompt: str, seed: int = 42, temperature: float = 0.0)

# Embeddings (uses daemon by default)
steadytext.embed(text: str | List[str], seed: int = 42) -> np.ndarray

# Document reranking (v2.3.0+)
steadytext.rerank(
    query: str,
    documents: Union[str, List[str]],
    task: str = "Given a web search query, retrieve relevant passages that answer the query",
    return_scores: bool = True,
    seed: int = 42
) -> Union[List[Tuple[str, float]], List[str]]

# Daemon management
from steadytext.daemon import use_daemon
with use_daemon():  # Ensure daemon connection
    text = steadytext.generate("Hello")

# Model preloading
steadytext.preload_models(verbose=True)

# Cache management
from steadytext import get_cache_manager
cache_manager = get_cache_manager()
stats = cache_manager.get_cache_stats()
```

### Vector Operations (CLI)

```bash
# Compute similarity between texts
st vector similarity "text1" "text2" [--metric cosine|dot]

# Calculate distance between texts
st vector distance "text1" "text2" [--metric euclidean|manhattan|cosine]

# Find most similar text from candidates
st vector search "query" file1.txt file2.txt [--top-k 3]

# Average multiple text embeddings
st vector average "text1" "text2" "text3"

# Vector arithmetic
st vector arithmetic "king" - "man" + "woman"
```

### Index Management (CLI)

```bash
# Create FAISS index from documents
st index create doc1.txt doc2.txt --output my_index.faiss

# View index information
st index info my_index.faiss

# Search index
st index search my_index.faiss "query text" --top-k 5

# Use index with generation
echo "question" | st --index-file my_index.faiss
```

📚 [Full API Documentation](docs/api.md)

---

## 🔧 Configuration

### Cache Configuration

Control caching behavior via environment variables (affects both daemon and direct access):

```bash
# Generation cache (default: 256 entries, 50MB)
export STEADYTEXT_GENERATION_CACHE_CAPACITY=256
export STEADYTEXT_GENERATION_CACHE_MAX_SIZE_MB=50

# Embedding cache (default: 512 entries, 100MB)
export STEADYTEXT_EMBEDDING_CACHE_CAPACITY=512
export STEADYTEXT_EMBEDDING_CACHE_MAX_SIZE_MB=100
```

### Daemon Configuration

```bash
# Disable daemon globally (use direct model loading)
export STEADYTEXT_DISABLE_DAEMON=1

# Daemon connection settings
export STEADYTEXT_DAEMON_HOST=127.0.0.1
export STEADYTEXT_DAEMON_PORT=5678
```

### Model Downloads

```bash
# Allow model downloads in tests
export STEADYTEXT_ALLOW_MODEL_DOWNLOADS=true
```

---

## 📖 API Reference

### Text Generation

#### `generate(prompt: str, return_logprobs: bool = False, temperature: float = 0.0, model: Optional[str] = None, unsafe_mode: bool = False, options: Optional[Dict] = None) -> Union[str, Tuple[str, Optional[Dict]]]`

Generate text from a prompt with optional temperature control and remote model support.

```python
# Deterministic generation (default)
text = steadytext.generate("Write a haiku about Python")

# With controlled randomness
creative_text = steadytext.generate("Write a haiku about Python", temperature=0.8)

# With log probabilities
text, logprobs = steadytext.generate("Explain AI", return_logprobs=True)

# With remote models (v2025.8.17+)
text = steadytext.generate(
    "Solve this problem",
    model="openai:gpt-5-mini",
    unsafe_mode=True,
    options={"top_p": 0.9}  # Custom provider options
)
```

- **Parameters:**
  - `prompt`: Input text to generate from
  - `return_logprobs`: If True, returns tuple of (text, logprobs)
  - `temperature`: Controls randomness (0.0 = deterministic, >0 = more random)
  - `model`: Optional remote model (e.g., "openai:gpt-4o-mini", "openai:gpt-5-mini")
  - `unsafe_mode`: Enable remote models (non-deterministic)
  - `options`: Optional dict of provider-specific parameters (v2025.8.17+)
- **Returns:** Generated text string, or tuple if `return_logprobs=True`

#### `generate_iter(prompt: str, temperature: float = 0.0, model: Optional[str] = None, unsafe_mode: bool = False, options: Optional[Dict] = None) -> Iterator[str]`

Generate text iteratively, yielding tokens as they are produced.

```python
# Deterministic streaming
for token in steadytext.generate_iter("Tell me a story"):
    print(token, end="", flush=True)

# Creative streaming with temperature
for token in steadytext.generate_iter("Tell me a story", temperature=0.7):
    print(token, end="", flush=True)

# Streaming with remote models (v2025.8.17+)
for token in steadytext.generate_iter(
    "Explain reasoning",
    model="openai:gpt-5-mini",
    unsafe_mode=True
):
    print(token, end="", flush=True)
```

- **Parameters:**
  - `prompt`: Input text to generate from
  - `temperature`: Controls randomness (0.0 = deterministic, >0 = more random)
  - `model`: Optional remote model (e.g., "openai:gpt-4o-mini", "openai:gpt-5-mini")
  - `unsafe_mode`: Enable remote models (non-deterministic)
  - `options`: Optional dict of provider-specific parameters (v2025.8.17+)
- **Yields:** Text tokens/words as they are generated

### Embeddings

#### `embed(text_input: Union[str, List[str]], model: Optional[str] = None, unsafe_mode: bool = False) -> np.ndarray`

Create deterministic embeddings for text input, with optional remote provider support.

```python
# Local model (deterministic)
vec = steadytext.embed("Hello world")

# List of strings (averaged)
vecs = steadytext.embed(["Hello", "world"])

# Remote models (requires unsafe_mode)
vec = steadytext.embed(
    "Hello world",
    model="openai:text-embedding-3-small",
    unsafe_mode=True
)

vec = steadytext.embed(
    "Hello world", 
    model="voyageai:voyage-3-lite",
    unsafe_mode=True
)

vec = steadytext.embed(
    "Hello world",
    model="jina:jina-embeddings-v3",
    unsafe_mode=True
)
```

- **Parameters:**
  - `text_input`: String or list of strings to embed
  - `model`: Optional remote model (e.g., "openai:text-embedding-3-small", "voyageai:voyage-3-lite", "jina:jina-embeddings-v3")
  - `unsafe_mode`: Enable remote models (non-deterministic)
- **Returns:** 1024-dimensional L2-normalized numpy array (float32)

### Utilities

#### `preload_models(verbose: bool = False) -> None`

Preload models before first use.

```python
steadytext.preload_models()  # Silent
steadytext.preload_models(verbose=True)  # With progress
```

#### `get_model_cache_dir() -> str`

Get the path to the model cache directory.

```python
cache_dir = steadytext.get_model_cache_dir()
print(f"Models are stored in: {cache_dir}")
```

### Constants

```python
steadytext.DEFAULT_SEED  # 42
steadytext.GENERATION_MAX_NEW_TOKENS  # 512
steadytext.EMBEDDING_DIMENSION  # 1024
```

---

## 🤝 Contributing

Contributions are welcome!
See [CONTRIBUTING.md](CONTRIBUTING.md) for guidelines.

---

## 📄 License

* **Code:** MIT
* **Models:** MIT (Qwen3)

---
## 📈 What's New

### Structured Generation (v2.4.1+)
- **Native llama.cpp grammar support** for JSON, regex, and choice constraints
- **PostgreSQL extension integration** - all structured generation features in SQL
- **Async structured generation** functions for high-performance applications

### PostgreSQL Extension (v1.1.0+)
- **Production-ready SQL functions** for text generation and embeddings
- **Async operations** with queue-based background processing
- **AI summarization** aggregate functions with TimescaleDB support
- **Structured generation** in SQL (JSON schemas, regex patterns, choices)
- **Docker support** for easy deployment

### Document Reranking (v2.3.0+)
- **Reranking support** using Qwen3-Reranker-4B model for query-document relevance scoring
- **Python API** - `steadytext.rerank()` function with customizable task descriptions
- **CLI command** - `st rerank` for command-line reranking operations
- **PostgreSQL functions** - SQL functions for reranking with async support (PostgreSQL extension v1.3.0+)
- **Fallback scoring** - simple word overlap when model unavailable
- **Dedicated cache** - separate frecency cache for reranking results

### Daemon Architecture (v1.2.0+)
- **Persistent model serving** with ZeroMQ for 10-100x faster repeated calls
- **Automatic fallback** to direct model loading when daemon unavailable
- **Explicit startup required** - start daemon with `st daemon start`
- **Background operation** - daemon runs silently in the background

### Centralized Cache System
- **Unified caching** - consistent behavior between daemon and direct access
- **Thread-safe SQLite backend** for reliable concurrent access
- **Shared cache files** across all access modes
- **Cache integration** with daemon server for optimal performance

### Improved CLI Experience
- **Streaming by default** - see output as it's generated
- **Quiet by default** - clean output without informational messages
- **New pipe syntax** - `echo "prompt" | st` for better unix integration
- **Daemon management** - built-in commands for daemon lifecycle

---

## 🔧 Troubleshooting

### Installation Issues

#### llama-cpp-python Build Errors

If you encounter build errors related to llama-cpp-python, especially with the error "Failed to load model", this is likely due to the package requiring the inference-sh fork with specific CMAKE flags:

```bash
# Set required environment variables before installation
export FORCE_CMAKE=1
export CMAKE_ARGS="-DLLAVA_BUILD=OFF -DGGML_ACCELERATE=OFF -DGGML_BLAS=OFF -DGGML_CUDA=OFF -DGGML_BUILD_TESTS=OFF -DGGML_BUILD_EXAMPLES=OFF"

# Then install
pip install steadytext

# Or install from source
git clone https://github.com/julep-ai/steadytext.git
cd steadytext
uv sync  # or pip install -e .
```

#### Model Loading Issues

If you see "Failed to load model from file" errors:

1. **Try fallback models**: Set `STEADYTEXT_USE_FALLBACK_MODEL=true`
2. **Clear model cache**: `rm -rf ~/.cache/steadytext/models/`
3. **Check disk space**: Models require ~2-4GB per model

### Common Issues

- **"No module named 'llama_cpp'"**: Reinstall with the CMAKE flags above
- **Daemon connection refused**: Check if daemon is running with `st daemon status`
- **Slow first run**: Models download on first use (~2-4GB)

---

Built with ❤️ for developers tired of flaky AI tests.
