# Langfuse Lens Agent

[![PyPI version](https://img.shields.io/pypi/v/langfuse-lens-agent.svg)](https://pypi.org/project/langfuse-lens-agent/)
[![Python](https://img.shields.io/pypi/pyversions/langfuse-lens-agent.svg)](https://pypi.org/project/langfuse-lens-agent/)
[![License](https://img.shields.io/badge/license-MIT-green.svg)](./LICENSE)

Korean version: [README_ko.md](./README_ko.md)

**Langfuse Lens Agent** is an open-source LLM operations agent that turns Langfuse traces into actionable insights through an **Observe → Analyze → Experiment → Improve** cycle.

Ask natural-language questions in the web console, and the AI agent queries traces, diagnoses issues, suggests prompt improvements, and compares model outputs side by side.

---

## Features

### AI Agent (LangGraph)

- **Trace analytics** — filter by date, user, session, tags, metadata; summarize cost/latency/error trends
- **Observability diagnostics** — error classification, latency & cost trends, bottleneck detection
- **Prompt engineering** — A/B/C prompt rewrites with automatic scoring and a recommended variant
- **Model comparison** — run the same input across multiple LLMs and compare outputs
- **Trace replay** — re-run historical traces with a different model or prompt
- **A/B experiments** — dataset-driven variant execution + LLM-as-Judge scoring
- **Response ranking** — automatically compare and rank multiple model responses

### Web Console (FastAPI + Jinja2)

| Page | Purpose |
|------|---------|
| **Dashboard** | Server health, chat stats, audit events, LLM key status |
| **Agent Chat** | Multi-turn conversation for trace analysis, prompt tuning, evaluation |
| **Prompts** | Langfuse prompt CRUD, version control, label promotion |
| **Datasets** | Langfuse dataset management, trace-to-item import |
| **API Console** | Dynamic Langfuse OpenAPI explorer and executor |
| **Reports** | Daily / weekly / monthly activity reports with auto-scheduling |
| **Experiments** | Trace replay, A/B testing, evaluation, response comparison |
| **Settings** | LLM keys, Langfuse connections, multi-project, local inference |
| **Admin** | User & API token management, audit logs, system health |

### Additional

- **i18n** — Korean / English real-time switching
- **Dark / Light theme**
- **RBAC** — 6 roles: `viewer`, `analyst`, `editor`, `owner`, `ops`, `super_admin`
- **Multi-project** — per-user multiple Langfuse project connections (encrypted at rest)
- **Report scheduler** — asyncio-based auto-generation (daily/weekly/monthly, KST)
- **JSON / CSV export** — experiment result download
- **Dynamic OpenAPI tools** — agent can call Langfuse API endpoints directly
- **Python 3.11–3.14** — on 3.14, automatically uses Langfuse REST fallback

---

## Supported LLMs

| Provider | Examples |
|----------|----------|
| OpenAI | gpt-4o, gpt-4o-mini, gpt-4.1, o1, o3-mini |
| Anthropic | claude-sonnet-4-20250514, claude-haiku |
| Google | gemini-2.0-flash, gemini-2.5-pro |
| OpenRouter | deepseek/deepseek-chat-v3, meta-llama, etc. |
| Ollama (local) | llama3, phi3, mistral, etc. |
| vLLM / LM Studio | Local inference servers |

---

## Quick Start

### 1. Install

```bash
pip install langfuse-lens-agent
```

Or from source:

```bash
git clone https://github.com/BAEM1N/langfuse-lens-agent.git
cd langfuse-lens-agent
pip install -e .
```

### 2. Configure

```bash
cp .env.example .env
```

Minimal `.env`:

```bash
# At least one LLM key
OPENAI_API_KEY=sk-...

# Bootstrap admin (first run)
AGENT_BOOTSTRAP_ADMIN_TOKEN=your-secret-token
```

Langfuse connections are best configured **per-user in the web UI** (Settings > Langfuse Projects).

### 3. Run

```bash
langfuse-lens-web
```

Open `http://localhost:8010` and log in with the bootstrap token.

---

## Configuration

### .env key reference

```bash
# Langfuse (system-level fallback, optional)
LANGFUSE_BASE_URL=https://cloud.langfuse.com
LANGFUSE_PUBLIC_KEY=
LANGFUSE_SECRET_KEY=
LANGFUSE_ENVIRONMENT=production

# LLM provider keys
OPENAI_API_KEY=
ANTHROPIC_API_KEY=
GOOGLE_API_KEY=
OPENROUTER_API_KEY=
OPENROUTER_BASE_URL=https://openrouter.ai/api/v1

# Local inference (optional)
OLLAMA_BASE_URL=
VLLM_BASE_URL=
LMSTUDIO_BASE_URL=

# Agent behaviour
AGENT_LLM_TEMPERATURE=0.2
AGENT_LLM_MAX_WORKERS=3
AGENT_LLM_COMPARISON_MODELS=openai:gpt-4o,anthropic:claude-sonnet-4-20250514

# Security
AGENT_BOOTSTRAP_ADMIN_TOKEN=
AGENT_DATA_ENCRYPTION_KEY=          # Fernet key for credential encryption
AGENT_AUTH_ALLOW_SIGNUP=true
AGENT_LANGFUSE_CONTROL_ALLOW_MUTATION=false

# Report scheduler (KST)
REPORT_SCHEDULER_ENABLED=true
REPORT_SCHEDULER_DAILY_HOUR=0
REPORT_SCHEDULER_DAILY_MINUTE=0

# Admin token policy
AGENT_TOKEN_DEFAULT_EXPIRES_DAYS=30
AGENT_TOKEN_DEFAULT_ROTATE_BEFORE_DAYS=3
```

### Config.yaml (optional)

Supplements missing `.env` keys. Search order:

`./Config.yaml` → `./config/Config.yaml` → `AGENT_CONFIG_YAML_PATH`

---

## Docker

```bash
cp .env.example .env
# edit .env
docker compose up -d --build
curl http://localhost:8091/health
```

The container runs as a non-root user with a built-in health check.

---

## Project Structure

```
langfuse-lens-agent/
├── api/                        # FastAPI routers & middleware
│   ├── routers/                # auth, admin, langfuse, experiments, reports, settings, pages
│   ├── deps.py                 # Auth / authz dependency helpers
│   ├── models.py               # Pydantic request/response models
│   └── state.py                # RBAC role & permission definitions
├── src/                        # Core agent logic
│   ├── agent_core.py           # LangGraph agent builder
│   ├── report_generator.py     # Daily / weekly / monthly report generation
│   ├── report_scheduler.py     # Async report auto-scheduler
│   ├── user_store.py           # SQLite store (users, chats, reports, experiments)
│   ├── tools/                  # Agent tools
│   │   ├── analysis.py         # Trace analysis & model comparison
│   │   ├── experiment_tools.py # Replay, A/B, evaluation, ranking
│   │   ├── langfuse_fetch.py   # Langfuse data fetching
│   │   └── openapi_registry.py # Dynamic OpenAPI tool generation
│   └── web_entry.py            # CLI entry point
├── templates/                  # Jinja2 HTML templates (11 pages)
├── static/                     # CSS, JS, locales (en/ko)
├── config/                     # Config.yaml, tool presets, runtime profiles
├── skills/                     # Agent skill references
├── tests/                      # 88 unit tests across 9 files
├── Dockerfile
├── docker-compose.yml
└── pyproject.toml
```

---

## Security

| Feature | Detail |
|---------|--------|
| **Authentication** | Bearer token (session-based) + bootstrap admin token |
| **RBAC** | 6 hierarchical roles with scope-based permissions |
| **Encryption** | Fernet encryption for user Langfuse credentials at rest |
| **Token management** | Expiry, auto-rotation, revocation via admin API |
| **Audit logging** | All admin and write operations logged to SQLite |
| **Timing-safe comparison** | `secrets.compare_digest` for token validation |
| **Input validation** | Pydantic max_length, date parsing, field validators |
| **Docker** | Non-root user, health check, .dockerignore for secrets |

---

## API Overview

| Method | Path | Description |
|--------|------|-------------|
| POST | `/api/auth/login` | Log in |
| POST | `/api/auth/register` | Register |
| GET | `/api/auth/me` | Current user info |
| POST | `/api/chat` | Agent chat message |
| GET | `/api/chat/threads` | Chat thread list |
| GET | `/api/reports` | Report list |
| POST | `/api/reports/generate` | Generate report (daily/weekly/monthly) |
| GET | `/api/experiments` | Experiment list |
| POST | `/api/experiments` | Create experiment |
| GET | `/api/experiments/{id}/export` | Export as JSON/CSV |
| GET | `/api/langfuse/prompts` | Prompt list |
| GET | `/api/langfuse/datasets` | Dataset list |
| POST | `/api/langfuse/openapi/execute` | Execute Langfuse API operation |
| GET | `/api/langfuse/tools/catalog` | Agent tool catalog |
| GET | `/api/admin/users` | User management |
| GET | `/api/admin/tokens` | API token management |
| GET | `/api/admin/audit-logs` | Audit log list |
| GET | `/api/admin/system/health` | System health |

Full API is browsable in the **API Console** page of the web UI.

---

## Development

```bash
# Install dev dependencies
pip install -e ".[dev]"

# Run tests
python -m pytest tests/ -v

# Dev server (hot reload)
langfuse-lens-web --reload
```

---

## License

MIT — [LICENSE](./LICENSE)

---

## Contributing

Issues and PRs are welcome: [GitHub Issues](https://github.com/BAEM1N/langfuse-lens-agent/issues)
