# Phoenix (Arize AI) - Deep Dive

> **TL;DR**: Phoenix is an open-source AI observability platform (8,100+ stars) from Arize AI that lets you trace, evaluate, and debug LLM applications. It's the "APM for AI" - think DataDog or New Relic but for AI workflows. Built on OpenTelemetry, it captures traces/spans for every LLM call, provides built-in evaluators, and works with any LLM provider. Apache 2.0 licensed.

---

## What Is Phoenix?

**Phoenix** is an observability library designed specifically for AI applications. It helps you understand what your LLM-powered applications are doing, why they're failing, and how to improve them.

### The Problem It Solves

```
┌─────────────────────────────────────────────────────────────────────┐
│                    THE AI OBSERVABILITY PROBLEM                      │
├─────────────────────────────────────────────────────────────────────┤
│                                                                     │
│   TRADITIONAL APPS:                                                 │
│   ├── Deterministic → Same input = same output                     │
│   ├── Debuggable → Stack traces show exactly what happened         │
│   ├── Measurable → Response time, error rate, throughput           │
│   └── Testable → Unit tests catch regressions                      │
│                                                                     │
│   LLM APPS:                                                         │
│   ├── Non-deterministic → Same input = different output            │
│   ├── Black box → Why did it say that?                             │
│   ├── Quality varies → "Correct" is subjective                     │
│   └── Complex chains → RAG, agents, tools = many steps             │
│                                                                     │
│   PHOENIX SOLUTION:                                                 │
│   ├── Tracing → See every step in multi-step workflows             │
│   ├── Evaluation → Measure quality with LLM-as-judge               │
│   ├── Datasets → Create test sets from production data             │
│   └── Experiments → A/B test prompts and models                    │
│                                                                     │
└─────────────────────────────────────────────────────────────────────┘
```

---

## Quick Stats

| Metric | Value |
|--------|-------|
| **GitHub Stars** | 8,100+ |
| **Forks** | 600+ |
| **Contributors** | 100+ |
| **License** | Apache 2.0 |
| **Created By** | Arize AI |
| **Standard** | OpenTelemetry (OTEL) |

---

## Key Features

### 1. Tracing

Capture detailed traces of every LLM interaction:

```
┌─────────────────────────────────────────────────────────────────────┐
│                    PHOENIX TRACING                                   │
├─────────────────────────────────────────────────────────────────────┤
│                                                                     │
│   USER QUERY: "What's the weather in NYC?"                          │
│                              ↓                                      │
│   TRACE CAPTURES:                                                   │
│   ├── Span 1: Query received                                        │
│   │   └── Timestamp, metadata                                       │
│   ├── Span 2: RAG retrieval                                         │
│   │   └── Documents retrieved, embeddings, latency                 │
│   ├── Span 3: LLM call                                              │
│   │   └── Prompt, response, tokens, cost, latency                  │
│   ├── Span 4: Tool execution                                        │
│   │   └── Weather API call, result                                 │
│   └── Span 5: Final response                                        │
│       └── Output, total duration                                   │
│                                                                     │
│   VISIBILITY: See inside every step of complex AI workflows        │
│                                                                     │
└─────────────────────────────────────────────────────────────────────┘
```

### 2. Evaluation

Built-in evaluators to measure quality:

| Evaluator Type | What It Measures |
|----------------|------------------|
| **Hallucination** | Is the response grounded in provided context? |
| **QA Correctness** | Does the answer match expected output? |
| **Relevance** | Is the response relevant to the query? |
| **Toxicity** | Does the response contain harmful content? |
| **Summarization** | Is the summary accurate and complete? |
| **Code Generation** | Does the code compile and work? |
| **Custom** | Define your own evaluation criteria |

### 3. Datasets & Experiments

```
┌─────────────────────────────────────────────────────────────────────┐
│                    DATASETS & EXPERIMENTS                            │
├─────────────────────────────────────────────────────────────────────┤
│                                                                     │
│   1. CREATE DATASET                                                 │
│      ├── From production traces (real user queries)                │
│      ├── From manual examples                                       │
│      └── From synthetic generation                                  │
│                                                                     │
│   2. RUN EXPERIMENT                                                 │
│      ├── Test different prompts                                     │
│      ├── Test different models                                      │
│      ├── Test different parameters                                  │
│      └── Compare results side-by-side                              │
│                                                                     │
│   3. ANALYZE RESULTS                                                │
│      ├── Aggregate metrics                                          │
│      ├── Drill into failures                                        │
│      └── Identify patterns                                          │
│                                                                     │
└─────────────────────────────────────────────────────────────────────┘
```

### 4. OpenTelemetry Native

Phoenix is built on OpenTelemetry, the industry standard for observability:

| Benefit | Description |
|---------|-------------|
| **Standard Format** | Compatible with existing observability tools |
| **Auto-instrumentation** | Drop-in tracing for popular frameworks |
| **Vendor Neutral** | No lock-in, export anywhere |
| **Rich Context** | Attributes, events, links between spans |

---

## Supported Integrations

### LLM Providers

| Provider | Auto-Instrumentation |
|----------|---------------------|
| **OpenAI** | ✅ |
| **Anthropic** | ✅ |
| **Google/Vertex** | ✅ |
| **Azure OpenAI** | ✅ |
| **AWS Bedrock** | ✅ |
| **Ollama** | ✅ |
| **Mistral** | ✅ |

### Frameworks

| Framework | Support |
|-----------|---------|
| **LangChain** | ✅ Auto-instrumentation |
| **LlamaIndex** | ✅ Auto-instrumentation |
| **CrewAI** | ✅ Auto-instrumentation |
| **DSPy** | ✅ Auto-instrumentation |
| **Haystack** | ✅ Auto-instrumentation |
| **AutoGen** | ✅ Auto-instrumentation |

### Vector Databases

| Database | Support |
|----------|---------|
| **Pinecone** | ✅ |
| **ChromaDB** | ✅ |
| **Weaviate** | ✅ |
| **Milvus** | ✅ |
| **Qdrant** | ✅ |

---

## Installation & Usage

### Quick Start

```bash
# Install
pip install arize-phoenix

# Launch UI
python -m phoenix.server.main serve

# Open browser at localhost:6006
```

### Basic Instrumentation

```python
import phoenix as px
from openinference.instrumentation.openai import OpenAIInstrumentor

# Start Phoenix
px.launch_app()

# Instrument OpenAI (one line!)
OpenAIInstrumentor().instrument()

# Now all OpenAI calls are traced automatically
import openai
client = openai.OpenAI()
response = client.chat.completions.create(
    model="gpt-4",
    messages=[{"role": "user", "content": "Hello!"}]
)
# → Trace automatically captured in Phoenix UI
```

### Running Evaluations

```python
from phoenix.evals import HallucinationEvaluator, run_evals

# Create evaluator
evaluator = HallucinationEvaluator(model="gpt-4")

# Run on traces
results = run_evals(
    traces=my_traces,
    evaluators=[evaluator]
)

# Analyze results
print(f"Hallucination rate: {results.hallucination_rate}")
```

---

## Architecture

```
┌─────────────────────────────────────────────────────────────────────┐
│                    PHOENIX ARCHITECTURE                              │
├─────────────────────────────────────────────────────────────────────┤
│                                                                     │
│   YOUR APPLICATION                                                  │
│        ↓                                                            │
│   Instrumentation Layer (OpenInference)                             │
│   ├── Auto-instruments LLM calls                                    │
│   ├── Captures prompts, responses, tokens                          │
│   └── Records latency, errors                                       │
│        ↓                                                            │
│   OpenTelemetry Collector                                           │
│   ├── Processes spans                                               │
│   └── Routes to Phoenix                                             │
│        ↓                                                            │
│   Phoenix Server                                                    │
│   ├── Stores traces                                                 │
│   ├── Runs evaluations                                              │
│   └── Serves UI                                                     │
│        ↓                                                            │
│   Phoenix UI (localhost:6006)                                       │
│   ├── Trace explorer                                                │
│   ├── Evaluation dashboards                                         │
│   ├── Dataset management                                            │
│   └── Experiment comparison                                         │
│                                                                     │
└─────────────────────────────────────────────────────────────────────┘
```

---

## Deployment Options

### Local Development

```bash
# Single command
python -m phoenix.server.main serve
```

### Docker

```bash
docker run -p 6006:6006 arizephoenix/phoenix
```

### Cloud (Arize Platform)

For production, Arize offers a managed cloud version with:
- Persistent storage
- Team collaboration
- Advanced analytics
- Enterprise security

---

## Comparison with Alternatives

| Feature | Phoenix | LangSmith | Helicone | Weights & Biases |
|---------|---------|-----------|----------|------------------|
| **Open Source** | ✅ Apache 2.0 | Partial | No | Partial |
| **Self-Hosted** | ✅ | ✅ | ✅ | ✅ |
| **OpenTelemetry** | ✅ Native | ❌ | ❌ | ❌ |
| **LLM Evals** | ✅ Built-in | ✅ | Limited | Limited |
| **Framework Agnostic** | ✅ | LangChain focus | ✅ | ✅ |
| **Free Tier** | ✅ Unlimited local | Limited | Limited | Limited |

---

## Use Cases

### 1. Debugging RAG Applications

```
Problem: "Why is my RAG system giving wrong answers?"

Phoenix helps you:
├── See which documents were retrieved
├── Check if context was relevant
├── Evaluate if LLM hallucinated
└── Compare before/after prompt changes
```

### 2. Optimizing Prompts

```
Problem: "Which prompt performs better?"

Phoenix helps you:
├── Create dataset from production
├── Run A/B experiments
├── Measure quality metrics
└── Identify failure patterns
```

### 3. Production Monitoring

```
Problem: "Is my AI app healthy?"

Phoenix helps you:
├── Track latency trends
├── Monitor error rates
├── Detect quality degradation
└── Alert on anomalies
```

### 4. Cost Management

```
Problem: "Why are my LLM costs so high?"

Phoenix helps you:
├── Track tokens per request
├── Identify expensive patterns
├── Compare model costs
└── Optimize for efficiency
```

---

## Who Is Arize AI?

| Fact | Detail |
|------|--------|
| **Founded** | 2020 |
| **Headquarters** | San Francisco |
| **Focus** | ML/AI Observability |
| **Funding** | $62M+ |
| **Core Product** | Arize Platform (enterprise) |
| **Open Source** | Phoenix |

---

## How This Connects to Our Other Deep Dives

| Deep Dive | Connection |
|-----------|------------|
| **Awesome LLMOps** | Phoenix is a key LLMOps monitoring tool |
| **lm-evaluation-harness** | Different focus (benchmarks vs. production monitoring) |
| **Promptfoo** | Complementary (testing vs. observability) |
| **LLM Guard** | Could trace LLM Guard evaluations in Phoenix |

---

## Key Links

| Resource | URL |
|----------|-----|
| GitHub | https://github.com/Arize-ai/phoenix |
| Documentation | https://docs.arize.com/phoenix |
| Arize Platform | https://arize.com |
| OpenInference | https://github.com/Arize-ai/openinference |

---

## Bottom Line

Phoenix is the **open-source standard for AI observability**.

**Why it matters**:
- See inside your LLM applications
- Measure quality with built-in evaluators
- Debug failures with detailed traces
- Experiment with prompts and models
- Built on OpenTelemetry (no vendor lock-in)

**The key insight**: You can't improve what you can't measure. Phoenix gives you the visibility into LLM applications that traditional APM tools provide for regular software - but designed specifically for the unique challenges of AI systems.

For our repository, Phoenix represents the monitoring layer that every production AI application needs. It's the bridge between "it works in my notebook" and "it works reliably in production."

---

*Document created: January 2026*
*Sources: [GitHub - Arize-ai/phoenix](https://github.com/Arize-ai/phoenix), [Phoenix Documentation](https://docs.arize.com/phoenix), [Arize AI](https://arize.com)*
