Metadata-Version: 2.4
Name: detra
Version: 0.1.0
Summary: End-to-end LLM observability for vertical AI applications with Datadog
Project-URL: Homepage, https://github.com/adc77/detra
Project-URL: Repository, https://github.com/adc77/detra
Project-URL: Documentation, https://github.com/adc77/detra#readme
Project-URL: Issues, https://github.com/adc77/detra/issues
Author: detra Contributors
License: MIT
License-File: LICENSE
Keywords: ai,datadog,evaluation,llm,ml,monitoring,observability,security,tracing,vertical-ai
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: Topic :: System :: Logging
Classifier: Topic :: System :: Monitoring
Requires-Python: >=3.11
Requires-Dist: certifi>=2024.2.2
Requires-Dist: datadog-api-client>=2.20.0
Requires-Dist: datadog>=0.49.0
Requires-Dist: ddtrace>=2.10.0
Requires-Dist: google-cloud-aiplatform>=1.40.0
Requires-Dist: google-genai>=0.2.0
Requires-Dist: httpx>=0.26.0
Requires-Dist: pydantic-settings>=2.1.0
Requires-Dist: pydantic>=2.5.0
Requires-Dist: python-dotenv>=1.0.0
Requires-Dist: pyyaml>=6.0.1
Requires-Dist: structlog>=24.1.0
Requires-Dist: tenacity>=8.2.0
Requires-Dist: tiktoken>=0.5.0
Provides-Extra: all
Requires-Dist: dspy-ai>=2.4.0; extra == 'all'
Requires-Dist: mypy>=1.8.0; extra == 'all'
Requires-Dist: pytest-asyncio>=0.23.0; extra == 'all'
Requires-Dist: pytest-cov>=4.1.0; extra == 'all'
Requires-Dist: pytest>=8.0.0; extra == 'all'
Requires-Dist: ruff>=0.2.0; extra == 'all'
Provides-Extra: dev
Requires-Dist: mypy>=1.8.0; extra == 'dev'
Requires-Dist: pytest-asyncio>=0.23.0; extra == 'dev'
Requires-Dist: pytest-cov>=4.1.0; extra == 'dev'
Requires-Dist: pytest>=8.0.0; extra == 'dev'
Requires-Dist: ruff>=0.2.0; extra == 'dev'
Provides-Extra: optimization
Requires-Dist: dspy-ai>=2.4.0; extra == 'optimization'
Provides-Extra: server
Requires-Dist: fastapi>=0.104.0; extra == 'server'
Requires-Dist: uvicorn[standard]>=0.24.0; extra == 'server'
Description-Content-Type: text/markdown

# detra

**End-to-end LLM Observability for Vertical AI Applications with Datadog Integration**

detra is a comprehensive Python framework for monitoring, evaluating, and securing LLM applications. It provides automatic tracing, Gemini-powered evaluation, security scanning, alerting, and full integration with Datadog's LLM Observability platform.

## Features

- **Automatic Tracing**: Decorator-based tracing that captures inputs, outputs, and metadata
- **LLM Evaluation**: Gemini-powered evaluation to check adherence to expected behaviors
- **Security Scanning**: Detection of PII, prompt injection, and sensitive content
- **Alerting**: Integration with Slack, PagerDuty, and custom webhooks
- **Datadog Integration**: Full integration with Datadog LLM Observability, metrics, events, monitors, and dashboards
- **Incident Management**: Automatic incident creation for critical issues

## Installation

```bash
pip install detra

# With optional dependencies
pip install detra[server]      # FastAPI/uvicorn support
pip install detra[dev]        # Development tools
pip install detra[optimization]  # DSPy optimization
pip install detra[all]        # All optional dependencies
```

## Quick Start

### 1. Install and Configure

```bash
# Install
pip install detra

# Set environment variables
export DD_API_KEY=your_datadog_api_key
export DD_APP_KEY=your_datadog_app_key
export GOOGLE_API_KEY=your_google_api_key
```

### 2. Create Configuration

Create `detra.yaml`:

```yaml
app_name: my-llm-app
datadog:
  api_key: ${DD_API_KEY}
  app_key: ${DD_APP_KEY}
  site: datadoghq.com
  service: my-service

gemini:
  api_key: ${GOOGLE_API_KEY}
  model: gemini-2.5-flash

nodes:
  extract_entities:
    expected_behaviors:
      - "Must return valid JSON"
      - "Must extract party names accurately"
    unexpected_behaviors:
      - "Hallucinated party names"
      - "Fabricated dates"
    adherence_threshold: 0.85
```

### 3. Use in Your Code

```python
import detra

# Initialize
vg = detra.init("detra.yaml")

# Decorate your LLM functions
@vg.trace("extract_entities")
async def extract_entities(document: str):
    # Your LLM call here
    result = await llm.complete(prompt)
    return result

# Use the function - tracing and evaluation happen automatically
result = await extract_entities("Contract text...")
```

### 4. Setup Monitoring

```python
# Create monitors and dashboard
setup_results = await vg.setup_all(slack_channel="#llm-alerts")
print(f"Dashboard URL: {setup_results['dashboard']['url']}")
```

## Usage Examples

### Basic Tracing

```python
import detra

vg = detra.init("detra.yaml")

@vg.trace("summarize")
async def summarize(text: str):
    return await llm.summarize(text)
```

### Different Trace Types

```python
@vg.workflow("document_processing")  # Workflow trace
@vg.llm("llm_call")                  # LLM call trace
@vg.task("data_extraction")         # Task trace
@vg.agent("agent_name")              # Agent trace
```

### Manual Evaluation

```python
result = await vg.evaluate(
    node_name="extract_entities",
    input_data="Document text",
    output_data={"entities": [...]},
)
print(f"Score: {result.score}, Flagged: {result.flagged}")
```

### Security Scanning

```yaml
nodes:
  my_node:
    security_checks:
      - pii_detection
      - prompt_injection
      - sensitive_content
```

### Alerting

```yaml
integrations:
  slack:
    enabled: true
    webhook_url: ${SLACK_WEBHOOK_URL}
    notify_on:
      - flag_raised
      - incident_created
      - security_issue
```

## Evaluation Pipeline

1. **Rule-Based Checks**: Fast validation (JSON format, empty output, etc.)
2. **Security Scans**: PII detection, prompt injection scanning
3. **LLM Evaluation**: Gemini-based semantic evaluation of behaviors
4. **Flagging**: Automatic flagging when thresholds are breached
5. **Alerting**: Notifications sent based on severity

## Requirements

- Datadog account with API key and Application key
- Google API key for Gemini evaluation (optional but recommended)

## Documentation

- **Github**: See [GitHub Repository](https://github.com/adc77/detra)
- **Examples**: [examples/](https://github.com/adc77/detra/tree/main/examples)

## License

MIT License
