# 🦉 Trustwise Python SDK

A powerful, flexible SDK for evaluating AI-generated content with Trustwise's Safety and Alignment metrics. This SDK provides a clean path-based versioning approach that makes working with different API versions intuitive and explicit.

## 🔥 Features

- **Path-based API versioning**: Access specific API versions directly using intuitive paths (e.g., `sdk.safety.v3.faithfulness`)
- **Backward compatibility**: Maintain code simplicity with default version methods (e.g., `sdk.safety.faithfulness`)
- **Comprehensive evaluation metrics**: Assess faithfulness, relevancy, clarity, helpfulness, and more
- **LLM guardrails**: Set thresholds and automatically validate responses against multiple metrics
- **Framework integrations**: Ready-to-use callback handlers for LangChain and LlamaIndex
- **Detailed analytics**: Get comprehensive insights from every evaluation, including scores, facts, and analysis

## 📦 Installation

```bash
pip install trustwise
```

## 🚀 Quick Start

Get started with Trustwise in just a few lines of code:

```python
import os
from trustwise.sdk import TrustwiseSDK
from trustwise.sdk.config import TrustwiseConfig

# Set your API key
os.environ["TW_API_KEY"] = "your-api-key"

# Initialize the SDK
config = TrustwiseConfig()
trustwise = TrustwiseSDK(config)
```

### 🛡️ Safety Metrics

Evaluate content safety and faithfulness:

```python
# Example context
context = [{
    "node_text": "Paris is the capital of France.",
    "node_score": 0.95,
    "node_id": "doc:idx:1"
}]

# Evaluate faithfulness
result = trustwise.safety.v3.faithfulness.evaluate(
    query="What is the capital of France?",
    response="The capital of France is Paris.",
    context=context
)
print(f"Faithfulness score: {result['score']}")

# Evaluate PII detection
pii_result = trustwise.safety.v3.pii.evaluate(
    text="My email is john@example.com and my phone is 123-456-7890",
    allowlist=["john@example.com"]  # Allow specific PII
)
print(f"PII detection result: {pii_result}")
```

### 🎯 Alignment Metrics

Assess content quality and alignment:

```python
# Evaluate clarity
clarity_result = trustwise.alignment.v1.clarity.evaluate(
    query="What is the capital of France?",
    response="The capital of France is Paris."
)
print(f"Clarity score: {clarity_result['score']}")

# Evaluate tone
tone_result = trustwise.alignment.v1.tone.evaluate(
    response="The capital of France is Paris."
)
print(f"Tone result: {tone_result}")
```

### 💰 Performance Metrics

Evaluate cost and carbon emissions for your AI models:

```python
# Evaluate cost for OpenAI model
cost_result = trustwise.performance.v1.cost(
    model_name="gpt-3.5-turbo",
    model_type="LLM",
    model_provider="OpenAI",
    number_of_queries=5,
    total_prompt_tokens=950,
    total_completion_tokens=50
)
print(f"Cost per run: ${cost_result['cost_estimate_per_run']}")
print(f"Total project cost: ${cost_result['total_project_cost_estimate']}")

# Evaluate carbon emissions
carbon_result = trustwise.performance.v1.carbon(
    processor_name="NVIDIA A100",
    provider_name="AWS",
    provider_region="us-east-1",
    instance_type="p4d.24xlarge",
    average_latency=100
)
print(f"Carbon emissions: {carbon_result['carbon_emissions']} kg CO2e")
```

### 🚧 Guardrails

Create guardrails to automatically validate responses:

```python
# Create a multi-metric guardrail
guardrail = trustwise.guardrails(
    thresholds={
        "faithfulness": 0.8,
        "answer_relevancy": 0.7,
        "clarity": 0.7
    },
    block_on_failure=True
)

# Evaluate with multiple metrics
evaluation = guardrail.evaluate_response(
    query="What is the capital of France?",
    response="The capital of France is Paris.",
    context=context
)

print("Guardrail Evaluation:")
print(f"Passed all checks: {evaluation['passed']}")
print(f"Response blocked: {evaluation['blocked']}")
for metric, result in evaluation['results'].items():
    print(f" - {metric}: {result['passed']} (score: {result['result'].get('score')})")
```

## 🔐 API Key Setup

Get your API Key by logging in through Github -> [link](https://trustwise.ai)

The SDK requires an API key to authenticate requests. You can provide the API key in several ways:

```python
# Option 1: From environment variable (recommended)
import os
from trustwise.sdk import TrustwiseSDK
from trustwise.sdk.config import TrustwiseConfig

# Set environment variable (you can also set this in your shell or .env file)
os.environ["TW_API_KEY"] = "your-api-key"

# Initialize with configuration object
config = TrustwiseConfig()  # Will automatically use TW_API_KEY from environment
trustwise = TrustwiseSDK(config)

# Option 2: Direct initialization
config = TrustwiseConfig(api_key="your-api-key")
trustwise = TrustwiseSDK(config)

# Option 3: Mixed approach (direct values take precedence)
os.environ["TW_API_KEY"] = "env-key"  # Set environment variable
config = TrustwiseConfig(api_key="direct-key")  # Will override TW_API_KEY
trustwise = TrustwiseSDK(config)
```

## 📚 API Versioning

The SDK supports both explicit version paths and default version usage:

```python
# Using explicit version path
result = trustwise.safety.v3.faithfulness.evaluate(...)

# Using default version (backward compatibility)
result = trustwise.safety.faithfulness.evaluate(...)

# Set default version explicitly
trustwise.safety.set_version("v3")
```

## 📊 Available Metrics

### Safety Metrics (v3)
- Faithfulness
- Answer Relevancy
- Context Relevancy
- Summarization
- PII Detection
- Prompt Injection Detection

### Alignment Metrics (v1)
- Clarity
- Helpfulness
- Toxicity
- Tone
- Formality
- Simplicity
- Sensitivity

### Performance Metrics (v1)
- Cost Estimation
  - Support for multiple providers (OpenAI, Hugging Face, Azure)
  - Support for both LLM and Reranker models
  - Detailed cost breakdown per run and total project cost
- Carbon Emissions
  - Processor-specific emissions calculation
  - Provider and region-aware estimates
  - Latency-based calculations

## 📝 License

This project is licensed under the MIT License - see the LICENSE file for details.

## 🤝 Contributing

We welcome contributions! If you find a bug, have a feature request, or want to contribute code, please create an issue or submit a pull request.

### Git Hooks

This repository includes git hooks to ensure code quality. To install them:

```bash
# Make sure you're in the repository root
./scripts/install-hooks.sh
```

The hooks will run tests, linting, and documentation checks before each push to ensure code quality.