Metadata-Version: 2.4
Name: trustwise
Version: 0.1.0a6
Summary: Trustwise SDK for evaluating AI-generated content
Project-URL: Homepage, https://trustwise.ai
Project-URL: Documentation, https://trustwiseai.github.io/tw-docs/docs/intro
Project-URL: Repository, https://github.com/trustwise/trustwise
Project-URL: Issues, https://github.com/trustwise/trustwise/issues
Author-email: Mayank Chutani <mayank@trustwise.ai>
License-Expression: MIT
Keywords: ai,alignment,evaluation,metrics,safety
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Requires-Python: >=3.11
Requires-Dist: importlib-metadata>=6.0.0; python_version < '3.8'
Requires-Dist: pydantic>=2.11.0
Requires-Dist: python-dotenv>=1.0.0
Requires-Dist: requests>=2.31.0
Requires-Dist: urllib3>=2.4.0
Provides-Extra: dev
Requires-Dist: docutils>=0.21.0; extra == 'dev'
Requires-Dist: mypy>=1.15.0; extra == 'dev'
Requires-Dist: pytest-cov>=4.0.0; extra == 'dev'
Requires-Dist: pytest-mock>=3.10.0; extra == 'dev'
Requires-Dist: pytest>=7.0.0; extra == 'dev'
Requires-Dist: ruff>=0.11.0; extra == 'dev'
Requires-Dist: sphinx-autodoc-typehints>=2.0.0; extra == 'dev'
Requires-Dist: sphinx-rtd-theme>=3.0.2; extra == 'dev'
Requires-Dist: sphinx>=8.2.0; extra == 'dev'
Requires-Dist: tox>=4.0.0; extra == 'dev'
Description-Content-Type: text/markdown

# 🦉 Trustwise Python SDK

A powerful, flexible SDK for evaluating AI-generated content with Trustwise's Safety and Alignment metrics. This SDK provides a clean path-based versioning approach that makes working with different API versions intuitive and explicit.

## 🔥 Features

- **Path-based API versioning**: Access specific API versions directly using intuitive paths (e.g., `sdk.safety.v3.faithfulness`)
- **Backward compatibility**: Maintain code simplicity with default version methods (e.g., `sdk.safety.faithfulness`)
- **Comprehensive evaluation metrics**: Assess faithfulness, relevancy, clarity, helpfulness, and more
- **LLM guardrails**: Set thresholds and automatically validate responses against multiple metrics
- **Framework integrations**: Ready-to-use callback handlers for LangChain and LlamaIndex
- **Detailed analytics**: Get comprehensive insights from every evaluation, including scores, facts, and analysis

## 📦 Installation

```bash
pip install trustwise
```

## 🚀 Quick Start

Get started with Trustwise in just a few lines of code:

```python
import os
from trustwise.sdk import TrustwiseSDK
from trustwise.sdk.config import TrustwiseConfig

# Set your API key
os.environ["TW_API_KEY"] = "your-api-key"

# Initialize the SDK
config = TrustwiseConfig()
trustwise = TrustwiseSDK(config)
```

### 🛡️ Safety Metrics

Evaluate content safety and faithfulness:

```python
# Example context
context = [{
    "node_text": "Paris is the capital of France.",
    "node_score": 0.95,
    "node_id": "doc:idx:1"
}]

# Evaluate faithfulness
result = trustwise.safety.v3.faithfulness.evaluate(
    query="What is the capital of France?",
    response="The capital of France is Paris.",
    context=context
)
print(f"Faithfulness score: {result.score}")

# Evaluate PII detection
pii_result = trustwise.safety.v3.pii.evaluate(
    text="My email is john@example.com and my phone is 123-456-7890",
    allowlist=["john@example.com"]  # Allow specific PII
)
print(f"PII detection result: {pii_result}")
```

### 🎯 Alignment Metrics

Assess content quality and alignment:

```python
# Evaluate clarity
clarity_result = trustwise.alignment.v1.clarity.evaluate(
    query="What is the capital of France?",
    response="The capital of France is Paris."
)
print(f"Clarity score: {clarity_result.score}")

# Evaluate tone
tone_result = trustwise.alignment.v1.tone.evaluate(
    response="The capital of France is Paris."
)
print(f"Tone result: {tone_result}")
```

### 💰 Performance Metrics

Evaluate cost and carbon emissions for your AI models:

```python
# Evaluate cost for OpenAI model
cost_result = trustwise.performance.v1.cost(
    model_name="gpt-3.5-turbo",
    model_type="LLM",
    model_provider="OpenAI",
    number_of_queries=5,
    total_prompt_tokens=950,
    total_completion_tokens=50
)
print(f"Cost per run: ${cost_result.cost_estimate_per_run}")
print(f"Total project cost: ${cost_result.total_project_cost_estimate}")

# Evaluate carbon emissions
carbon_result = trustwise.performance.v1.carbon(
    processor_name="NVIDIA A100",
    provider_name="AWS",
    provider_region="us-east-1",
    instance_type="p4d.24xlarge",
    average_latency=100
)
print(f"Carbon emissions: {carbon_result.carbon_emissions} kg CO2e")
```

### 🚧 Guardrails

Create guardrails to automatically validate responses:

```python
# Create a multi-metric guardrail
guardrail = trustwise.guardrails(
    thresholds={
        "faithfulness": 0.8,
        "answer_relevancy": 0.7,
        "clarity": 0.7
    },
    block_on_failure=True
)

# Evaluate with multiple metrics
evaluation = guardrail.evaluate_response(
    query="What is the capital of France?",
    response="The capital of France is Paris.",
    context=context
)

print("Guardrail Evaluation:", evaluation)
print("Guardrail Evaluation:", evaluation.to_json())
```

## 🔐 API Key Setup

Get your API Key by logging in through Github -> [link](https://trustwise.ai)

The SDK requires an API key to authenticate requests. You can provide the API key in several ways:

```python
# Option 1: From environment variable (recommended)
import os
from trustwise.sdk import TrustwiseSDK
from trustwise.sdk.config import TrustwiseConfig

# Set environment variable (you can also set this in your shell or .env file)
os.environ["TW_API_KEY"] = "your-api-key"

# Initialize with configuration object
config = TrustwiseConfig()  # Will automatically use TW_API_KEY from environment
trustwise = TrustwiseSDK(config)

# Option 2: Direct initialization
config = TrustwiseConfig(api_key="your-api-key")
trustwise = TrustwiseSDK(config)

# Option 3: Mixed approach (direct values take precedence)
os.environ["TW_API_KEY"] = "env-key"  # Set environment variable
config = TrustwiseConfig(api_key="direct-key")  # Will override TW_API_KEY
trustwise = TrustwiseSDK(config)
```

## 📚 API Versioning

The SDK supports both explicit version paths and default version usage:

```python
# Using explicit version path
result = trustwise.safety.v3.faithfulness.evaluate(...)

# Using default version (backward compatibility)
result = trustwise.safety.v3.faithfulness.evaluate(...)

## 📊 Available Metrics

### Safety Metrics (v3)
- Faithfulness
- Answer Relevancy
- Context Relevancy
- Summarization
- PII Detection
- Prompt Injection Detection

### Alignment Metrics (v1)
- Clarity
- Helpfulness
- Toxicity
- Tone
- Formality
- Simplicity
- Sensitivity

### Performance Metrics (v1)
- Cost Estimation
  - Support for multiple providers (OpenAI, Hugging Face, Azure)
  - Support for both LLM and Reranker models
  - Detailed cost breakdown per run and total project cost
- Carbon Emissions
  - Processor-specific emissions calculation
  - Provider and region-aware estimates
  - Latency-based calculations

## 📝 License

This project is licensed under the MIT License - see the LICENSE file for details.

## 🤝 Contributing

We welcome contributions! If you find a bug, have a feature request, or want to contribute code, please create an issue or submit a pull request.

### Git Hooks

This repository includes git hooks to ensure code quality. To install them:

```bash
# Make sure you're in the repository root
./scripts/install-hooks.sh
```

The hooks will run tests, linting, and documentation checks before each push to ensure code quality.