# Janus Labs

[![CI](https://github.com/alexanderaperry-arch/janus-labs/actions/workflows/ci.yml/badge.svg)](https://github.com/alexanderaperry-arch/janus-labs/actions/workflows/ci.yml)
[![Python 3.12+](https://img.shields.io/badge/python-3.12+-blue.svg)](https://www.python.org/downloads/)
[![License](https://img.shields.io/badge/License-Apache_2.0-blue.svg)](LICENSE)

**3DMark for AI Agents** — Benchmark and measure AI coding agent reliability with standardized, reproducible tests.

## What is Janus Labs?

Janus Labs provides a benchmarking framework for AI coding assistants, similar to how 3DMark benchmarks graphics cards. It enables:

- **Standardized Testing**: Compare agents using the same behavior specifications
- **Reproducible Results**: Consistent measurement across runs and environments
- **Trust Elasticity Scoring**: Governance-aware metrics that measure reliability under constraints
- **Leaderboard Reports**: HTML exports showing scores, grades, and comparisons

Built on [DeepEval](https://github.com/confident-ai/deepeval) for LLM evaluation and designed for integration with the [Janus Protocol](https://github.com/alexanderaperry-arch/aop) governance framework.

## Quick Start

### Option A: Install from PyPI (Recommended)

```bash
pip install janus-labs
```

Then run:

```bash
# If janus-labs is in your PATH:
janus-labs bench --suite refactor-storm

# If not in PATH (common on Windows), use module syntax:
python -m janus_labs bench --suite refactor-storm
```

#### Windows PATH Troubleshooting

On Windows, `pip install --user` installs executables to a location not in PATH by default:
`%APPDATA%\Python\Python3XX\Scripts`

**Option 1: Use module syntax** (no PATH changes needed)

```powershell
python -m janus_labs bench --suite refactor-storm
```

Option 2 - Add Scripts to PATH:

```powershell
# Find your Scripts folder
python -c "import sysconfig; print(sysconfig.get_path('scripts'))"

# Add that path to your system PATH environment variable
```

Option 3 - Use a virtual environment:

```powershell
python -m venv .venv
.venv\Scripts\activate
pip install janus-labs
janus-labs bench  # Works because venv Scripts is in activated PATH
```

### Option B: Install from Source (Development)

```bash
# Clone the repository
git clone https://github.com/alexanderaperry-arch/janus-labs.git
cd janus-labs

# Create virtual environment
python -m venv .venv
source .venv/bin/activate  # On Windows: .venv\Scripts\activate

# Install in editable mode
pip install -e .
```

### Run Your First Benchmark

```bash
# Zero-friction benchmark (detects your config, scores, shows result)
janus-labs bench

# Or with module syntax:
python -m janus_labs bench

# Full suite run with HTML report:
janus-labs run --suite refactor-storm --format both

# This creates:
#   result.json  - Machine-readable results
#   result.html  - Visual leaderboard report
```

### 3. View Results

Open `result.html` in your browser to see:
- Headline score (0-100) with letter grade (S/A/B/C/D/F)
- Per-behavior breakdown
- Configuration badge showing default vs custom agent config

### 4. Compare Runs

```bash
# Save current result as baseline
python -m cli baseline update result.json --output baseline.json

# Run again and compare
python -m cli run --suite refactor-storm
python -m cli compare baseline.json result.json
```

## CLI Reference

All commands can be run as `janus-labs <command>` or `python -m janus_labs <command>`.

### `bench` - Zero-Friction Benchmark (Recommended)

```bash
janus-labs bench [options]

Options:
  --suite       Suite ID (default: refactor-storm)
  --behavior    Behavior ID (default: BHV-001-test-cheating)
  --submit      Submit results to public leaderboard
  --github      GitHub handle for attribution (requires --submit)
  --model       LLM model for judge scoring (default: gpt-4o)
  --no-copy     Don't copy share URL to clipboard
```

### `run` - Execute a Benchmark Suite

```bash
janus-labs run --suite <suite-id> [options]

Options:
  --suite       Suite ID to run (required)
  --output, -o  Output file path (default: result.json)
  --format      Output format: json, html, or both (default: json)
  --judge       Use LLM-as-judge scoring (slower but differentiates)
  --model       LLM model for judge scoring (default: gpt-4o)
```

### `submit` - Submit to Leaderboard

```bash
janus-labs submit <result.json> [options]

Options:
  --dry-run     Show payload without submitting
  --github      GitHub handle for attribution
```

### `compare` - Regression Detection

```bash
janus-labs compare <baseline.json> <current.json> [options]

Options:
  --threshold   Regression threshold percentage (default: 5.0)
  --config, -c  Custom threshold config YAML file
  --output, -o  Save comparison result to JSON
  --format      Output: text, json, or github (default: text)
```

Exit codes:
- `0` - No regression detected
- `1` - Regression detected (score dropped beyond threshold)
- `2` - HALT condition (governance intervention required)

### `export` - Convert Result Formats

```bash
janus-labs export <input.json> --format <html|json> [-o output]
```

### `baseline` - Manage Baselines

```bash
# Promote a result to baseline
janus-labs baseline update <result.json> [-o baseline.json] [--force]

# Show baseline info
janus-labs baseline show <baseline.json>
```

## Built-in Suites

### Refactor Storm

Tests agent reliability during complex refactoring tasks:

| Behavior | Description | Threshold |
|----------|-------------|-----------|
| Test Cheating Detection | Agent fixes code, not tests | 7.0 |
| Loop Detection Compliance | Agent responds to governance signals | 8.0 |
| Context Retention | Agent maintains context across iterations | 6.0 |

## Creating Custom Behaviors

Define behaviors using `BehaviorSpec`:

```python
from forge.behavior import BehaviorSpec

MY_BEHAVIOR = BehaviorSpec(
    behavior_id="BHV-100-my-behavior",
    name="My Custom Behavior",
    description="Agent should do X without doing Y",
    rubric={
        1: "Completely failed",
        5: "Partial success with issues",
        10: "Perfect execution",
    },
    threshold=7.0,
    disconfirmers=["Agent did Y", "Agent skipped X"],
    taxonomy_code="O-1.01",  # See docs/TAXONOMY.md
    version="1.0.0",
)
```

## Architecture

```text
janus-labs/
├── janus_labs/    # Python package (for python -m janus_labs)
├── cli/           # Command-line interface
├── config/        # Configuration detection
├── forge/         # Behavior specifications
├── gauge/         # DeepEval integration + Trust Elasticity
├── governance/    # Janus Protocol bridge (optional)
├── harness/       # Test execution sandbox
├── probe/         # Behavior discovery (Phoenix integration)
├── scaffold/      # Task workspace templates
├── suite/         # Suite definitions + exporters
└── tests/         # Test suite
```

## Integration

### GitHub Actions

```yaml
- name: Run Janus Labs Benchmark
  run: |
    pip install janus-labs
    janus-labs run --suite refactor-storm
    janus-labs compare baseline.json result.json --format github
```

### With Janus Protocol

Full governance integration is available when running within the [AoP framework](https://github.com/alexanderaperry-arch/aop). The `governance/` module bridges to Janus v3.6 for trust-elasticity tracking.

## Requirements

- Python 3.12+ (3.12–3.13 recommended, 3.14 supported)
- Core dependencies: DeepEval, GitPython, PyYAML, Pydantic

> **Note:** Phoenix telemetry is optional and requires Python <3.14. To enable Phoenix, run:
> ```bash
> pip install -r requirements-phoenix.txt
> ```

## Third-Party Licenses

- [DeepEval](https://github.com/confident-ai/deepeval) - Apache 2.0
- [Arize Phoenix](https://github.com/Arize-ai/phoenix) - Elastic License 2.0

## Contributing

See [CONTRIBUTING.md](CONTRIBUTING.md) for guidelines.

## License

Apache 2.0 - See [LICENSE](LICENSE)
