# Janus Labs

[![CI](https://github.com/alexanderaperry-arch/janus-labs/actions/workflows/ci.yml/badge.svg)](https://github.com/alexanderaperry-arch/janus-labs/actions/workflows/ci.yml)
[![Baselines](https://github.com/alexanderaperry-arch/janus-labs/actions/workflows/baseline.yml/badge.svg)](https://github.com/alexanderaperry-arch/janus-labs/actions/workflows/baseline.yml)
[![Python 3.12+](https://img.shields.io/badge/python-3.12+-blue.svg)](https://www.python.org/downloads/)
[![License](https://img.shields.io/badge/License-Apache_2.0-blue.svg)](LICENSE)

**3DMark for AI Agents**. Profile AI coding agents across two active axes: Code Quality and Error Resilience. Measure Instruction Resilience separately with `janus-labs diagnose` when you have configured and vanilla results to compare.

## What Janus Labs Does

Janus Labs benchmarks real coding-agent behavior on reproducible tasks. Instead of reducing everything to one opaque number, it runs a fixed suite of coding tasks and produces a capability profile that shows where an agent is strong, weak, or uneven.

- `refactor-storm` ships 4 built-in behaviors grouped into 2 active axes
- `janus-labs run` is the primary workflow for generating a full suite result
- Backend-hosted judging is available without a local API key, and `--mock` supports offline dry runs
- Results can be compared against bundled baselines and submitted to a public leaderboard

Public leaderboard: <https://fulfilling-courtesy-production-9c2c.up.railway.app>

## Quick Start

### Install

```bash
pip install janus-labs
janus-labs --version
```

If `janus-labs` is not on your `PATH`, use:

```bash
python -m janus_labs --version
```

### Run The Full Suite

The primary workflow is `run`: execute the full suite, save one result file, then submit it.

```bash
# Run the full suite offline with mock scoring
janus-labs run --suite refactor-storm --mock -o result.json

# Or use backend-hosted judging (no API key needed)
janus-labs run --suite refactor-storm -o result.json

# Suite alias for the same workflow
janus-labs refactor-storm -o result.json

# Submit to the leaderboard
janus-labs submit result.json --github your-handle
```

On Windows, replace `janus-labs` with `python -m janus_labs` if the launcher is not available in `PATH`.

### Alternative: Single-Behavior Manual Workflow

Use `init -> status -> score` when you want to hand a single behavior workspace to an external agent and inspect the repo diff yourself.

```bash
# Generate one workspace per behavior
janus-labs init --suite refactor-storm --output ./janus-task

# Let your agent work in one behavior directory
cd janus-task/BHV-001-test-cheating

# Inspect the workspace and score the finished result
janus-labs status --workspace .
janus-labs score --workspace . --output result.json
```

### Compare Against Baselines

```bash
# Generate capability profiles from bundled baseline results
janus-labs profile --baselines-dir data/baselines

# Compare your result against an auto-selected vanilla baseline
janus-labs compare result.json --auto-baseline

# Measure optional instruction resilience
janus-labs diagnose result.json
```

### Install From Source

```bash
git clone https://github.com/alexanderaperry-arch/janus-labs.git
cd janus-labs
python -m venv .venv
source .venv/bin/activate  # Windows: .venv\Scripts\activate
pip install -e .
```

## CLI Reference

All commands can be run as:

- `janus-labs <command>`
- `janus <command>`
- `python -m janus_labs <command>`

### Global

```bash
janus-labs --help
janus-labs --version
janus-labs
```

Running `janus-labs` with no arguments opens the interactive menu.

### `run`

Run an end-to-end suite directly.

```bash
janus-labs run --suite refactor-storm --mock -o result.json
janus-labs run --suite refactor-storm -o result.json
janus-labs refactor-storm -o result.json
```

Options:

- `--suite`: suite ID, required
- `--output`, `-o`: output file, default `result.json`
- `--format`: `json`, `html`, or `both`
- `--mock`: use deterministic offline scoring
- `--judge`: use local LLM-as-judge scoring
- `--model`: judge model, default `gpt-4o`
- `--no-interactive`: disable prompts on backend rate limits
- `--request-delay`: delay between judge requests
- `--full`: run the full `init -> agent -> score -> output` pipeline

### `init`

Initialize workspaces for every behavior in a suite.

```bash
janus-labs init --suite refactor-storm --output ./janus-task
```

Options:

- `--suite`: suite ID, default `refactor-storm`
- `--output`, `-o`: output directory for generated behavior workspaces

### `status`

Inspect a task workspace and get the recommended next step.

```bash
janus-labs status --workspace ./janus-task/BHV-001-test-cheating
```

### `score`

Score one completed task workspace.

```bash
janus-labs score --workspace ./janus-task/BHV-001-test-cheating --output result.json
```

Options:

- `--workspace`, `-w`: workspace path, default current directory
- `--output`, `-o`: output file, default `result.json`
- `--judge`: enable LLM-as-judge scoring
- `--model`: judge model, default `gpt-4o`
- `--bundle`: optional bundle file for judge scoring
- `--agent`: override detected agent identifier
- `--agent-model`: override detected agent model

### `submit`

Submit a scored result to the public leaderboard.

```bash
janus-labs submit result.json --github your-handle
```

Options:

- `--dry-run`: print the payload without submitting
- `--github`: GitHub handle for attribution
- `--no-judge`: skip backend LLM judging during submission
- `--no-open`: skip opening the share URL in a browser (useful in CI)

### `compare`

Detect regressions between two results, or compare a result against a precomputed vanilla baseline.

```bash
janus-labs compare baseline.json current.json
janus-labs compare result.json --auto-baseline
```

### `profile`

Generate capability profiles from one baseline file or a directory of baselines.

```bash
janus-labs profile --baseline data/baselines/baseline.json
janus-labs profile --baselines-dir data/baselines
janus-labs profile --baselines-dir data/baselines --json
janus-labs profile --leaderboard
janus-labs profile --reliability
```

### `diagnose`

Measure optional instruction resilience by comparing configured results against vanilla baselines.

```bash
janus-labs diagnose result.json
```

### `smoke-test`

Quick validation path using mock data. This does not benchmark your real config.

```bash
janus-labs smoke-test --suite refactor-storm
```

### `baseline`

Inspect or update baseline files.

```bash
janus-labs baseline list
janus-labs baseline show baseline.json
janus-labs baseline update result.json -o baseline.json
```

## Built-In Suite

### Refactor Storm (`v2.0.0`)

`refactor-storm` ships 4 built-in behaviors grouped into 2 active axes:

- `BHV-001` Test Cheating Detection -> `Code Quality`
- `BHV-002` Reduce Cyclomatic Complexity -> `Code Quality`
- `BHV-003` Add Comprehensive Error Handling -> `Error Resilience`
- `BHV-004` Loop Detection Compliance -> `Error Resilience`

Composite score = `avg(Code Quality, Error Resilience)`.

`janus-labs diagnose` can compute optional Instruction Resilience separately by comparing configured runs against vanilla baselines. It is not part of the 2-axis composite.

## GitHub Actions

```yaml
- name: Install Janus Labs
  run: pip install janus-labs

- name: Run mock suite
  run: janus-labs run --suite refactor-storm --mock --no-interactive -o current.json

- name: Compare to baseline
  run: janus-labs compare baseline.json current.json --format github
```

## Requirements

- Python 3.12+
- Core dependencies include DeepEval, GitPython, PyYAML, and Pydantic

Phoenix telemetry is optional and requires Python `<3.14`:

```bash
pip install -r requirements-phoenix.txt
```

## Contributing

See [CONTRIBUTING.md](CONTRIBUTING.md).

## License

Apache 2.0. See [LICENSE](LICENSE).
