Metadata-Version: 2.4
Name: openrouter-inspector
Version: 0.1.2
Summary: A command-line tool for querying OpenRouter AI models and providers
Project-URL: Homepage, https://github.com/matdev83/openrouter-inspector/
Project-URL: Repository, https://github.com/matdev83/openrouter-inspector/
Project-URL: Documentation, https://github.com/matdev83/openrouter-inspector#readme
Project-URL: Bug Tracker, https://github.com/matdev83/openrouter-inspector/issues
Author-email: "Mateusz B." <matdev83@github.com>
Maintainer-email: "Mateusz B." <matdev83@github.com>
License: MIT License
        
        Copyright (c) 2024 OpenRouter CLI Team
        
        Permission is hereby granted, free of charge, to any person obtaining a copy
        of this software and associated documentation files (the "Software"), to deal
        in the Software without restriction, including without limitation the rights
        to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
        copies of the Software, and to permit persons to whom the Software is
        furnished to do so, subject to the following conditions:
        
        The above copyright notice and this permission notice shall be included in all
        copies or substantial portions of the Software.
        
        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
        FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
        AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
        LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
        OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
        SOFTWARE.
License-File: LICENSE
Keywords: ai,api,cli,models,openrouter
Classifier: Development Status :: 3 - Alpha
Classifier: Environment :: Console
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: System Administrators
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Internet :: WWW/HTTP :: HTTP Servers
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: Topic :: System :: Monitoring
Classifier: Topic :: Utilities
Classifier: Typing :: Typed
Requires-Python: >=3.10
Requires-Dist: click>=8.0.0
Requires-Dist: httpx>=0.24.0
Requires-Dist: pydantic>=2.0.0
Requires-Dist: rich>=13.0.0
Requires-Dist: tiktoken>=0.5.0
Requires-Dist: tomli>=2.0.1; python_version < '3.11'
Provides-Extra: all
Requires-Dist: bandit[toml]>=1.7.5; extra == 'all'
Requires-Dist: black>=23.0.0; extra == 'all'
Requires-Dist: build>=0.10.0; extra == 'all'
Requires-Dist: hatch-vcs>=0.3.0; extra == 'all'
Requires-Dist: hatch>=1.9.0; extra == 'all'
Requires-Dist: hatchling>=1.18.0; extra == 'all'
Requires-Dist: hypothesis>=6.0.0; extra == 'all'
Requires-Dist: mkdocs-material>=9.0.0; extra == 'all'
Requires-Dist: mkdocs>=1.5.0; extra == 'all'
Requires-Dist: mkdocstrings[python]>=0.20.0; extra == 'all'
Requires-Dist: mypy>=1.0.0; extra == 'all'
Requires-Dist: pre-commit>=3.0.0; extra == 'all'
Requires-Dist: pytest-asyncio>=0.21.0; extra == 'all'
Requires-Dist: pytest-cov>=4.0.0; extra == 'all'
Requires-Dist: pytest-httpx>=0.21.0; extra == 'all'
Requires-Dist: pytest>=7.0.0; extra == 'all'
Requires-Dist: ruff>=0.1.0; extra == 'all'
Requires-Dist: safety>=3.2.0; extra == 'all'
Requires-Dist: twine>=4.0.0; extra == 'all'
Requires-Dist: vulture>=2.0; extra == 'all'
Requires-Dist: wheel>=0.40.0; extra == 'all'
Provides-Extra: build
Requires-Dist: build>=0.10.0; extra == 'build'
Requires-Dist: hatch-vcs>=0.3.0; extra == 'build'
Requires-Dist: hatch>=1.9.0; extra == 'build'
Requires-Dist: hatchling>=1.18.0; extra == 'build'
Requires-Dist: twine>=4.0.0; extra == 'build'
Requires-Dist: wheel>=0.40.0; extra == 'build'
Provides-Extra: complete
Requires-Dist: bandit[toml]>=1.7.5; extra == 'complete'
Requires-Dist: black>=23.0.0; extra == 'complete'
Requires-Dist: build>=0.10.0; extra == 'complete'
Requires-Dist: hatch-vcs>=0.3.0; extra == 'complete'
Requires-Dist: hatch>=1.9.0; extra == 'complete'
Requires-Dist: hatchling>=1.18.0; extra == 'complete'
Requires-Dist: hypothesis>=6.0.0; extra == 'complete'
Requires-Dist: mkdocs-material>=9.0.0; extra == 'complete'
Requires-Dist: mkdocs>=1.5.0; extra == 'complete'
Requires-Dist: mkdocstrings[python]>=0.20.0; extra == 'complete'
Requires-Dist: mypy>=1.0.0; extra == 'complete'
Requires-Dist: pre-commit>=3.0.0; extra == 'complete'
Requires-Dist: pytest-asyncio>=0.21.0; extra == 'complete'
Requires-Dist: pytest-cov>=4.0.0; extra == 'complete'
Requires-Dist: pytest-httpx>=0.21.0; extra == 'complete'
Requires-Dist: pytest>=7.0.0; extra == 'complete'
Requires-Dist: ruff>=0.1.0; extra == 'complete'
Requires-Dist: safety>=3.2.0; extra == 'complete'
Requires-Dist: twine>=4.0.0; extra == 'complete'
Requires-Dist: vulture>=2.0; extra == 'complete'
Requires-Dist: wheel>=0.40.0; extra == 'complete'
Provides-Extra: dev
Requires-Dist: bandit[toml]>=1.7.5; extra == 'dev'
Requires-Dist: black>=23.0.0; extra == 'dev'
Requires-Dist: mypy>=1.0.0; extra == 'dev'
Requires-Dist: pre-commit>=3.0.0; extra == 'dev'
Requires-Dist: ruff>=0.1.0; extra == 'dev'
Requires-Dist: safety>=3.2.0; extra == 'dev'
Requires-Dist: vulture>=2.0; extra == 'dev'
Provides-Extra: docs
Requires-Dist: mkdocs-material>=9.0.0; extra == 'docs'
Requires-Dist: mkdocs>=1.5.0; extra == 'docs'
Requires-Dist: mkdocstrings[python]>=0.20.0; extra == 'docs'
Provides-Extra: security
Requires-Dist: bandit[toml]>=1.7.5; extra == 'security'
Requires-Dist: safety>=3.2.0; extra == 'security'
Provides-Extra: test
Requires-Dist: hypothesis>=6.0.0; extra == 'test'
Requires-Dist: pytest-asyncio>=0.21.0; extra == 'test'
Requires-Dist: pytest-cov>=4.0.0; extra == 'test'
Requires-Dist: pytest-httpx>=0.21.0; extra == 'test'
Requires-Dist: pytest>=7.0.0; extra == 'test'
Description-Content-Type: text/markdown

# OpenRouter Inspector

[![CI](https://github.com/matdev83/openrouter-inspector/actions/workflows/ci.yml/badge.svg?branch=main)](https://github.com/matdev83/openrouter-inspector/actions/workflows/ci.yml)
[![codecov](https://codecov.io/gh/matdev83/openrouter-inspector/branch/main/graph/badge.svg)](https://codecov.io/gh/matdev83/openrouter-inspector)
[![PyPI](https://img.shields.io/badge/PyPI-not%20published-lightgrey)](https://pypi.org/project/openrouter-inspector/)
![Python](https://img.shields.io/badge/python-%3E%3D3.10-blue)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
[![Ruff](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/astral-sh/ruff/main/assets/badge/v2.json)](https://github.com/astral-sh/ruff)
[![security: bandit](https://img.shields.io/badge/security-bandit-yellow.svg)](https://github.com/PyCQA/bandit)
[![Pre-commit](https://img.shields.io/badge/pre--commit-enabled-brightgreen?logo=pre-commit&logoColor=white)](https://github.com/pre-commit/pre-commit)
[![Tests](https://github.com/matdev83/openrouter-inspector/actions/workflows/tests.yml/badge.svg?branch=main)](https://github.com/matdev83/openrouter-inspector/actions/workflows/tests.yml)
![Last commit](https://img.shields.io/github/last-commit/matdev83/openrouter-inspector)
[![Issues](https://img.shields.io/github/issues/matdev83/openrouter-inspector)](https://github.com/matdev83/openrouter-inspector/issues)

A lightweight CLI for exploring OpenRouter AI models, listing provider endpoints with supported model parameters, and benchmarking endpoint latency and throughput.

## Installation

### Requirements

- Python 3.10+

### From source (current)

The package is not on PyPI yet. Install from source:

- With pipx (recommended for CLIs):
  ```bash
  pipx install .
  ```
- Or with pip into your active environment:
  ```bash
  pip install .
  ```

To try the latest in editable mode (for experimentation), use:
```bash
pip install -e .
```

### Contributing / Development

If you want to hack on the project (dev setup, tests, QA, pre-commit, etc.), see
the dedicated contributor guide:

[CONTRIBUTING.md](CONTRIBUTING.md)

## Features

- Explore available models and provider-specific endpoints from OpenRouter.
- Rich table output with pricing per 1M tokens and optional provider counts.
- Change detection for new models and pricing changes between runs.
- JSON output for easy scripting.

## Usage

The CLI supports both subcommands and lightweight global flags.

### Authentication

Set your OpenRouter API key via environment variable (required):

```bash
export OPENROUTER_API_KEY=sk-or-...
```

For security, the CLI does not accept API keys via command-line flags. It reads the key only from the `OPENROUTER_API_KEY` environment variable. If the key is missing or invalid, the CLI shows a friendly error and exits.

### Quick starts

Subcommands:

```bash
# List all models
openrouter-inspector list

# List models filtered by substring (matches id or display name)
openrouter-inspector list openai

# List models with multiple filters (AND logic)
openrouter-inspector list meta free

# Or just type your search terms; default action is `list`
openrouter-inspector gemini-2.0 free

# Detailed provider endpoints (exact model id)
openrouter-inspector endpoints deepseek/deepseek-r1

# To check the endpoint health and latency
openrouter-inspector ping google/gemini-2.0-flash-exp:free
```

 

### Commands

#### list

```bash
openrouter-inspector list [filters...] [--with-providers] [--sort-by id|name|context|providers] [--desc] [--format table|json|yaml]
```

- Displays all available models with enhanced table output (Name, ID, Context, Input/Output pricing).
- Optional positional `filters` performs case-insensitive substring matches against model id and name using AND logic.
- Context values are displayed with K suffix (e.g., 128K).
- Input/Output prices are shown per million tokens in USD.
- **Change Detection**: Automatically detects new models and pricing changes compared to previous runs with the same parameters. New models are shown in a separate table, and pricing changes are highlighted in yellow.

Options:
- `--format [table|json|yaml]` (default: table)
- `--with-providers` add a Providers column (makes extra API calls per model)
- `--sort-by [id|name|context|providers]` (default: id)
- `--desc` sort descending

#### endpoints

```bash
openrouter-inspector endpoints MODEL_ID [--min-quant VALUE] [--min-context VALUE] [--sort-by provider|model|quant|context|maxout|price_in|price_out] [--desc] [--per-1m] [--format table|json|yaml]
```

Shows detailed provider offers for an exact model id (`author/slug`), with:
- Provider, Model (provider endpoint name), Reason (+/-), Quant, Context (K), Max Out (K), Input/Output price (USD/1M)

Behavior:
- Fails if model id does not match an exact existing model or returns no offers.

Filters and sorting:
- `--min-quant VALUE` minimum quantization (e.g., fp8). Unspecified quant (“—”) is included as best.
- `--min-context VALUE` minimum context window (e.g., `128K` or `131072`).
- `--sort-by [provider|model|quant|context|maxout|price_in|price_out]` (default: provider)
- `--desc` sort descending

#### check

```bash
openrouter-inspector check MODEL_ID PROVIDER_NAME ENDPOINT_NAME
```

Checks a specific provider endpoint's health using OpenRouter API status. Web-scraped metrics have been removed.

Behavior:
- Returns one of: `Functional`, `Disabled`.
- If API indicates provider is offline/disabled or not available → `Disabled`.
- Otherwise → `Functional`.

Options:
- `--log-level [CRITICAL|ERROR|WARNING|INFO|DEBUG|NOTSET]` set logging level

#### ping

```bash
openrouter-inspector ping MODEL_ID [PROVIDER_NAME]
openrouter-inspector ping MODEL_ID@PROVIDER_NAME

# Examples
openrouter-inspector ping openai/o4-mini
openrouter-inspector ping deepseek/deepseek-chat-v3-0324:free Chutes
openrouter-inspector ping deepseek/deepseek-chat-v3-0324:free@Chutes
```

- Performs an end-to-end chat completion call to verify the functional state of a model or a specific provider endpoint.
- Uses a tiny “Ping/Pong” prompt and minimizes completion size for a fast and inexpensive check.
- When a provider is specified (positional or `@` shorthand), the request pins routing order to that provider and disables fallbacks.
- Prints the provider that served the request, token usage, USD cost (unrounded when provided by the API), measured latency, and effective TTL.
- Returns OS exit code `0` on 100% success (zero packet loss) and `1` otherwise, making it suitable for scripting.

Behavior:
- Default timeout: 60s. Change via `--timeout <seconds>`.
- Default ping count: 3. Change via `-n <count>` or `-c <count>`.
- Reasoning minimized by default for low-cost pings (reasoning.effort=low, exclude=true; legacy include_reasoning=false).
- Caps `max_tokens` to 4 for expected “Pong” reply.
- Dynamically formats latency: `<1000ms` prints in `ms`; `>=1s` prints in seconds with two decimals (e.g., `1.63s`).

Options:
- `--timeout <seconds>`: Per-request timeout override (defaults to 60 if missing or invalid).
- `-n <count>`, `-c <count>`: Number of pings to send (defaults to 3).
- `--filthy-rich`: Required if sending more than 10 pings to acknowledge potential API costs.
- `--log-level [CRITICAL|ERROR|WARNING|INFO|DEBUG|NOTSET]`: Set logging level.

Example output:

```

Pinging https://openrouter.ai/api/v1/chat/completions/tngtech/deepseek-r1t2-chimera:free@Chutes with 26 input tokens:
Reply from: https://openrouter.ai/api/v1/chat/completions/tngtech/deepseek-r1t2-chimera:free@Chutes tokens: 4 cost: $0.00 time=2.50s TTL=60s

Pinging https://openrouter.ai/api/v1/chat/completions/tngtech/deepseek-r1t2-chimera:free@Chutes with 26 input tokens:
Reply from: https://openrouter.ai/api/v1/chat/completions/tngtech/deepseek-r1t2-chimera:free@Chutes tokens: 4 cost: $0.00 time=2.30s TTL=60s

Ping statistics for tngtech/deepseek-r1t2-chimera:free@Chutes:
    Packets: Sent = 2, Received = 2, Lost = 0 (0% loss),
Approximate round trip times in seconds:
    Minimum = 2.30s, Maximum = 2.50s, Average = 2.40s
Total API cost for this run: $0.000000

```

Notes:
- Provider pinning uses the OpenRouter provider routing preferences (order, allow_fallbacks=false when a provider is specified). See provider routing docs for details.

> ⚠️ **Warning**
>
> Running `ping` against paid endpoints will make a real completion call and can consume your API credits. It is not a simulated or “no-op” health check. Use with care on metered providers.
>
> Additionally, even when using "free" models, each ping counts against the daily request limit of OpenRouter's free tier. Use with caution, especially if incorporating the command into monitoring scripts or frequent, automated checks.

#### benchmark

```bash
openrouter-inspector benchmark MODEL_ID [PROVIDER_NAME] \
  [--timeout <seconds>] [--max-tokens <limit>] [--format table|json|text] [--min-tps <threshold>] [--debug-response]
```

- Measures model or provider-specific throughput (tokens per second, TPS) by streaming a long response.
- When **`PROVIDER_NAME`** is specified (either as a second positional argument *or* using the shorthand `MODEL_ID@PROVIDER_NAME`), routing is *pinned* to that provider and fallbacks are disabled. If omitted, OpenRouter automatically selects the best provider.
- Supports multiple output modes so you can use it in scripts:
  - `table` (default): Rich table with metrics (Status, Duration, Input/Output/Total tokens, Throughput, Cost). Includes a short “Benchmarking …” preface.
  - `json`: Emits a JSON object with the same metrics as the table.
  - `text`: Emits a single line: `TPS: <value>`.

Options:
- `--timeout <seconds>`: Request timeout (default: 120).
- `--max-tokens <limit>`: Safety cap for generated tokens (default: 3000).
- `--format [table|json|text]`: Output format (default: table).
- `--min-tps <threshold>`: Enforce a minimum TPS threshold (range 1–10000) in `text` mode. Exit code is `1` when measured TPS is lower than threshold, otherwise `0`.
- `--debug-response`: Print streaming chunk JSON for debugging (noisy).

Examples:

```bash
# Human-friendly table for auto-selected provider
openrouter-inspector benchmark google/gemini-2.0-flash-exp:free

# Pin benchmark to a specific provider (positional argument)
openrouter-inspector benchmark google/gemini-2.0-flash-exp:free Chutes

# Same, using @ shorthand
openrouter-inspector benchmark google/gemini-2.0-flash-exp:free@Chutes

# JSON for automation
openrouter-inspector benchmark google/gemini-2.0-flash-exp:free --format json

# Text-only TPS with threshold suitable for CI/monitoring (non-zero exit code on breach)
openrouter-inspector benchmark google/gemini-2.0-flash-exp:free --format text --min-tps 200
```

Scripting/monitoring notes:
- In `text` format with `--min-tps`, the command exits with code `1` if TPS is below the threshold (else `0`). Use this in CI/CD, cron, or health checks.
- In `table`/`json` formats, the exit code reflects execution success, not a threshold check.

### Examples

```bash
# Top-level listing filtered by vendor substring
openrouter-inspector list "google"

# List models with multiple filters (AND logic)
openrouter-inspector list "meta" "free"

# Endpoints with filters and sorting: min quant fp8, min context 128K, sort by price_out desc
openrouter-inspector endpoints deepseek/deepseek-r1 --min-quant fp8 --min-context 128K --sort-by price_out --desc

# Lightweight mode with sorting
openrouter-inspector --list --sort-by name
```

## Notes

- Models are retrieved from `/api/v1/models`. Provider offers per model are retrieved from `/api/v1/models/:author/:slug/endpoints`.
- Supported parameters listed on `/models` are a union across providers. Use `/endpoints` for per-provider truth.
- Some fields may vary by provider (context, pricing, features); the CLI reflects these differences.

## License

MIT License - see LICENSE file for details.
