# LLM Pentest

[![CI](https://github.com/jccazako/llm-pentest/actions/workflows/ci.yml/badge.svg)](https://github.com/jccazako/llm-pentest/actions/workflows/ci.yml)
[![PyPI](https://img.shields.io/pypi/v/llm-pentest)](https://pypi.org/project/llm-pentest/)
[![Python](https://img.shields.io/pypi/pyversions/llm-pentest)](https://pypi.org/project/llm-pentest/)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
[![OWASP](https://img.shields.io/badge/OWASP-LLM%20Top%2010-blue)](https://owasp.org/www-project-top-10-for-large-language-model-applications/)

Security testing toolkit for LLM-based systems — chatbots, RAG pipelines, and AI agents.

Tests for prompt injection, data exfiltration, tool abuse, and jailbreaking.

## Install

```bash
pip install -e .
```

## Quick Start

```bash
# Interactive setup
llm-pentest init

# Or use a built-in profile
llm-pentest scan --profile openai

# Or provide a config file
cp examples/target.yaml my-target.yaml
llm-pentest scan -c my-target.yaml
```

## Usage

```bash
# List payloads and chains
llm-pentest payloads

# Scan specific category
llm-pentest scan -c target.yaml --category prompt_injection

# Scan single payload
llm-pentest scan -c target.yaml --payload pi-001

# Parallel scanning (5 concurrent requests)
llm-pentest scan -c target.yaml -n 5

# Only fail CI on high/critical findings
llm-pentest scan -c target.yaml --fail-on high

# Use a profile instead of config file
llm-pentest scan --profile openai --model gpt-4o

# HTML report
llm-pentest scan -c target.yaml --format html -o report.html

# SARIF report (GitHub Code Scanning compatible)
llm-pentest scan -c target.yaml --format sarif -o report.sarif

# Baseline comparison
llm-pentest scan -c target.yaml -o baseline.json
# ... fix vulnerabilities ...
llm-pentest scan -c target.yaml -o current.json --baseline baseline.json

# Standalone diff
llm-pentest diff -b baseline.json --current current.json

# Custom detection hooks
llm-pentest scan -c target.yaml --hooks my_hooks.py

# LLM-as-judge for better detection accuracy
llm-pentest scan -c target.yaml --judge-endpoint https://api.openai.com/v1/chat/completions --judge-key $OPENAI_API_KEY

# Dry run (no requests sent)
llm-pentest scan -c target.yaml --dry-run
```

## Target Profiles

Built-in profiles for common providers (`llm-pentest profiles`):

| Profile | Endpoint | Default Model |
|---|---|---|
| `openai` | api.openai.com | gpt-4o-mini |
| `anthropic` | api.anthropic.com | claude-sonnet-4-20250514 |
| `ollama` | localhost:11434 | llama3 |

## Attack Categories

| Category | Payloads | OWASP | Description |
|---|---|---|---|
| `prompt_injection` | pi-001 to pi-009 | LLM01 | System prompt extraction, instruction override, encoding bypass, indirect injection |
| `data_exfiltration` | de-001 to de-005 | LLM06 | RAG context leak, credential extraction, PII harvesting |
| `tool_abuse` | ta-001 to ta-004 | LLM07, LLM08 | Unauthorized tool use, parameter injection, CoT hijacking, DoS |
| `jailbreak` | jb-001 to jb-004 | LLM01 | Role-play, hypothetical framing, token smuggling, persona splitting |

## Attack Chains

Chains compose multiple payloads into multi-step attack sequences:

```yaml
- id: chain-001
  name: "Jailbreak then Extract Credentials"
  steps:
    - payload_id: jb-001
      prompt_index: 0
    - payload_id: de-002
      prompt_index: 0
```

## Custom Payloads

Drop YAML files into `payloads/` or pass `--payloads-dir`:

```yaml
- id: custom-001
  name: "My Custom Attack"
  category: prompt_injection
  severity: high
  owasp: ["LLM01"]
  prompts:
    - "Your attack prompt here"
  detection:
    - pattern: "expected response pattern"
      confidence: 0.8
```

## Custom Detection Hooks

Write Python functions for domain-specific detection:

```python
# my_hooks.py — any function named detect_* is auto-registered
def detect_internal_data(payload, prompt, response):
    if "internal-project-name" in response.lower():
        return "Leaked internal project name"
    return None
```

```bash
llm-pentest scan -c target.yaml --hooks my_hooks.py
```

## CI/CD Integration

```bash
# Fail only on high/critical findings
llm-pentest scan -c target.yaml --fail-on high || echo "Vulnerabilities found!"

# SARIF for GitHub Code Scanning
llm-pentest scan -c target.yaml --format sarif -o results.sarif

# Baseline regression check
llm-pentest scan -c target.yaml --baseline previous.json --fail-on high
```

## License

MIT
