# Competitive Analysis

## Summary

| Tool | Type | Language | Stars | License | Key Differentiator |
|---|---|---|---|---|---|
| **Guardrails AI** | Runtime guardrails (defensive) | Python | ~6.4k | Apache 2.0 | Validator Hub + corrective actions |
| **Promptfoo** | LLM eval + red teaming | TypeScript | ~10.4k | MIT | 131 plugins, compliance mapping, web dashboard |
| **PyRIT** | Red teaming framework | Python | ~3.4k | MIT | Microsoft-backed, multi-turn orchestrators, multimodal |
| **NVIDIA Garak** | LLM vulnerability scanner | Python | ~6.9k | Apache 2.0 | 150+ probes, research-grounded, NVIDIA-backed |
| **DeepTeam** | LLM red teaming | Python | ~1.3k | Apache 2.0 | 50+ vulns, agentic category, OWASP/NIST mapping |
| **agent-audit** | Static analysis (SAST) | Python | new | — | AST scanning, taint tracking, MCP security |

### Leshy's Position

Leshy is the only tool purpose-built for **dynamic attack simulation against live AI agents** with:
- Config-driven YAML (no code required)
- Composable buff pipeline (encoding + LLM-based payload transformers)
- Dual evaluation (pattern regex + LLM judge)
- Native LangGraph support
- CI/CD-native (exit codes 0/1/2)
- Zero-cost basic scans (no LLM API needed for pattern evaluation)

### Priority Gaps to Close

| Gap | Priority | Source |
|---|---|---|
| Multi-turn adaptive attacks (conversation state + LLM-driven refinement) | HIGH | PyRIT, Promptfoo, DeepTeam |
| Dynamic attack generation (attacker LLM generates contextual payloads) | HIGH | Promptfoo, Garak (`atkgen`) |
| OWASP Agentic Top 10 / NIST mapping | MEDIUM | agent-audit, DeepTeam, Promptfoo |
| SARIF output + GitHub Action | MEDIUM | agent-audit |
| MCP-specific attack payloads | MEDIUM | agent-audit, Promptfoo |
| CWE IDs on attack categories | LOW | agent-audit |
| Multimodal testing (image/audio) | LOW | PyRIT, Promptfoo |

---

## Guardrails AI

**Repository:** https://github.com/guardrails-ai/guardrails
**Type:** Runtime guardrails (defensive) — validates and corrects LLM I/O in production
**Stars:** ~6,400 | **License:** Apache 2.0 | **Team:** ~10 people, $7.5M seed funding

### What it does

Runtime validation framework. A `Guard` wraps LLM API calls, runs validators (from a community Hub), and takes corrective actions (reask, fix, filter, refrain). Covers safety, PII, hallucination, toxicity, jailbreak detection, and structured output enforcement. Validators installed individually from the Guardrails Hub (npm-style). Deployment modes: in-app Python library, standalone server, or managed Guardrails Pro.

### How it compares to Leshy

**Fundamentally different tools.** Guardrails AI is a shield (runtime defense), Leshy is a sword (pre-deployment offensive testing). They're complementary:

```
[Leshy: discover vulnerabilities] → [Deploy guardrails based on findings] → [Guardrails AI: enforce at runtime]
```

Research confirms: "Red-teaming is a great way to detect which vulnerabilities your LLM needs guardrails for." Without offensive testing, you deploy guardrails blindly. Without runtime guardrails, you have no production protection.

### What they have that we don't

| Feature | Why it matters |
|---|---|
| Corrective action system (reask/fix/filter) | Goes beyond detection to auto-remediation |
| Guardrails Hub (community validators) | Ecosystem/marketplace model for extensions |
| Guardrails Index benchmark | Published comparison of 24 guardrail solutions — thought leadership |
| Server mode (validation microservice) | Decoupled deployment for production |

### What we have that they can't do

| Feature | Details |
|---|---|
| Proactive vulnerability discovery | Finds unknown vulnerabilities through adversarial probing |
| Adversarial sophistication | Payload generation with buffs, multi-vector attacks |
| Agent awareness | Tests tool use, multi-step reasoning, excessive agency |
| Evasion techniques | Encoding, LLM rephrasing bypass guardrail pattern matching |

### Takeaways

- **Not a competitor.** Position alongside Guardrails AI — "test with Leshy, defend with Guardrails."
- Research shows guardrails achieve 70-80% effectiveness but collapse to ~33.8% against novel adversarial attacks — proving offensive testing is essential.
- Guardrails' weakness (reactive, not proactive) is exactly Leshy's strength.

---

## Promptfoo

**Repository:** https://github.com/promptfoo/promptfoo
**Type:** LLM eval + red teaming framework
**Stars:** ~10.4k | **License:** MIT | **Language:** TypeScript (96.7%)

### What it does

Open-source LLM evaluation and red teaming framework. Generates adversarial inputs dynamically based on target's declared "purpose" using an attacker LLM. 131 attack plugins across 6 categories (brand protection, compliance, dataset-based, security, trust & safety, custom). 20+ delivery strategies including multi-turn (Crescendo, GOAT, Hydra with 70-90% ASR). Compliance mapping to OWASP, NIST, MITRE ATLAS, EU AI Act, GDPR. Web dashboard for result visualization. Enterprise offering with continuous monitoring.

### How it compares to Leshy

The most direct competitor — both are config-driven CLI tools for adversarial testing. Key differences:

| Dimension | Promptfoo | Leshy |
|---|---|---|
| Language | TypeScript (Node.js) | Python |
| Config | YAML | YAML |
| Attack generation | Dynamic (attacker LLM) + static | Static library + buff transformers |
| Plugins | 131 | 28 |
| Strategies | 20+ (multi-turn, adaptive) | Buffs (encoding + LLM-based) |
| Evaluation | LLM-as-judge only | Pattern (regex) + LLM judge |
| Framework mapping | OWASP, NIST, MITRE, EU AI Act | None yet |
| Agent support | Bolt-on (agent hooks) | Native (LangGraph target) |
| Cost model | LLM API required for all scans | Zero-cost basic scans possible |
| Install | `npm install` | `pip install` |

### What they have that we don't

| Feature | Why it matters |
|---|---|
| Dynamic attack generation | Contextual payloads based on target's purpose — more realistic |
| 131 plugins (industry-specific) | Telecom, healthcare, real estate, insurance compliance |
| Multi-turn strategies (Crescendo, GOAT, Hydra) | 70-90% ASR, adaptive refinement |
| Compliance framework mapping | Enterprise reporting requirement |
| Web dashboard | Visual comparison, polished enterprise reporting |
| Multi-modal red teaming | Image, audio, video-based attacks |
| 10.4k stars, Fortune 500 adoption | Trust and ecosystem momentum |

### What we have that they don't

| Feature | Details |
|---|---|
| Python-native | No Node.js dependency; pip install, pytest integration, same ecosystem as AI/ML teams |
| Composable buff pipeline | Chain: encode → prefix → suffix → translate → send. Strategies aren't composable in Promptfoo |
| Pattern evaluation (regex) | Deterministic, zero-cost, no LLM needed for basic scans |
| Purpose-built for security | Not retrofitted from an eval framework |
| Agent-first architecture | LangGraph SDK integration, not bolt-on hooks |
| Lighter weight | Focused tool vs. eval + red team + dashboard + enterprise platform |

### Takeaways

- **Primary competitor.** Promptfoo has massive breadth and momentum.
- **Our angle:** Python-native, agent-first, composable buffs, zero-cost basic scans. "Promptfoo is Node.js eval framework that added security. Leshy is a Python security tool built for agents."
- **Must close gaps:** Dynamic attack generation (HIGH), multi-turn attacks (HIGH), compliance mapping (MEDIUM).

---

## Microsoft PyRIT

**Repository:** https://github.com/Azure/PyRIT
**Type:** Red teaming framework (code-first)
**Stars:** ~3,400 | **License:** MIT | **Status:** Alpha (v0.11.0)

### What it does

Python Risk Identification Tool for generative AI, built by Microsoft's AI Red Team. Used in 100+ internal red teaming operations (Copilots, Phi-3). Composable "Lego-brick" architecture with 6 component types: Orchestrators (attack coordination), Targets, Converters (prompt transformers), Scorers, Memory (DuckDB persistence), Datasets. Supports multi-turn attacks (Crescendo, TAP, PAIR, GCG), multimodal I/O, and Azure ecosystem integration.

### How it compares to Leshy

PyRIT is code-first (Python scripts/notebooks), Leshy is config-first (YAML + CLI). Different audiences:

| Dimension | PyRIT | Leshy |
|---|---|---|
| Interface | Python API, Jupyter notebooks | YAML config + CLI |
| Time to first test | Write script, configure memory, async setup | Write YAML, run `leshy run` |
| Learning curve | 6 component types, async patterns | Flat model: attacks, targets, evaluators, buffs |
| CI/CD integration | Build custom wrapper | Exit codes work natively |
| Configuration | Code scattered across files | Single declarative YAML |
| Onboarding | Security engineers who know Python | Any developer who can write YAML |
| Multi-turn attacks | Crescendo, TAP, PAIR (state-of-the-art) | Not yet |
| Multimodal | Text, image, audio | Text only |
| Backing | Microsoft AI Red Team | Independent |

### What they have that we don't

| Feature | Why it matters |
|---|---|
| Multi-turn orchestrators (Crescendo, TAP, PAIR) | Most advanced open-source multi-turn attacks |
| Multimodal support (image, audio) | Cross-modal attack vectors |
| DuckDB memory persistence | Post-hoc analysis across sessions |
| Microsoft backing, 100+ real operations | Battle-tested credibility |
| Converter stacking | Chain encoding + LLM rewriting + translation freely |

### What we have that they don't

| Feature | Details |
|---|---|
| Zero-code entry point | YAML config vs. Python scripts |
| CI/CD-native | Exit codes, structured output, pipeline-friendly |
| Fast feedback loops | `leshy run config.yaml` vs. notebook kernels + async boilerplate |
| Reproducible config | YAML file is the complete specification, version-controllable |
| Lighter footprint | httpx + few deps vs. DuckDB + Azure SDKs + optional heavyweight extras |

### Takeaways

- **Different audiences.** PyRIT for research/deep-dive red teams with Python expertise. Leshy for dev teams wanting CI/CD security scans.
- **Our angle:** "PyRIT is a research framework. Leshy is a developer tool."
- **Learn from:** Their converter stacking design (similar to our buffs), multi-turn orchestration patterns.

---

## NVIDIA Garak

**Repository:** https://github.com/NVIDIA/garak
**Type:** LLM vulnerability scanner
**Stars:** ~6,900 | **License:** Apache 2.0

### What it does

LLM-focused vulnerability scanner analogous to Nmap/Metasploit for LLMs. Plugin-based architecture: Generators (targets), Probes (attacks — 150+ across 35+ modules, 3,000+ prompts), Detectors (signature + ML-based), Buffs (prompt transformers), Harnesses (orchestrators), Evaluators. Covers prompt injection, jailbreaks, data extraction, hallucination, toxicity, malware generation, misinformation. Has `atkgen` probe for dynamic attack generation via attacker LLM.

### How it compares to Leshy

Garak scans LLMs, Leshy tests agents. Garak treats targets as text-in/text-out black boxes:

| Dimension | Garak | Leshy |
|---|---|---|
| Target model | LLM (prompt → response) | Agent (HTTP endpoint, LangGraph) |
| Agent awareness | None — prompt-response only | Tool use, excessive agency, multi-step workflows |
| Attack breadth | 150+ probes, 3,000+ prompts | 28 attacks (focused) |
| Detection | Signature + ML classifiers (DistilBERT) | Pattern (regex) + LLM judge |
| Setup | Conda env, heavy deps | pip install, single YAML |
| CI/CD | Minutes-to-hours scans, impractical | Selective runs, fast feedback, exit codes |
| Scope control | All-or-nothing by default | Explicit attack selection per config |
| Real-time output | JSON logs + HTML report | Console, JSON, HTML, LangFuse, LangSmith |

### What they have that we don't

| Feature | Why it matters |
|---|---|
| 150+ probes, 3,000+ prompts | Widest curated attack corpus in open source |
| ML-based detectors (DistilBERT) | More sophisticated detection than regex |
| `atkgen` dynamic probe | Attacker LLM generates novel prompts |
| Research-grounded (arXiv paper) | Academic credibility, cited by industry |
| NVIDIA backing | Long-term institutional support |

### What we have that they don't

| Feature | Details |
|---|---|
| Agent-first testing | Tool use, function calling, excessive agency, session-aware attacks |
| Config-driven YAML | vs. complex plugin paths and CLI flag soup |
| LLM-based evaluation | LLM judge for nuanced attack success assessment |
| Composable buffs | Chain payload transformers declaratively |
| Real-time rich console | Live progress, warnings, results streaming |
| CI/CD-native | Exit codes, fast selective runs |

### Takeaways

- **Complementary, not competing.** Garak for deep LLM-level vulnerability scanning. Leshy for agent behavior testing.
- **Our angle:** "Garak scans LLMs. Leshy attacks agents."
- Garak is English-only probes, heavyweight install, and minutes-to-hours scan times — all our strengths.

---

## DeepTeam

**Repository:** https://github.com/confident-ai/deepteam
**Type:** LLM red teaming framework (Python API + YAML/CLI)
**Stars:** ~1,300 | **License:** Apache 2.0 | **Parent:** DeepEval (~29k stars)

### What it does

Red teaming layer built on top of DeepEval evaluation framework. 50+ vulnerability types across 6 categories (Responsible AI, Data Privacy, Security, Safety, Business, Agentic). 19 attack methods including 5 multi-turn progressions (CrescendoJailbreaking, TreeJailbreaking). Target via Python callback or LiteLLM model string. LLM-based evaluation (GPT-4o default). Maps to OWASP Top 10 for LLMs and Agentic Applications, NIST AI RMF.

### How it compares to Leshy

Both Python, both support YAML config. But architecturally different:

| Dimension | DeepTeam | Leshy |
|---|---|---|
| Install weight | ~29 deps (gRPC, Sentry, PostHog, OpenTelemetry, pytest) | Minimal (httpx, pydantic, typer, rich, pyyaml) |
| Telemetry | PostHog + Sentry + OpenTelemetry baked in | None |
| Target interface | Python callback function required | HTTP endpoint or agent SDK — no code needed |
| LLM requirement | OpenAI API key effectively required | Optional — regex evaluation works without LLM |
| Evaluation | LLM-as-judge binary (0/1), opaque | Pattern (regex) + LLM judge, auditable |
| Standalone | Requires DeepEval ecosystem | Self-contained |
| Agentic vulns | 9 agentic-specific types | Excessive agency category |
| Standards mapping | OWASP + NIST | Not yet |
| Stability | Documented async/sync bugs | Stable |

### What they have that we don't

| Feature | Why it matters |
|---|---|
| 50+ vulnerability types | Wider taxonomy, especially agentic and business categories |
| Multi-turn progressions (Tree, Crescendo) | Research-backed multi-turn attacks |
| OWASP/NIST mapping | Compliance story for enterprise |
| Custom vulnerability class | Domain-specific risk definitions |
| DeepEval ecosystem (29k stars) | Larger community surface area |

### What we have that they don't

| Feature | Details |
|---|---|
| Zero telemetry | No PostHog, Sentry, or OpenTelemetry |
| Lightweight install | Seconds vs. heavy dependency tree |
| No LLM required for basic scans | Pattern evaluation is free and deterministic |
| HTTP-native targets | Test any API without writing Python |
| Transparent evaluation | Regex is auditable; LLM judge results include reasoning |
| Stable async | No documented sync/async mode bugs |
| Composable buffs | Independent payload transformation pipeline |

### Takeaways

- **Weaker competitor.** Heavy deps, telemetry, stability issues, and DeepEval coupling are liabilities.
- **Our angle:** "DeepTeam is a heavy framework with telemetry. Leshy is a lightweight sharp instrument."
- **Adopt from them:** Agentic vulnerability taxonomy (Goal Theft, Recursive Hijacking, Tool Orchestration Abuse), OWASP/NIST mapping approach.

---

## agent-audit

**Repository:** https://github.com/HeadyZhang/agent-audit
**Type:** Static analysis (SAST) — scans source code, does NOT test running agents
**Created:** February 2026, v0.15.1

### What it does

Static analyzer for AI agent code. Parses Python AST to find dangerous patterns (eval in tool functions, unsanitized LLM output flowing to subprocess, etc.). Five detection engines:

1. **Python AST Scanner** — dangerous function calls, decorator patterns, framework-specific risks
2. **Taint Tracker** — data flow from user input/LLM output to dangerous sinks (shell exec, SQL, file write)
3. **Semantic Analyzer** — credential detection (regex + entropy + context scoring)
4. **MCP Config Scanner** — validates MCP configs for filesystem exposure, missing auth
5. **MCP Runtime Inspector** — probes live MCP servers without executing tools

53 detection rules mapped to OWASP Agentic Top 10 (2026). Framework-aware for LangChain, CrewAI, AutoGen, MCP.

### How it compares to Leshy

**Different tools, same problem space.** agent-audit is SAST, Leshy is DAST. They're complementary:

```
Dev --> [agent-audit: scan code] --> Deploy --> [Leshy: attack live agent] --> Production
```

agent-audit finds "your code has eval() in a tool handler." Leshy finds "your deployed agent leaks its system prompt when asked in Spanish."

### What they have that we don't

| Feature | Why it matters |
|---|---|
| OWASP Agentic Top 10 mapping | Enterprise credibility — our 6 categories cover most of it, need explicit mapping |
| CWE IDs on findings | Standard vulnerability classification expected by security teams |
| SARIF output | Lights up GitHub Security tab, integrates with enterprise vuln management |
| GitHub Action (action.yml) | One-liner CI/CD integration — huge for adoption |
| Baseline/incremental scanning | "Only show new vulnerabilities" — useful for regression testing |
| MCP security focus | MCP becoming standard for agent tooling, we have no MCP-specific attacks |
| Confidence-based tiering | BLOCK/WARN/INFO/SUPPRESSED with float scores, more nuanced than pass/fail |
| Suppression/allowlist system | Ignore known false positives with documented reasons |
| Benchmark suite | Known-vulnerable code samples with precision/recall metrics |

### What we have that they can't do

| Feature | Details |
|---|---|
| Dynamic attack simulation | Actually sends attack payloads to running agents |
| Payload evasion (buffs) | base64, leetspeak, unicode, LLM rephrasing to bypass defenses |
| LLM judge evaluation | AI evaluating if attacks succeeded |
| Multi-turn conversation attacks | Session tracking, context-dependent attack chains |
| Guardrail generation | Create defensive code from test results |
| LangGraph native support | Direct SDK integration for testing LangGraph agents |
| Observability integration | LangFuse, LangSmith logging |
| Token tracking | Monitor LLM API consumption during tests |
| Concurrency control | Parallel payload execution |

### Takeaways

- **Not a direct competitor.** Position Leshy alongside agent-audit, not against it.
- **Adopt:** OWASP Agentic Top 10 mapping, SARIF output, GitHub Action — modest effort, big enterprise credibility boost.
- **Consider:** MCP-specific attack payloads, baseline scanning for regression testing, CWE IDs on attack categories.
- **Our edge:** No other open-source tool does dynamic attack simulation against live AI agents with evasion techniques and automated evaluation. Static analysis can't verify if defenses actually work at runtime.

---

## Competitive Angles (Elevator Pitches)

| vs. | Pitch |
|---|---|
| **Guardrails AI** | "Test with Leshy, defend with Guardrails. You need to know what to guard against." |
| **Promptfoo** | "Python-native security testing built for agents, not a Node.js eval framework with red teaming bolted on." |
| **PyRIT** | "YAML config + CLI instead of Python notebooks. Developer tool, not research framework." |
| **Garak** | "Garak scans LLMs. Leshy attacks agents. Different layer, different tool." |
| **DeepTeam** | "Lightweight, zero-telemetry, no OpenAI key required for basic scans." |
| **agent-audit** | "SAST + DAST. Scan your code with agent-audit, then attack your deployment with Leshy." |
