# AIandMe CLI (Beta)

> CLI-first security testing for AI agents and chatbots. Adversarial attacks, behavioral QA, posture scoring, and guardrails export — from your terminal to your CI/CD pipeline.

[![PyPI](https://img.shields.io/pypi/v/aiandme-cli)](https://pypi.org/project/aiandme-cli/)
[![License](https://img.shields.io/badge/license-proprietary-blue)]()

```
pip install aiandme-cli
```

---

## Overview

AIandMe runs automated adversarial attacks against your bot's live endpoint, evaluates responses using LLM-as-a-judge, and produces structured findings aligned with the [OWASP Top 10 for LLM Applications](https://owasp.org/www-project-top-10-for-large-language-model-applications/) and the [OWASP Agentic AI Threats](https://genai.owasp.org/resource/agentic-ai-threats-and-mitigations/).

### Platform Services

| Service | Description |
|---------|-------------|
| **CLI Tool** | Full-featured command line interface. Initialize projects, run tests, check posture, export guardrails. |
| **pytest Plugin** | Native pytest integration with markers, fixtures, and baseline comparison. Run security tests alongside unit tests. |
| **Adversarial Testing** | OWASP-aligned attack scenarios: single-turn, multi-turn, adaptive, and agentic. |
| **Behavioral Testing** | Validate intent boundaries, response quality, and functional correctness. |
| **Posture Scoring** | Quantified 0–100 security score with breakdown by findings, coverage, and resilience. Track over time. |
| **Guardrails Export** | Generate protection rules from test findings. Export to OpenAI, Azure AI Content Safety, AWS Bedrock, or AIandMe format. |

### Why AIandMe?

Manual red-teaming doesn't scale. Static analysis can't catch runtime behavior. Generic pentesting tools don't understand LLM-specific attack vectors like prompt injection, jailbreaks, or tool abuse.

AIandMe is built for this. Point it at your bot's endpoint, define the scope (or let it extract one from your system prompt), and get a structured security report with actionable findings — all mapped to OWASP LLM and Agentic AI categories.

Testing feeds into hardening: export guardrails, track posture across releases, and catch regressions before they reach production. Works with any chatbot or agent, cloud or on-prem.

---

## Quick Start

```bash
# 1. Install & authenticate
pip install aiandme-cli
aiandme login

# 2. Initialize a project
aiandme init --name "My Bot" --prompt ./system_prompt.txt

# 3. Run security tests
aiandme test \
  --chat-endpoint https://my-bot.com/chat \
  --chat-header "Authorization: Bearer token"

# 4. Review results
aiandme logs
aiandme posture
```

---

## Test Categories

| Category | Mode | Description |
|----------|------|-------------|
| `owasp_single_turn` | Adversarial | Single-prompt attacks: prompt injection, jailbreaks, data exfiltration. Fast coverage of basic vulnerabilities. |
| `owasp_multi_turn` | Adversarial | Conversational attacks that build context over multiple turns. Tests context manipulation and gradual escalation. |
| `owasp_agentic_multi_turn` | Adversarial | Targets tool-using agents. Tests goal hijacking, tool misuse, and privilege escalation. |
| `behavioral` | QA | Intent boundary validation and response quality testing. Ensures agent behaves within defined scope. |

**Adaptive mode:** Both `owasp_multi_turn` and `owasp_agentic_multi_turn` support an adaptive flag that enables evolutionary search — the attack strategy adapts based on bot responses instead of following scripted prompts.

### Testing Levels

| Level | Description |
|-------|-------------|
| `unit` | Standard coverage (~20 min) — default |
| `system` | Deep testing (~45 min) |
| `acceptance` | Full coverage (~90 min) |

---

## pytest Integration

Run security tests alongside your existing test suite with native pytest markers and fixtures.

```python
# test_security.py
import pytest

@pytest.mark.aiandme
def test_prompt_injection(aiandme):
    """Test prompt injection defenses."""
    result = aiandme.test("llm001")
    assert result.passed, f"Failed: {result.findings}"

@pytest.mark.aiandme
def test_posture_threshold(aiandme_posture):
    """Ensure posture meets minimum."""
    assert aiandme_posture["score"] >= 70

@pytest.mark.aiandme
def test_no_regressions(aiandme, aiandme_baseline):
    """Compare against baseline."""
    result = aiandme.test("llm001")
    if aiandme_baseline:
        regressions = result.compare(aiandme_baseline)
        assert not regressions
```

```bash
# Run with AIandMe enabled
pytest --aiandme tests/

# Filter by category
pytest --aiandme --aiandme-category=adversarial

# Set failure threshold
pytest --aiandme --aiandme-fail-on=high

# Compare to baseline
pytest --aiandme --aiandme-baseline=baseline.json

# Save new baseline
pytest --aiandme --aiandme-save-baseline=baseline.json
```

---

## CI/CD Integration

Block insecure deployments automatically with exit codes.

```
Build → Unit Tests → AI Security (aiandme test) → Deploy
```

```yaml
# .github/workflows/security.yml
name: AI Security Tests
on: [push, pull_request]

jobs:
  security:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: pip install aiandme
      - name: Run Security Tests
        env:
          AIANDME_API_KEY: ${{ secrets.AIANDME_API_KEY }}
        run: |
          aiandme test --wait --fail-on=high
```

---

## Usage

```
aiandme [--base-url URL] COMMAND [OPTIONS] [ARGS]
```

### Authentication

| Command | Description |
|---------|-------------|
| `login` | Authenticate via browser (OAuth PKCE) |
| `logout` | Clear stored credentials |
| `whoami` | Show current authentication status |

### Organisation Management

| Command | Description |
|---------|-------------|
| `orgs list` | List available organisations |
| `orgs current` | Show current organisation |
| `switch <id>` | Switch to organisation |

### Provider Management

Providers are LLM configurations used for running security tests.

| Command | Description |
|---------|-------------|
| `providers list` | List configured providers |
| `providers add` | Add new provider |
| `providers update <id>` | Update provider config |
| `providers remove <id>` | Remove provider |

<details>
<summary><code>providers add</code> options</summary>

```
--name, -n        Provider name: openai, claude, azureopenai, gemini, grok, custom
--api-key, -k     API key
--endpoint, -e    Endpoint URL (required for azureopenai, custom)
--model, -m       Model name (optional)
--default         Set as default provider
--interactive     Interactive configuration mode
```

</details>

### Project Management

| Command | Description |
|---------|-------------|
| `projects list` | List projects |
| `projects use <id>` | Select project |
| `projects current` | Show current project |
| `projects show [id]` | Show project details |

<details>
<summary><code>init</code> — create project from various sources</summary>

```
aiandme init --name NAME [OPTIONS]

--prompt PATH       Extract scope from system prompt file
--endpoint URL      Probe live chatbot endpoint
--repo PATH         Scan repository for prompts/tools
--openapi PATH      Parse OpenAPI specification
--description       Project description
--yes, -y           Skip confirmation prompts
```

</details>

### Test Execution

<details>
<summary><code>test</code> — run security tests on current project</summary>

```
aiandme test [OPTIONS]

Test Category:
  --test-category, -t   Test to run (default: owasp_multi_turn)
                        Values: owasp_single_turn, owasp_multi_turn,
                                owasp_agentic_multi_turn, behavioral

Testing Level:
  --testing-level, -l   Depth of testing (default: unit)
                        unit | system | acceptance

Chat Endpoint (required):
  --chat-endpoint       Chat completion URL of the bot to test
  --chat-header         Header for chat endpoint (repeatable)
  --chat-payload        JSON payload template for chat

Init Endpoint (optional):
  --init-endpoint       Thread initialization URL
  --init-header         Header for init endpoint (repeatable)
  --init-payload        JSON payload for init

Auth Endpoint (optional):
  --auth-endpoint       Auth/token endpoint URL
  --auth-header         Header for auth endpoint (repeatable)
  --auth-payload        JSON payload for auth

Other:
  --provider-id         Provider to use (default: first available)
  --name, -n            Experiment name (auto-generated if omitted)
  --lang                Language (default: english). Accepts codes: en, de, es...
  --adaptive            Enable adaptive mode (evolutionary attack strategy)
  --streaming           Enable streaming mode (requires wss:// endpoint)
  --no-auto-start       Create without starting (manual mode)
  --wait, -w            Wait for completion
  --fail-on SEVERITY    Exit non-zero if findings >= severity
                        Values: critical, high, medium, low, any
```

</details>

### Experiment Management

| Command | Description |
|---------|-------------|
| `experiments list` | List experiments |
| `experiments show <id>` | Show experiment details |
| `experiments status <id>` | Check status |
| `experiments status <id> --watch` | Watch until completion |
| `experiments wait <id>` | Wait with progressive backoff (30s → 60s → 120s → 300s) |
| `experiments logs <id>` | List experiment logs |
| `experiments report <id>` | Download HTML report |

`status` is also available as a top-level alias — without an ID it shows the most recent experiment:

```bash
aiandme status [experiment_id] [--watch]
```

### Results & Export

```bash
# View experiment results (table, json, or csv)
aiandme logs [experiment_id] [--format table] [--verdict pass|fail] [--page N] [--size N]

# Security posture score
aiandme posture [--json]

# Export guardrails configuration
aiandme guardrails [--vendor aiandme|openai|azure|bedrock] [--format json|yaml] [-o FILE]
```

### Documentation

```bash
aiandme docs
```

Opens documentation in browser.

---

## Examples

### Basic workflow

```bash
aiandme login
aiandme switch abc123
aiandme projects use def456
aiandme test --chat-endpoint https://bot.com/chat
aiandme status --watch
aiandme logs
```

### Multi-endpoint bot with auth

```bash
aiandme test \
  --auth-endpoint https://bot.com/oauth/token \
  --auth-payload '{"client_id":"x","client_secret":"y"}' \
  --init-endpoint https://bot.com/threads \
  --init-header "Content-Type: application/json" \
  --chat-endpoint https://bot.com/chat \
  --chat-header "Content-Type: application/json" \
  --chat-payload '{"messages":[{"role":"user","content":"$PROMPT"}]}'
```

### Export guardrails

```bash
aiandme guardrails --vendor openai --format json -o guardrails.json
```

### Create project from repository

```bash
aiandme init \
  --name "Support Bot" \
  --repo ./my-agent \
  --prompt ./prompts/system.txt
```

---

## Configuration

### Environment Variables

| Variable | Description | Default |
|----------|-------------|---------|
| `AIANDME_BASE_URL` | API base URL | `https://api.aiandme.io` |
| `AIANDME_AUTH0_DOMAIN` | Auth0 domain (on-prem) | `aiandme.eu.auth0.com` |
| `AIANDME_AUTH0_CLIENT_ID` | Auth0 client ID (on-prem) | — |

### On-Premises

```bash
export AIANDME_BASE_URL=https://api.your-domain.com
aiandme login
```

### Files

| Path | Description |
|------|-------------|
| `~/.aiandme/` | Configuration directory |
| `~/.aiandme/credentials.json` | Auth tokens (mode `600`) |

---

## Exit Codes

| Code | Meaning |
|------|---------|
| `0` | Success |
| `1` | Error or test failure (with `--fail-on`) |

---

## Links

- [Documentation](https://docs.aiandme.io)
- [GitHub](https://github.com/aiandme-io/aiandme-cli)
