# AI Testing Swarm

AI Testing Swarm is a **super-advanced, mutation-driven API testing framework** (with optional OpenAPI + OpenAI augmentation) built on top of **pytest**.

It generates a large set of deterministic negative/edge/security test cases for an API request, executes them (optionally in parallel, with retries/throttling), and produces a JSON report with summaries.

> Notes:
> - UI testing is not the focus of the current releases.
> - OpenAI features are **optional** and disabled by default.

---

## Installation

```bash
pip install ai-testing-swarm
```

CLI entrypoint:

```bash
ai-test --help
```

---

## Quick start (cURL input)

Create `request.json`:

```json
{
  "curl": "curl --location https://postman-echo.com/post --header \"Content-Type: application/json\" --data \"{\\\"hello\\\":\\\"world\\\",\\\"count\\\":1}\""
}
```

Run:

```bash
ai-test --input request.json
```

A JSON report is written under:

- `./ai_swarm_reports/<METHOD>_<endpoint>/<METHOD>_<endpoint>_<timestamp>.json`

Reports include:
- per-test results
- summary counts by status code / failure type
- optional AI summary (if enabled)

---

## Input formats

### 1) Raw cURL

```json
{ "curl": "curl ..." }
```

### 2) Normalized request

```json
{
  "method": "POST",
  "url": "https://example.com/api/login",
  "headers": {"content-type": "application/json"},
  "params": {"a": "b"},
  "body": {"username": "u", "password": "p"}
}
```

### 3) OpenAPI-driven (optional)

```json
{
  "openapi": "./openapi.json",
  "path": "/pets",
  "method": "get",
  "headers": {"accept": "application/json"},
  "path_params": {"petId": "123"},
  "query_params": {"limit": 10},
  "body": null
}
```

- OpenAPI **JSON** works by default.
- OpenAPI **YAML** requires `PyYAML` installed.
- Base URL is read from `spec.servers[0].url`.
  - Override with `AI_SWARM_OPENAPI_BASE_URL` if your spec doesn’t include servers.

---

## What test cases are generated?

The swarm always includes:
- **happy_path** (baseline)

Then generates broad coverage across:
- **Method misuse**: same path with wrong HTTP methods (GET/PUT/PATCH/DELETE etc.)
- **Headers**: missing/invalid `Content-Type`, accept variations, and other header tampering
- **Auth** (if `Authorization` header exists): missing/invalid token tests
- **Body/query mutations** (per field):
  - missing / null / empty / whitespace
  - type probes (int/bool/float/array/object)
  - boundary inputs (very long strings, huge ints, negative values)
  - unicode + special character payloads
- **Security payload probes** (per field): SQLi/XSS/path traversal/log4j patterns
- **Whole-body mutations**: null body, empty object, extra unexpected field

> Output is deterministic unless OpenAI augmentation is enabled.

---

## Safety mode (recommended for CI/demos)

Mutation testing can be noisy and may accidentally stress a real environment.
To force safe demo runs only against public test hosts:

```bash
ai-test --input request.json --public-only
```

Or via env:

```bash
export AI_SWARM_PUBLIC_ONLY=1
```

Allowed hosts in public-only mode:
- `httpbin.org`
- `postman-echo.com`
- `reqres.in`

---

## Performance features

### Parallel execution
- Enabled by default via thread pool.
- Control with:
  - `AI_SWARM_WORKERS` (default: `5`)

### Retry + backoff (flaky endpoints)
- Retries on transient errors and status codes (408/429/5xx etc.)
- Control with:
  - `AI_SWARM_RETRY_COUNT` (default: `1`)
  - `AI_SWARM_RETRY_BACKOFF_MS` (default: `250`)

### Throttling (RPS)
- Global throttle to avoid hammering a target:
  - `AI_SWARM_RPS` (default: `0` = disabled)

### Max test cap
- Avoids accidental DoS / CI timeouts:
  - `AI_SWARM_MAX_TESTS` (default: `80`)

---

## Reporting

Reports include:
- `summary.counts_by_failure_type`
- `summary.counts_by_status_code`
- `summary.slow_tests` (based on SLA)
- `summary.risk_score` + `summary.risk_grade`
- `summary.delta_vs_previous` (from endpoint history)

SLA threshold:
- `AI_SWARM_SLA_MS` (default: `2000`)

Security:
- Redaction is optional and OFF by default.
- Enable redaction with:
  - `AI_SWARM_REDACT_ENABLED=1`
- When enabled, sensitive headers are redacted (Authorization/Cookie/api tokens etc.)
- When enabled, sensitive keys in request params/body and response snippets are recursively redacted
- Add extra redaction keys with:
  - `AI_SWARM_REDACT_KEYS` (comma-separated, e.g. `customerEmail,ssn`)

HTML + history:
- Per run HTML: `ai_swarm_reports/<METHOD_endpoint>/<METHOD_endpoint>_<timestamp>.html`
- Endpoint latest: `ai_swarm_reports/<METHOD_endpoint>/latest.html`
- Endpoint history file: `ai_swarm_reports/<METHOD_endpoint>/history.json`
- Project dashboard: `ai_swarm_reports/index.html`
- Project manual-pass overrides: `ai_swarm_reports/manual_pass_overrides.json` (shared across browsers/runs)
- Detailed results support `MANUALLY_PASSED` from report UI, sourced only from project-level `manual_pass_overrides.json` (no browser fallback).
- Owner & escalation links are configurable:
  - `AI_SWARM_OWNER_MAP=GET_quickview=platform-api;*=qa-platform`
  - `AI_SWARM_ENABLE_JIRA_ACTION=1` (`0` disables Jira button in report UI)
  - `AI_SWARM_ENABLE_SLACK_ACTION=1` (`0` disables Slack button in report UI)
  - `AI_SWARM_JIRA_URL_TEMPLATE=https://jira.example.com/create?summary={summary}&description={description}`
  - `AI_SWARM_SLACK_URL_TEMPLATE=https://slack.example.com/notify?text={summary}`
  - Jira/Slack templates support placeholders:
    - `{method}`, `{url}`, `{run_time}`, `{endpoint}`, `{endpoint_name}`, `{owner}`
    - `{decision}`, `{health}`, `{risk_score}`, `{risk_grade}`, `{risk_label}`
    - `{total_tests}`, `{pass_count}`, `{manual_pass_count}`, `{effective_pass_count}`, `{fail_count}`, `{unknown_count}`, `{unknown_status_count}`, `{blocking_count}`
    - `{summary}` and `{description}` are URL-encoded
    - `{summary_raw}` and `{description_raw}` are plain text
- Configure with:
  - `AI_SWARM_HTML_REPORT=1` (default enabled)
  - `AI_SWARM_REPORT_HISTORY_LIMIT=120`
  - `AI_SWARM_OPEN_REPORT=off|dashboard|latest|both` (default `off`)
  - `AI_SWARM_OPEN_REPORT_FORCE=1` to allow auto-open in CI/headless environments

---

## Optional OpenAI augmentation (advanced)

### A) Generate additional test cases (planner augmentation)
Enable:

```bash
export AI_SWARM_USE_OPENAI=1
export OPENAI_API_KEY=... 
export AI_SWARM_MAX_AI_TESTS=30
```

### B) Human-readable AI summary in report
Enable:

```bash
export AI_SWARM_USE_OPENAI=1
export AI_SWARM_AI_SUMMARY=1
export OPENAI_API_KEY=...
```

Model selection:
- `AI_SWARM_OPENAI_MODEL` (default: `gpt-4.1-mini`)

---

## CLI help

```bash
ai-test --help
```

## Plain Pytest Integration Test (No Allure)

Run the offers-list swarm test without Allure dependency:

```bash
AI_SWARM_RUN_LIVE_TESTS=1 PYTHONPATH=src pytest -q tests/test_offers_list_api_swarm_plain.py -s
```

Public API plain tests (no Allure):

```bash
AI_SWARM_RUN_LIVE_TESTS=1 PYTHONPATH=src pytest -q tests/test_public_postman_get_swarm_plain.py -s
AI_SWARM_RUN_LIVE_TESTS=1 PYTHONPATH=src pytest -q tests/test_public_postman_post_swarm_plain.py -s
AI_SWARM_RUN_LIVE_TESTS=1 PYTHONPATH=src pytest -q tests/test_public_httpbin_anything_swarm_plain.py -s
```

For private/preprod offers tests, also set:

```bash
AI_SWARM_RUN_PRIVATE_TESTS=1
```

---

## Release decisions

The swarm produces a release decision:
- `APPROVE_RELEASE`
- `APPROVE_RELEASE_WITH_RISKS`
- `REJECT_RELEASE`

The decision is derived from deterministic rules (not an LLM).

---

## License

MIT
