# E2E Scaled Testing Implementation Plan

> **For Claude:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task.

**Goal:** Build a full-coverage pytest e2e suite covering all MongoClaw features via Python SDK and Node SDK against a live server, plus load/stress scenario packs, producing HTML + JSON reports.

**Architecture:** Single pytest suite under tests/e2e/. NodeSDKClient subprocess wrapper generates temp .mjs scripts calling the Node SDK and parses JSON from stdout. Feature tests parametrized over both clients. Load tests use concurrent.futures.ThreadPoolExecutor.

**Tech Stack:** pytest, pytest-html, pytest-json-report, motor, httpx, Node.js 18+, mongoclaw Python SDK, mongoclaw npm SDK

---

## Prerequisites

1. MongoDB replica set running (rs0)
2. Redis running
3. Node SDK compiled: cd sdk-nodejs && npm run build
4. MongoClaw server running: mongoclaw server start
5. Env vars set: MONGOCLAW_BASE_URL, MONGOCLAW_API_KEY, MONGOCLAW_MONGODB__URI

---

## Task 0: Add test dependencies

**Files:** Modify pyproject.toml

Add to dev extras: pytest-html>=4.0.0, pytest-json-report>=1.5.0
Run: pip install pytest-html pytest-json-report
Commit: chore: add pytest-html and pytest-json-report for e2e reporting

---

## Task 1: Create tests/e2e/conftest.py

**Key fixtures:**
- py_client (session): MongoClawClient(base_url, api_key) - asserts is_healthy() on init
- node_client (session): NodeSDKClient wrapping sdk-nodejs/dist/index.js
- mongo_direct (session): AsyncIOMotorClient for direct DB assertions
- cleanup_agents (session, autouse): deletes all registered e2e agent IDs at teardown
- uid(prefix): returns unique ID like e2e_abc12345
- wait_for_exec(py_client, execution_id, timeout=60): polls until completed/failed
- wait_for_any_exec(py_client, agent_id, timeout=60): polls until any execution exists
- base_agent_config(agent_id, db, collection, strategy, target_field, consistency_mode, policy, max_concurrency, operations): returns minimal agent dict

Reads from env: MONGOCLAW_BASE_URL (default http://localhost:8000), MONGOCLAW_API_KEY, MONGOCLAW_MONGODB__URI, E2E_EXEC_TIMEOUT (default 60)

Commit: test(e2e): add e2e conftest with live fixtures and helpers

---

## Task 2: Create tests/e2e/node_adapter.py

Class NodeSDKClient(base_url, api_key, sdk_dist, tmp_dir):
- _run(script): writes script to tmp .mjs file, runs node, parses JSON stdout
- _header(): returns ES module import + client init as string
- Methods mirroring MongoClawClient: health, is_healthy, create_agent, get_agent,
  list_agents, update_agent, delete_agent, enable_agent, disable_agent,
  validate_agent, trigger_agent, get_execution, list_executions, wait_for_execution
- Each method calls _run with: header + client.methodName(JSON.stringify(args)) + console.log(JSON.stringify(r))

Raises NodeSDKError on non-zero exit or bad JSON.

Commit: test(e2e): add NodeSDKClient subprocess adapter for Node SDK

---

## Task 3: Create tests/e2e/test_A_health.py

TestA1Health:
- test_basic_health_python: py_client.health().status == healthy
- test_liveness_probe: GET /health/live -> 200
- test_readiness_probe: GET /health/ready -> 200
- test_detailed_health_python: health_detailed().components has mongodb key
- test_basic_health_node: node_client.health()[status] == healthy
- test_detailed_health_node: GET /health/detailed -> status in (healthy, degraded)

TestA2Metrics:
- test_metrics_endpoint_available: get_metrics()[raw] is non-empty
- test_metrics_contains_expected_keys: mongoclaw_ prefix present in raw
- test_metrics_parseable: non-comment lines exist

Run: pytest tests/e2e/test_A_health.py -v -m e2e
Commit: test(e2e): A1+A2 health and metrics endpoint tests

---

## Task 4: Create tests/e2e/test_B_method_coverage.py

TestB1CLI (subprocess calls to mongoclaw CLI):
- test_cli_agent_lifecycle: create (from yaml) -> list -> disable -> enable -> delete
- Assert returncode==0 and agent_id in stdout for list

TestB2PythonSDK:
- test_python_sdk_full_lifecycle: create -> get -> list -> update -> disable -> enable -> validate -> trigger + wait -> delete

TestB3NodeSDK:
- test_node_sdk_full_lifecycle: create -> get -> list -> disable -> enable -> trigger + wait -> delete

TestB4RESTAPIParity (httpx direct):
- test_rest_create_get_delete: POST /api/v1/agents -> GET -> POST /disable -> DELETE
- test_rest_validate: POST /api/v1/agents/validate -> 200

Run: pytest tests/e2e/test_B_method_coverage.py -v -m e2e
Commit: test(e2e): B1-B4 method coverage (CLI, Python SDK, Node SDK, REST)

---

## Task 5: Create tests/e2e/test_C_ai_execution.py

TestC1InsertDriven:
- test_insert_triggers_enrichment_python: trigger_agent -> wait -> assert status==completed, written==True
- test_insert_triggers_enrichment_node: same via node_client

TestC2UpdateDriven:
- test_update_operation_triggers_agent: agent with operations=[insert,update] -> trigger -> assert terminal

TestC3SchemaGuided:
- test_structured_response_written: base config (has response_schema) -> trigger -> written==True

TestC4WriteStrategies (parametrize over merge/replace/append + target_field combinations):
- test_write_strategy: for each strategy combo -> trigger -> completed, written==True
- test_write_strategy_node: same parametrized via node_client

Run: pytest tests/e2e/test_C_ai_execution.py -v -m e2e
Commit: test(e2e): C1-C4 AI execution and write strategy tests

---

## Task 6: Create tests/e2e/test_D_consistency.py

TestD1Eventual:
- test_eventual_high_throughput: 5 triggers with consistency_mode=eventual -> >=4 completed

TestD2StrictPostCommit:
- test_strict_mode_records_execution: single trigger -> lifecycle_state not None
- test_strict_concurrent_at_least_one_wins: 2 concurrent triggers on same doc -> at least 1 completed

TestD3Shadow:
- test_shadow_mode_skips_write: consistency_mode=shadow -> completed, written==False
- test_shadow_mode_node: same via node_client

TestD4HashGuard:
- test_hash_guard_enabled: require_document_hash_match=True -> terminal with lifecycle_state

Run: pytest tests/e2e/test_D_consistency.py -v -m e2e
Commit: test(e2e): D1-D4 consistency mode tests

---

## Task 7: Create tests/e2e/test_E_loop_idempotency.py

TestE1LoopGuard:
- test_no_infinite_cascade: trigger -> wait -> sleep 3s -> list_executions -> assert count < 10
- test_loop_guard_metrics_present: mongoclaw_ in metrics raw

TestE2Idempotency:
- test_duplicate_trigger_bounded: trigger same doc twice -> both terminal, at least 1 written

Run: pytest tests/e2e/test_E_loop_idempotency.py -v -m e2e
Commit: test(e2e): E1-E2 loop guard and idempotency tests

---

## Task 8: Create tests/e2e/test_F_resilience.py

Helper _trigger_many(py_client, agent_id, n, timeout): triggers n docs concurrently, waits for all.

TestF1BaselineCRUD:
- test_concurrent_inserts_stable: 10 concurrent triggers -> >=8 completed
- test_runtime_stays_healthy_after_load: is_healthy() still True

TestF2BurstRetries:
- test_burst_bounded_failures: 15 triggers with max_retries=2, timeout=10s -> >=10 terminal, not all failed

TestF3Backpressure:
- test_backpressure_metrics_healthy: max_concurrency=1, 20 rapid triggers -> metrics still responsive

TestF4DLQ:
- test_dlq_metrics_present: dlq or dead in metrics raw

TestF5Quarantine:
- test_failing_agent_recorded: nonexistent model, max_retries=0, 5 triggers -> >=1 failed

Run: pytest tests/e2e/test_F_resilience.py -v -m e2e
Commit: test(e2e): F1-F5 resilience and backpressure tests

---

## Task 9: Create tests/e2e/test_G_policy.py

Helper _policy_cfg(agent_id, condition, action, fallback, simulate=False):
  - Uses mock_response with risk_score=0.95
  - Sets policy block with condition, action, fallback_action, optional simulate

TestG1PolicyActions (parametrize over enrich/block/tag):
- test_policy_action: each action -> trigger -> terminal status
- test_policy_node_sdk: enrich action via node_client -> terminal

TestG2PolicySimulation:
- test_simulation_mode: simulate=True -> terminal with lifecycle_state

Run: pytest tests/e2e/test_G_policy.py -v -m e2e
Commit: test(e2e): G1-G2 policy evaluation and simulation tests

---

## Task 10: Create tests/e2e/test_H_observability.py

TestH1ExecutionLifecycle:
- test_execution_record_has_all_fields: id, agent_id, document_id, status, lifecycle_state,
  started_at, completed_at, duration_ms all present and non-None
- test_execution_list_filtered_by_agent: list_executions(agent_id=X) returns only that agents records
- test_execution_stats_endpoint: GET /api/v1/executions/stats -> 200 dict
- test_execution_lifecycle_node: node trigger -> id, status, lifecycle_state all set

TestH2MetricsCompleteness:
- test_execution_counters_increment: trigger -> wait -> get_metrics() still non-empty
- test_metric_families_present: mongoclaw_ prefix in metrics raw

Run: pytest tests/e2e/test_H_observability.py -v -m e2e
Commit: test(e2e): H1-H2 observability and lifecycle field tests

---

## Task 11: Create tests/e2e/test_I_catalog.py

TestICatalog:
- test_catalog_overview: GET /api/v1/catalog/overview -> 200 dict
- test_catalog_overview_node: same endpoint reachable
- test_collection_profile_reachable: GET /api/v1/catalog/collection-profile -> 200 or 404
- test_catalog_shows_registered_agents: create agent then catalog -> 200

Run: pytest tests/e2e/test_I_catalog.py -v -m e2e
Commit: test(e2e): I1-I2 catalog overview and collection profile tests

---

## Task 12: Create tests/e2e/test_scenarios.py

Env overrides: E2E_LOAD_AGENTS=3, E2E_LOAD_DOCS=10, E2E_LOAD_CONCURRENCY=5,
  E2E_LOAD_TIMEOUT=120, E2E_MIN_SUCCESS_RATE=0.85, E2E_LATENCY_SLO_MS=15000

Helper _run_load(py_client, agents, docs_per_agent, doc_factory, concurrency, timeout):
  triggers all (agent x doc) combos concurrently, waits for all, returns results list

TestPack1SupportTickets (mark: e2e, load):
- test_ticket_triage_throughput: LOAD_AGENTS agents x LOAD_DOCS docs -> success_rate >= MIN
- test_ticket_triage_latency_slo: 5 sequential triggers -> p95 duration_ms <= LATENCY_SLO_MS

TestPack2OrderRisk (mark: e2e, load):
- test_order_risk_concurrent_strict: strict_post_commit agents x docs -> success_rate >= MIN-0.1

TestPack3BurstStress (mark: e2e, load):
- test_burst_no_collapse: burst=LOAD_DOCS*2 with 2x concurrency -> valid>=burst//2,
  sample terminal >= sample//2, is_healthy() after

TestPack4PolicyFlow (mark: e2e, load):
- test_policy_audit_trail: LOAD_DOCS policy triggers -> all terminal, all have lifecycle_state

Run: pytest tests/e2e/test_scenarios.py -v -m load
Commit: test(e2e): scenario packs 1-4 (tickets, orders, burst, policy)

---

## Task 13: Run full suite and generate reports

Feature tests:
  TIMESTAMP=20260226_230007
  mkdir -p tests/e2e/reports/
  pytest tests/e2e/ -v -m "e2e and not load"
    --html=tests/e2e/reports//report.html --self-contained-html
    --json-report --json-report-file=tests/e2e/reports//report.json

Load tests:
  E2E_LOAD_DOCS=20 E2E_LOAD_AGENTS=5
  pytest tests/e2e/test_scenarios.py tests/e2e/test_F_resilience.py -v -m load

Final commit: test(e2e): complete e2e scaled test suite

---

## Quick Reference

  pytest tests/e2e/ -v -m "e2e and not load"    # all feature tests
  pytest tests/e2e/ -v -m load                   # load tests
  pytest tests/e2e/test_C_ai_execution.py -v     # single file
  E2E_LOAD_DOCS=50 pytest tests/e2e/ -v -m load  # custom params
