---
name: critique-test-suite
description: Analyze the test suite for consistency, completeness, redundancy, parallel safety, and assertion quality. Produces a comprehensive report (no test modifications). Use when the user asks to critique tests, review the test suite, or generate a report for fixing tests.
---

# Critique Test Suite

Analyze all tests in the project and produce a **report only**—do not modify any test files. The report is intended to be used as a prompt for an AI agent (or developer) to fix the tests later.

## Scope

- **Tests:** All files under `tests/` (pytest).
- **Fixtures:** Include `conftest.py` and any shared fixtures in the analysis.
- **Package:** Assume a standard Python package layout (e.g. `src/` with the package under test; tests in `tests/`).

## Checklist for Analysis

Apply these criteria when reviewing each test file and each test case.

### 1. Return values and assertions

- **Explicit values:** Assert exact expected values where known (e.g. `assert result == expected`, not just `assert result` or `assert result is not None`).
- **Dynamic values:** When the value is dynamic (IDs, timestamps), assert **type** and **format** (e.g. regex, enum membership) rather than only existence.
- **Collections:** Prefer asserting **exact length** (e.g. `assert len(items) == 2`) when the expected count is known; avoid only `assert len(items) >= 1` unless the count truly varies.
- **Shape:** For dicts or structured return values, assert expected keys or shape where the contract is defined (e.g. no extra keys, required keys present).

### 2. Success and failure conditions

- **Success paths:** Every behavior under test should have at least one test that asserts the happy-path result (return value or side effect).
- **Failure paths:** For each operation, consider: invalid arguments (TypeError, ValueError), missing data (KeyError, custom exceptions), domain-specific errors. Note missing failure cases in the report.
- **Edge cases:** Empty collections, None/optional values, boundary values (min/max length, zero, negative where invalid).

### 3. Consistency

- **Naming:** Test names should follow a consistent style (e.g. `test_<action>_<condition>_<expected>` or `test_<function>_returns_<value>_when_<condition>`).
- **Structure:** Similar units (e.g. same module or class) should have similar test structure (success, validation error, edge case).
- **Fixtures:** Same concepts (e.g. "sample data", "minimal config") should be reused via fixtures; avoid duplicating setup logic.
- **Assertion style:** Prefer one logical assertion per concept; group related assertions consistently across files.

### 4. Completeness

- **Coverage map:** For each module or public API area, list which behaviors are tested and which are missing.
- **Parameters:** Arguments that affect behavior should have at least one test (valid and, where relevant, invalid).
- **Documentation:** If the project has a spec or docstrings that define behavior, note gaps between documented behavior and tests.

### 5. Redundancy

- **Duplicate coverage:** Identify tests that assert the same behavior in the same way; suggest merging or removing duplicates.
- **Overlap:** Note tests that are subsets of others (e.g. one test checks return type only, another checks return type and value for the same case).
- **Fixtures:** Flag repeated inline setup that could be a shared fixture.

### 6. Parallel execution

- **Isolation:** Tests must not depend on global state, shared mutable objects, or execution order. Note any use of module/class-level mutable state or singletons.
- **Resources:** Note any shared files, caches, or external services that could cause flakiness under `pytest -n auto`.
- **Database:** If the project uses a DB in tests, per-worker schema or transactional rollback should be used; note tests that commit data that could leak to other workers.

### 7. Mocking and dependency isolation

- **External services:** HTTP calls, file I/O to shared paths, or third-party APIs should be mocked in unit tests; note tests that make real external calls.
- **Time-sensitive logic:** Tests involving `datetime.now()`, `time.time()`, or expiration should freeze time (e.g. `freezegun`, `time_machine`) for determinism.
- **Pure logic:** Unit tests for pure business logic should not require a database or network; note functions that could be unit-tested but only have integration tests.
- **Environment variables:** Tests should not depend on real `.env` or env values; note tests that would fail with different env configs.

### 8. Security and input validation

- **Input validation:** Functions that accept user or external input should have tests for invalid input (wrong type, out-of-range, malicious patterns). Note missing validation tests.
- **Sensitive data:** Verify that tests do not log or assert on real secrets; test data should not contain real credentials. Note any exposure risk.
- **Path traversal / injection:** If the code handles paths or structured input, note missing tests for path traversal or injection where relevant.

### 9. Parameterization and data-driven tests

- **`@pytest.mark.parametrize`:** Similar test cases (e.g. multiple invalid inputs) should be parameterized instead of copy-pasted; note repeated test bodies that differ only in input.
- **Boundary values:** For numeric or length-sensitive fields, test min, max, and off-by-one values; note missing boundary tests.
- **Factories:** Test data should be created via factories or fixtures where it reduces duplication or collision risk; note tests with hard-coded values that could be shared.

### 10. Async (if the project uses async)

- **Async fixtures:** Fixtures returning async resources should use `@pytest_asyncio.fixture`; note misuse or sync fixtures in async test files.
- **Timeouts:** Long-running async operations should have explicit timeouts in tests; note tests that could hang.
- **Isolation:** For code that modifies shared state, note whether concurrent access is tested if relevant.

### 11. Output and contract

- **Return shape:** Where the public API defines a return type or shape (e.g. dataclass, TypedDict), tests should assert that shape or key fields; note tests that only spot-check.
- **Exceptions:** Verify that documented or expected exceptions are raised with correct types; note tests that only check "no exception" without testing failure paths.
- **Exception message contents:** When testing exceptions that have defined messages (e.g. validation errors), tests must assert on the **contents** of the exception message, not only that the exception was raised. Use `pytest.raises(SomeError) as exc_info` and assert on `str(exc_info.value)`. Note tests that only check exception type.

### 12. Error handling and messages

- **Error specificity:** Different error conditions should be distinguishable (e.g. by exception type or message); note tests that only check "an exception was raised" without verifying which one.
- **Exception propagation:** For unit tests of code that raises, verify that exceptions are raised with correct types and messages; note missing exception tests.
- **Message assertion:** When exceptions have defined messages, assert on message content (e.g. `pytest.raises(...) as exc_info`, then `assert "expected substring" in str(exc_info.value)`).

### 13. State and workflow

- **State transitions:** For code with status or lifecycle (e.g. state machine, pipeline stage), test valid and invalid transitions; note missing transition tests.
- **Idempotency:** Operations that should be idempotent should be tested for repeated calls; note missing idempotency tests.
- **Side effects:** Actions that trigger side effects (e.g. callbacks, file writes) should verify those occur; note untested side effects.

### 14. Test data and fixtures

- **Realistic data:** Test data should be realistic enough to catch edge cases (e.g. Unicode, long strings); note tests using only trivial data.
- **Cleanup:** Tests that create external resources (files, temp dirs) must clean up; note tests that leak state.
- **Fixture scope:** Fixtures should use the narrowest appropriate scope (`function` > `class` > `module` > `session`); note overly broad scopes that could cause isolation issues.

### 15. Flakiness indicators

- **Time-based assertions:** Tests asserting on wall-clock time are flaky; note and suggest freezing time.
- **Order dependence:** Tests that pass only when run in a specific order indicate shared state; note such patterns.
- **External dependencies:** Tests depending on network, file system state, or external services are flaky in CI; note and suggest mocking.
- **Random data:** Tests using `random` or `uuid4` for assertions without seeding are non-deterministic; note and suggest seeding or fixed values.

### 16. Regression and documentation

- **Bug reference:** Tests written to reproduce bugs should reference the issue in docstring or comment; note regression tests that lack context.
- **Spec alignment:** Tests should map to documented behavior (docstrings, specs); note tests for undocumented behavior or missing tests for documented behavior.
- **Deprecation:** If deprecated APIs exist, tests should verify deprecation warnings where appropriate; note gaps.

### 17. Other good practices

- **Independence:** Each test should be runnable in isolation; document any hidden dependencies (e.g. "must run after X").
- **Clarity:** Test names and docstrings should describe intent; report tests whose purpose is unclear.
- **Speed:** Note slow tests (e.g. many I/O calls, sleeps) that could be sped up with mocks or smaller scope.
- **Assertion messages:** Use clear messages where it helps (e.g. `assert x == y, f"Expected {x} to equal {y}"`); note assertions that would be hard to debug on failure.
- **Single responsibility:** Each test should verify one behavior; note tests that assert unrelated things or have multiple "acts".
- **Arrange-Act-Assert:** Tests should follow AAA pattern; note tests with interleaved setup and assertions.
- **No logic in tests:** Tests should not contain conditionals (`if`), loops, or complex logic; note tests that do and suggest splitting or parameterizing.

### 18. Code coverage

- **Target:** At least 80% line coverage for the package under test (or the project’s stated target).
- **Scope:** Coverage should cover almost all non-exception lines; exception branches may be excluded from the percentage but should still be tested where they represent distinct behavior.
- **Measurement:** Coverage must be checked by running the **entire test suite** (e.g. `pytest tests/ --cov=src --cov-report=term-missing`), not a subset. Note if 80% is met and whether measurement is full-suite.
- **Report:** List modules or packages below the target or with significant uncovered non-exception lines.

## Output: Report Format

Produce a single markdown report with the following structure. Do **not** edit any test files; only write the report.

```markdown
# Test Suite Critique Report

**Generated:** [date]
**Scope:** tests/ (and conftest.py)

## Executive summary
- Overall assessment (strengths, main gaps).
- **Coverage:** At least 80% and almost all non-exception lines; measured by running the **entire test suite**. Note if met and whether measurement is full-suite.
- **Exception messages:** When testing exceptions with defined messages, tests must assert on message contents (e.g. `pytest.raises(...) as exc_info`, `str(exc_info.value)`), not only that the exception was raised.
- High-priority fixes vs. nice-to-have.

## 1. Return values and assertions
[Existence-only asserts; exact length vs >=; shape checks.]

## 2. Success and failure conditions
[Per module/area: what's tested, what's missing (validation, exceptions, edge cases).]

## 3. Consistency
[Naming, structure, fixture usage, assertion style.]

## 4. Completeness
[Coverage map; spec/docstring gaps.]

## 5. Redundancy
[Duplicate or overlapping tests with file:test references.]

## 6. Parallel execution
[Global state, order dependence, shared resources.]

## 7. Mocking and dependency isolation
[Real external calls, time-sensitive tests, env dependencies.]

## 8. Security and input validation
[Missing validation tests, sensitive data, injection/traversal.]

## 9. Parameterization
[Tests that could be parameterized; missing boundary tests.]

## 10. Async (if applicable)
[Async fixture issues, timeouts, isolation.]

## 11. Output and contract
[Return shape, exception types, message assertions.]

## 12. Error handling
[Error specificity; exception message content assertions.]

## 13. State and workflow
[Transitions, idempotency, side effects.]

## 14. Test data and fixtures
[Realistic data, cleanup, fixture scope.]

## 15. Flakiness indicators
[Time, order, external deps, randomness.]

## 16. Regression and documentation
[Bug references, spec alignment, deprecation.]

## 17. Other
[Clarity, speed, assertion messages, AAA, logic in tests.]

## 18. Code coverage
[Target 80%; full-suite measurement; modules below target.]

## Prompt for an AI agent to fix tests

[Self-contained prompt for an AI to apply the fixes. Include:
- Report sections as context.
- **Coverage:** Run coverage using the entire test suite; ensure at least 80% and cover almost all non-exception lines.
- **Exception messages:** When testing exceptions with defined messages, assert on message contents (e.g. `pytest.raises(...) as exc_info`, `str(exc_info.value)`).
- Instruction to fix tests according to the report without changing production code.
- Instruction to preserve existing passing behavior and only add/change assertions and test structure.]
```

## Execution steps

1. **Gather:** List all test files under `tests/` and any `conftest.py`.
2. **Read:** For each file, read test names, docstrings, and assertion patterns (focus on `assert`, return checks, and fixtures).
3. **Classify:** For each criterion (1–18), note specific file names, test names, and line references or short quotes.
4. **Write:** Produce the full report in the format above, including the "Prompt for an AI agent" section at the end.
5. **Do not:** Change, add, or remove any line in any test or conftest file.

## When to use this skill

- User asks to "critique the test suite", "review the tests", "analyze tests", or "generate a report to fix tests".
- User wants a "prompt for an AI to fix the tests" based on the current test suite.
