# Testing Coverage & Improvement Analysis

**Status**: Proposal
**Created**: 2026-02-08
**Scope**: Test coverage mapping, test quality, configuration, missing test categories

---

## Executive Summary

Prism has **solid test infrastructure** with **1,829 test functions** across **155 test files** totaling 20,289 lines. The codebase demonstrates strong foundations with well-designed fixtures and comprehensive specification testing. However, **critical gaps exist** in infrastructure generator testing, generated code validation (no snapshot/golden file testing), and integration testing.

**Test Health Score: 7.5/10**

---

## Coverage Map

| Category | Files | Tests | Status |
|----------|-------|-------|--------|
| Spec models | 14 | ~200 | Complete |
| CLI commands | 18 | ~200 | Complete |
| Backend generators | 13/18 | ~400 | **Gaps** |
| Frontend generators | 21/23 | ~350 | **Minor gaps** |
| Infrastructure generators | 0/1 | 0 | **Missing** |
| E2E tests | 4 | ~30 | Minimal |
| Integration tests | Limited | — | **Insufficient** |
| Performance tests | 0 | 0 | **Missing** |
| Error path coverage | 73/1829 (4%) | — | **Low** |

---

## Critical Gaps (Phase 1)

### 1. Infrastructure Generator — Zero Tests

**Source**: `src/prisme/generators/infrastructure/` — No test coverage whatsoever.

**Missing file**: `tests/generators/infrastructure/test_authentik_generator.py`

**Impact**: Infrastructure generation is completely untested. Changes could silently break deployment configs.

**Effort**: 1 week (~350 lines, 12 tests)

### 2. GraphQL & MCP Generator Tests — Missing

**Sources**:
- `src/prisme/generators/backend/graphql.py` — Only indirect testing via `test_conditional.py`
- `src/prisme/generators/backend/mcp.py` — Only indirect testing

**Missing files**:
- `tests/generators/backend/test_graphql_generator.py` (~450 lines, 18 tests)
- `tests/generators/backend/test_mcp_generator.py` (~400 lines, 16 tests)

**Impact**: Two of the most complex generators lack dedicated tests.

### 3. No Snapshot/Golden File Testing

**Current state**: `tests/e2e/test_template_golden.py` (170 lines) validates linting passes but does NOT compare generated output against expected baselines.

**Missing**:
- Tests that validate generated file structure matches expectations
- Tests that verify type hints are correct
- Tests that confirm imports work
- Regression detection for output changes

**Recommendation**: Implement golden file framework in `tests/test_generated_output/` comparing generated code against checked-in baselines.

### 4. Integration Pipeline Tests — Insufficient

**Missing**: End-to-end validation of Spec → Generate → Importable Code pipeline.

**Needed scenarios**:
- Single model CRUD: spec → backend code → valid Python
- Relationships (1:1, 1:N, N:N): correct foreign keys and joins
- Authentication: proper middleware and route protection
- Complex field types: all 13 FieldTypes generate valid code

**Recommended file**: `tests/integration/test_spec_to_code_pipeline.py` (~400 lines, 12 tests)

---

## Undertested Generators (Phase 2)

### Models Generator — Minimal

**File**: `tests/generators/backend/test_models_generator.py` (79 lines, 2 tests)

Only tests DateTime timezone validation. Should cover:
- All 13 field types generate valid SQLAlchemy columns
- Relationship generation (FK, M2M)
- Soft delete mixin
- Timestamp mixins
- Model inheritance

**Target**: Expand to 220+ lines, 15+ tests.

### REST Generator — Minimal

**File**: `tests/generators/backend/test_rest_generator.py` (88 lines, 2 tests)

Only tests auth router inclusion. Should cover:
- Route generation for all CRUD operations
- HTTP method mapping
- Error response schemas
- Pagination parameters
- Filter parameters

**Target**: Expand to 240+ lines, 18+ tests.

---

## Test Quality Assessment

### Strengths

| Component | Score | Notes |
|-----------|-------|-------|
| `tests/conftest.py` | 5/5 | Well-designed fixture hierarchy (sample_field_spec → sample_model_spec → sample_stack_spec) |
| `tests/generators/test_base.py` | 5/5 | 38 tests covering base classes, edge cases, dry run, failure modes |
| `tests/tracking/test_protected_regions.py` | 5/5 | 200+ lines, multi-language support, merge conflict handling |
| `tests/spec/` | 4.5/5 | 14 files, 1,200+ lines covering validators, field types, relationships |
| `tests/cli/` | 4/5 | 18 files, good command coverage |

### Weaknesses

- **Error path coverage is only 4%** (73/1829 tests). Most tests only cover happy paths.
- Template testing (`tests/test_templates/test_template_syntax.py`, 61 lines) only validates Jinja2 syntax — no variable substitution, conditional logic, or filter testing.
- No snapshot testing means output regressions go undetected.

---

## Pytest Configuration Improvements

**File**: `pyproject.toml` (lines 112-128)

Current config is functional but missing:

| Setting | Current | Recommended |
|---------|---------|-------------|
| Coverage threshold | Not set | `--cov-fail-under=75` |
| Coverage reporting | Not configured | Add `[tool.coverage.report]` section |
| Unit/integration markers | Not defined | Add `unit`, `integration` markers |
| Test collection filters | Not set | Consider organizing by speed tier |

---

## Error Scenario Tests — Systematically Missing

Only 4% of tests cover error paths. Each generator test file should have a `TestXGeneratorErrors` class covering:

- Circular relationships
- Duplicate field names
- Invalid constraint values
- Missing required relationships
- Invalid GraphQL naming conventions
- Type mismatches
- Empty model specs

---

## Files to Create

| File | Lines | Tests | Priority |
|------|-------|-------|----------|
| `tests/generators/infrastructure/test_authentik_generator.py` | 350 | 12 | Critical |
| `tests/generators/backend/test_graphql_generator.py` | 450 | 18 | Critical |
| `tests/generators/backend/test_mcp_generator.py` | 400 | 16 | Critical |
| `tests/integration/test_spec_to_code_pipeline.py` | 400 | 12 | Critical |
| `tests/test_generated_output/test_models_golden.py` | 250 | 8 | High |
| `tests/test_generated_output/test_services_golden.py` | 250 | 8 | High |
| `tests/performance/test_generation_perf.py` | 200 | 5 | Medium |

## Files to Expand

| File | Current Lines | Target Lines | Additional Tests |
|------|---------------|--------------|-----------------|
| `tests/generators/backend/test_models_generator.py` | 79 | 220+ | +15 |
| `tests/generators/backend/test_rest_generator.py` | 88 | 240+ | +18 |
| `tests/test_templates/test_template_syntax.py` | 61 | 150+ | +10 |

---

## Impact Projection

| Metric | Current | After Phase 1 | After All Phases |
|--------|---------|---------------|-----------------|
| Generator coverage | 80% | 100% | 100% |
| Error path tests | 4% (73) | 8% (~150) | 12%+ (220+) |
| Golden file tests | 0 | 5+ suites | 10+ suites |
| Integration tests | Limited | Full pipeline | Full pipeline |
| Overall score | 7.5/10 | 8.5/10 | 9.5/10 |

---

## Effort Summary

| Phase | Scope | Effort |
|-------|-------|--------|
| Phase 1 (Critical) | Infrastructure, GraphQL, MCP, integration, golden files | 2 weeks |
| Phase 2 (Important) | Expand minimal tests, error scenarios, config improvements | 1-2 weeks |
| Phase 3 (Enhancement) | Performance tests, template rendering, additional golden files | 1 week |
| **Total** | | **3-4 weeks** |

---

**Analysis Date**: 2026-02-08
