Analysis: Caddy Config Testing Strategy

### What needs testing

Your configs define 5 distinct traffic paths, each with different TLS and routing behavior:

| Path | TLS Method | Template | Key Behavior |
| :--- | :--- | :--- | :--- |
| **CF-proxied infra** | DNS-01 (Cloudflare) | canonical (not yet built) | CF terminates, re-encrypts to origin |
| **Direct infra** | TLS-ALPN | canonical | Caddy gets its own cert |
| **On-demand (customer)** | on_demand + ask endpoint | canonical | Cert issued on first request |
| **Appx.app relay** | Pre-terminated by approx. | canonical | Routes on `Apx-Incoming-Host` header |
| **Custom install (.ge.)** | DNS-01 (Cloudflare) | Caddyfile-custom.template | Independent server |

For each path, you need to verify: TLS handshake, correct headers reaching Puma (`O-Via`, `O-Request-ID`, `Apx-Incoming-Host` forwarding, security headers), CORS for allowed origins, health checks, security blocking (sensitive paths, bad UAs, path traversal), and HTTP→HTTPS redirect with ACME passthrough.

> [!IMPORTANT]
> **Insight:** The challenge here is that Caddy config testing spans at least three different scopes with fundamentally different requirements:
> - **Syntax/semantic** — does the config parse and adapt?
> - **Routing logic** — do requests reach the right handler with correct headers?
> - **TLS issuance** — do certs actually get issued via the right ACME challenge method?
>
> You can't meaningfully test TLS issuance locally, and you can't safely test on-demand cert behavior in production. This is why the `.dev` domains are so valuable — they give you a real ACME environment without touching production DNS.

---

### Layer 1: Static validation (no network, runs in CI)

**What exists now:** `caddy validate` on rendered templates. This catches syntax errors and some semantic issues.

**What you can add:**

- `caddy adapt --config X --adapter caddyfile` — converts Caddyfile to Caddy's native JSON. This is more rigorous than `validate` because it fully resolves imports, snippets, and placeholders. You can diff the JSON output between old monolithic configs and new snippet-based configs to verify behavioral equivalence during the refactoring.
- **Template rendering assertions** — `envsubst` the templates and check that no `$VARIABLE` placeholders survive in the output. Simple `grep '\$[A-Z]' rendered.conf` catches missing env vars.
- **A Makefile or shell script** that renders all template variants and validates each:

```bash
# Pseudostructure
for template in Caddyfile-custom.template Caddyfile-canonical.template; do
  envsubst < "$template" > /tmp/rendered
  caddy validate --config /tmp/rendered
  caddy adapt --config /tmp/rendered --adapter caddyfile > /tmp/adapted.json
  # assert no unsubstituted vars
  # assert key JSON paths exist (on_demand_tls, health checks, etc.)
done
```

---

### Layer 2: Local integration (Caddy runs, no real DNS/certs)

Caddy has two features that make local testing realistic:

1.  `tls internal` — Caddy runs its own CA and issues self-signed certs. Combined with `caddy trust`, you get valid-looking TLS locally.
2.  `local_certs` global option — applies `tls internal` to all sites. You can create a test overlay:

```caddy
{
    local_certs
    # ... rest of global config
}
```

Then use `curl` with `--resolve` to test routing without touching DNS:

```bash
# Simulate Cloudflare-proxied request
curl -k --resolve "direct.us.onetime.dev:443:127.0.0.1" \
  https://direct.us.onetime.dev/health

# Simulate approximated.app relay
curl -k --resolve "us.onetime.dev:443:127.0.0.1" \
  -H "Apx-Incoming-Host: secrets.customer.com" \
  https://us.onetime.dev/api/v2/status

# Test CORS preflight
curl -k --resolve "direct.us.onetime.dev:443:127.0.0.1" \
  -X OPTIONS \
  -H "Origin: https://onetimesecret.dev" \
  -H "Access-Control-Request-Method: POST" \
  https://direct.us.onetime.dev/api/v2/secret/conceal -D -

# Test security blocking
curl -k --resolve "direct.us.onetime.dev:443:127.0.0.1" \
  https://direct.us.onetime.dev/.env # expect 404
```

For the **ask endpoint** (on-demand TLS), you can mock it or test it independently:

```bash
curl "http://localhost:12020/ask?domain=allowed-customer.com"   # expect 200
curl "http://localhost:12020/ask?domain=random-attacker.com"    # expect 4xx
```

> [!TIP]
> **Insight:** The `ask` endpoint is your gatekeeper for on-demand TLS. Testing this endpoint thoroughly is arguably more important than testing the TLS flow itself, because a misconfigured `ask` endpoint lets attackers burn your Let's Encrypt rate limits.

---

### Layer 3: Staging environment with .dev domains (real DNS, real certs)

This is where the `.dev` domains pay off. The approach:

1.  **DNS mapping** — Use Cloudflare API to point `.dev` records at a staging server.
2.  **Let's Encrypt staging CA** — Use `acme_ca https://acme-staging-v02.api.letsencrypt.org/directory` to avoid rate limits.
3.  **Template parameterization** — Swap the domain suffix during rendering:
    ```bash
    HOSTNAME=us-staging-web-01 JURISDICTION=us SITE_SLUG=B76s2 \
      envsubst < Caddyfile-canonical.template > /etc/caddy/Caddyfile
    ```
4.  **On-demand TLS testing** — Create a throwaway DNS A record for a test domain pointing at staging to verify the full flow.

**Cloudflare DNS automation tools:**

| Tool | Strengths | Good for |
| :--- | :--- | :--- |
| **flarectl** | Simple, official CLI | One-off DNS management |
| **curl + API** | Zero dependencies | CI/CD, shell scripts |
| **Terraform** | Declarative, versioned | Persistent staging setup |
| **Pulumi** | Real code (TS/Go/Python) | Programmatic infrastructure |

---

### Layer 4: What you could build

The most practical artifact is a single test script (`etc/caddy/test-configs.sh`) that covers layers 1-3:

- `validate()` — Layer 1: render + validate + adapt.
- `test-local()` — Layer 2: start Caddy with `local_certs`, run curl assertions.
- `test-staging()` — Layer 3: assumes `.dev` DNS is pointed, runs full integration.
- `test-smoke()` — Layer 4: post-deploy production smoke tests.

---

### Existing `rots` commands

The following commands are already implemented or in progress. They provide the
building blocks for the testing layers described above.

#### Proxy commands (`rots proxy ...`)

| Command | Status | Layer | Description |
| :--- | :--- | :--- | :--- |
| `render` | shipped | 1 | Render template via envsubst, validate, write output |
| `push` | shipped | 1 | Push local template to remote, render, reload Caddy |
| `validate` | shipped | 1 | Run `caddy validate` on a Caddyfile |
| `reload` | shipped | — | `systemctl reload caddy` |
| `status` | shipped | — | `systemctl status caddy` |
| `diff` | `feature/proxy-commands` | 1 | Adapt two Caddyfiles to JSON, unified diff. Verifies refactored configs produce equivalent routing |
| `trace` | `feature/proxy-commands` | 2 | Spin up local Caddy + echo server, send a request, print what the client got and what upstream saw. Supports `--render`, `--live`, `--header` |

**`rots proxy diff`** — Runs `caddy adapt` on two Caddyfiles, sorts the JSON
output, and prints a unified diff. Exit 0 means equivalent, exit 1 means they
differ. Directly addresses the Layer 1 need for verifying snippet-based
refactoring against the monolithic config.

**`rots proxy trace`** — Starts a real Caddy process on an ephemeral port with
a lightweight echo backend, sends a request through, and prints exactly what
the client received and what the upstream (Puma) would have seen. With
`--live`, forwards to real upstreams instead of the echo server. With
`--render`, runs envsubst first (like `rots proxy render`). This is Layer 2
in a single command.

#### DNS commands (`rots dns ...`)

| Command | Status | Description |
| :--- | :--- | :--- |
| `list` (default) | `feature/dns-commands` | List all managed DNS records from local SQLite |
| `add` | `feature/dns-commands` | Create a DNS record via dns-lexicon (auto-detects provider and public IP) |
| `show` | `feature/dns-commands` | Show record details and audit history |
| `update` | `feature/dns-commands` | Update/upsert a DNS record |
| `remove` | `feature/dns-commands` | Delete a DNS record from provider and local tracking |

DNS commands use dns-lexicon for provider-agnostic API access (Cloudflare,
Route53, DigitalOcean, etc.) and track all actions in the local SQLite
database for audit. This directly addresses the Layer 3 need for automated
staging DNS setup.

---

### Summary of gaps in current state

1.  **No canonical template exists yet** — `Caddyfile-webdirect.template` is the old version; only `Caddyfile-custom.template` is snippet-based.
2.  **No automated tests** — beyond manual `caddy validate`.
3.  **No staging DNS setup** — for `.dev` domains. *(Partially addressed by `rots dns` commands.)*
4.  **CORS snippet hardcoded** — origins for `.com` and `.dev` need to be parameterized for staging.
5.  **`proxy diff` and `proxy trace` not yet merged** — on `feature/proxy-commands` branch.
6.  **DNS command test coverage at 12%** — `dns/app.py` has zero command-level tests; `DnsClient` class entirely uncovered.
7.  **No DNS propagation verification** — after `rots dns add`, no way to confirm the record propagated.
8.  **Ask endpoint security gaps** — `CaddyOnDemandStrategy` bypasses verification; no negative caching; no rate limiting.

The `.dev` domains on Cloudflare are the key enabler. With API access and the Let's Encrypt staging CA, you can test every ACME challenge method against real infrastructure without touching production.

---

### Multi-agent analysis: improving tooling for canonical and custom domains

*Conducted 2026-03-02 with backend-dev, frontend-dev, stripe-gremlin,
rodauth-expert, and qa-automation-engineer agents.*

#### Convergent findings

Five specialists independently identified the same priority ordering for
new tooling. The strongest signal is where multiple agents arrive at the
same conclusion from different angles.

**1. `rots dns check` — DNS propagation verification (all 5 agents)**

The single highest-leverage addition. After `rots dns add`, there is
currently no way to verify the record propagated. The command would query
public resolvers (1.1.1.1, 8.8.8.8, 9.9.9.9) and compare against the
expected value from `dns_current`. Supports `--wait` with timeout for
scripted workflows. ~80 lines, no new dependencies (uses `dig` or
`socket.getaddrinfo`).

This is the missing link between the `rots dns` CRUD and the `rots proxy
trace` verification. Without it, the staging workflow (Layer 3) requires
manual `dig` polling.

**2. `rots proxy trace --json` — structured output (backend + QA)**

The trace command currently prints human-readable output. Adding `--json`
mode (~30 lines of refactoring) is the prerequisite for:
- Storing trace results in SQLite for cross-deployment diffing
- YAML-based trace fixture files (`trace-suite`) for automated regression
- CI pipeline assertions on routing behavior

**3. Ask endpoint hardening (security + backend)**

The rodauth-expert found concrete issues in the OTS application's ask
endpoint implementation:
- `CaddyOnDemandStrategy#validate_ownership` returns `validated: true`
  unconditionally, bypassing the TXT verification flow
- No negative response caching — every unknown-domain TLS handshake
  round-trips to Redis
- No input sanitization before Redis lookup (defense-in-depth)
- Caddy's `on_demand_tls` config lacks `interval`/`burst` rate limiting

**4. DNS command test coverage (QA)**

Current state: `dns/app.py` at 12% coverage, `DnsClient` at 0%. The QA
agent provided concrete test patterns using mocked `lexicon.client.Client`
+ real SQLite. Writing these tests would bring DNS coverage to ~75% and
keep the repo above the 70% CI threshold.

#### Proposed command roadmap

Ordered by leverage and dependency. Each item is self-contained and can
ship independently.

| # | Command | Est. size | Depends on | Addresses |
|---|---------|-----------|------------|-----------|
| 1 | `rots dns check <hostname>` | ~80 lines | nothing | Propagation verification, staging workflow |
| 2 | `rots proxy trace --json` | ~30 lines | merge `feature/proxy-commands` | Machine-readable trace output |
| 3 | DNS command tests | ~200 lines | nothing | 12% → ~75% coverage |
| 4 | `rots proxy trace-suite <fixtures.yaml>` | ~120 lines | #2 | Automated regression for all 5 paths |
| 5 | `trace_results` SQLite table | ~60 lines | #2 | Cross-deployment routing diffs |
| 6 | `rots dns verify <hostname>` | ~100 lines | #1 | End-to-end: DNS + HTTP + TLS + header check |
| 7 | `rots proxy certs` | ~80 lines | nothing | TLS certificate inventory and expiry monitoring |

**What NOT to build yet:** A composed `rots domain provision` or
`rots staging setup` workflow. The individual primitives (#1, #2, #6)
need operational experience first. Composition is trivial once the
building blocks are solid. The danger of building orchestration before
primitives are tested is debugging both layers simultaneously.

#### Security priorities (from rodauth-expert)

Ordered by risk:

| Priority | Issue | Fix |
|----------|-------|-----|
| High | `CaddyOnDemandStrategy` bypasses ownership verification | Return `validated: false`, require real DNS check |
| High | `VerifyDomain` lacks A/CNAME resolution check for Caddy strategy | Add DNS resolution check independent of Approximated |
| High | Ask endpoint has no negative caching | Add in-memory LRU, 60s TTL for rejected domains |
| Medium | No Caddy-level rate limiting on `on_demand_tls` | Add `interval`/`burst` config |
| Medium | No periodic re-verification of domain ownership | Cron job: if TXT disappears, revert to `:pending` |
| Medium | TXT challenge value never regenerated | Regenerate on re-verification failure |
| Lower | No CT log monitoring | Future: `rots domain audit --ct-check` |

The domain ownership verification flow (TXT-based) is architecturally
sound. The ordering — TXT verification before A/CNAME change before ask
endpoint approval — correctly prevents cert issuance for unverified
domains. The gap is that the Caddy on-demand strategy bypasses this
ordering.

#### Billing integration (from stripe-gremlin)

Recommended model: **tiered inclusion** matching the existing plan
metadata pattern (`limits.custom_domains.included` on each plan tier,
per-domain overage pricing via metered `Price`).

Key architectural decision: the billing gate lives in the **application
layer** (OTS web app checks plan entitlements), not the infrastructure
layer (`rots` CLI). The `rots` CLI should never query Stripe directly.
Usage is reported to Stripe *after* successful provisioning (DNS verified
+ cert issued), not before.

Deprovisioning on downgrade follows a 3-stage timeline:
1. Grace period (7-14 days): all domains work, customer notified
2. Soft deprovisioning: excess domains redirect to default subdomain,
   TLS renewal stops
3. Hard deprovisioning (30+ days): routes removed, certs expire naturally

DNS records are never touched — they are customer-owned infrastructure.

#### Frontend surface (from frontend-dev)

Two highest-value UIs, ordered by operational toil reduction:

1. **Admin domain table** — hostname, traffic path badge (5 colors for 5
   paths), TLS status/expiry, health check dot, DNS record, last update.
   Data source: `dns_current` table + Caddy admin API (requires enabling
   `admin` on localhost, currently `admin off` in most configs).

2. **Customer domain onboarding stepper** — 6-state flow: not-configured
   → pending-dns → dns-propagating → dns-verified → tls-provisioning →
   active. Shows CNAME/A record instructions with copy button.
   `rots dns check` (proposed above) provides the backend for the
   propagation polling step.

Both require an API layer in the OTS web application that reads from the
`rots` SQLite database (same filesystem, direct read) or calls `rots`
CLI commands with `--json`.

#### Test strategy (from qa-automation-engineer)

Test boundary for proxy trace:
- **Unit testable** (no Caddy): `parse_trace_url`, `patch_caddy_json`,
  `_inject_trace_header`, `_patch_handler_upstreams`, `find_free_port`
- **Integration** (Python only): `run_echo_server`
- **System** (needs Caddy binary): full `trace` end-to-end

CI pipeline additions:
1. **Template validation job** — render all templates with `envsubst`,
   run `caddy validate` + `caddy adapt`, assert no unsubstituted vars
2. **Config equivalence gate** — `rots proxy diff` on PR (when merged)
3. **Routing smoke tests** — `rots proxy trace` against rendered
   templates, assert expected status codes and headers
4. **Structural assertions** — pytest tests that verify adapted JSON has
   expected server entries, TLS policies, and health check configs

Line between pytest and external: if a test needs credentials, DNS
propagation, or real TLS issuance, it belongs in a separate CI workflow
with `environment: staging` approval gates, not in `pytest tests/`.
