# CertiSigma Census — Agent Bootstrap

## Identity

You are a senior developer (€1000/h, expert in cryptography, UX, forensics) working on **CertiSigma Census** — a CLI tool for cryptographic file inventory and exfiltration detection. You write production-grade code. No hacks, no kludges, only best practices.

## What Census Is

Census is a **client application** that uses the CertiSigma API (via the published Python SDK) to:

1. **Scan** directories and compute SHA-256 hashes of files (content never leaves the client)
2. **Attest** hashes in batch via the CertiSigma API (three-layer proof: ECDSA T0, qualified TSA T1, Bitcoin T2)
3. **Compare** suspect files against the attestation registry to detect data exfiltration (breach detection)
4. Maintain a **local manifest** mapping hashes to file paths
5. **Cooperate selectively** with third parties (forensic analysts, threat intel) via derived lists (roadmap §33)

Census makes inventoried content classifiable and operationally verifiable by third parties without transferring the data or its meaning. This enables forensic verification, threat intelligence cooperation, and AI governance workflows — all without exposing original content.

Census is a **consumer** of the CertiSigma SDK, not an extension of the backend. It treats the SDK as a black box.

## Project Location

```
/opt/app-certisigma-census/          ← THIS PROJECT (your workspace)
├── pyproject.toml                   ← deps: certisigma>=1.8.0, click>=8.1
├── action.yml                       ← GitHub Action composite definition (certisigma/census-action@v1)
├── README.md                        ← full CLI reference, feature table
├── AGENTS.md                        ← this file
├── docs/
│   ├── ROADMAP.md                   ← phased roadmap with API dependency matrix
│   ├── TECHNICAL-DEBT.md            ← deferred improvements (eBPF, auditd, encryption)
│   ├── census.html                  ← developers page (same design system as sdk.html)
│   └── features/                    ← per-feature documentation (27 files)
├── src/certisigma_census/
│   ├── __init__.py                  ← version 1.10.1
│   ├── cli.py                       ← Click CLI: scan, compare, status, export, watch, verify, verify-manifest, integrity, report, diff, hash, track, config, completion, doctor, audit-log, snapshot, annotate, share, tag, key-rotate, key-gen, derived-list, metadata, merge, seal, verify-seal, bulk-scan, stats, compliance-report
│   ├── scanner.py                   ← filesystem walk + hash_file() + filters + resume + parallel hashing
│   ├── manifest.py                  ← SQLite manifest (WAL mode, schema v2, JSON migration)
│   ├── compare.py                   ← breach comparison via batch_verify() + parallel hashing
│   ├── evidence.py                  ← verify_hash() + integrity_check() + verify_manifest() + evidence chain
│   ├── report.py                    ← HTML/PDF/ZIP forensic report generation
│   ├── retry.py                     ← exponential backoff on 429/5xx
│   ├── watcher.py                   ← watchdog filesystem monitoring + event processing
│   ├── diff.py                      ← manifest comparison with AIDE-style exit codes
│   ├── config.py                    ← TOML config with user/project precedence chain
│   ├── doctor.py                    ← self-diagnostic checks (connectivity, config, inotify, deps)
│   ├── audit_log.py                 ← tamper-evident JSONL with SHA-256 hash chain
│   ├── snapshot.py                  ← named manifest snapshots for compliance baselines
│   ├── annotate.py                  ← forensic annotation via SDK update_metadata/delete_metadata + AES-256-GCM
│   ├── sharing.py                   ← forensic share tokens (chain of custody)
│   ├── tagging.py                   ← structured key-value tagging with encryption
│   ├── key_rotation.py              ← AES-256 key rotation (NIST SP 800-57)
│   ├── derived_lists.py             ← HMAC-SHA256 opaque third-party hash verification
│   ├── seal.py                      ← manifest HMAC-SHA256 tamper-evidence seal
│   ├── sarif.py                     ← SARIF v2.1.0 output (help, tags, invocations, file write)
│   ├── differential.py              ← differential integrity (state file, filter, first_seen, atomic save)
│   └── compliance.py               ← NIS2/DORA/ISO 27001 compliance report generation
├── tests/
│   ├── conftest.py                  ← shared fixtures (mock_client, populated_dir)
│   ├── test_manifest.py             ← 18 tests
│   ├── test_scanner.py              ← 16 tests
│   ├── test_compare.py              ← 12 tests
│   ├── test_cli.py                  ← 190 tests (CLI integration for all commands)
│   ├── test_evidence.py             ← 20 tests (verify_hash, integrity_check, verify_manifest)
│   ├── test_report.py               ← 18 tests
│   ├── test_retry.py                ← 7 tests
│   ├── test_watcher.py              ← 25 tests
│   ├── test_diff.py                 ← 17 tests
│   ├── test_config.py               ← 27 tests
│   ├── test_doctor.py               ← 20 tests
│   ├── test_audit_log.py            ← 16 tests
│   ├── test_snapshot.py             ← 15 tests
│   ├── test_annotate.py             ← 13 tests
│   ├── test_sharing.py              ← 25 tests
│   ├── test_tagging.py              ← 18 tests
│   ├── test_key_rotation.py         ← 13 tests
│   ├── test_derived_lists.py        ← 24 tests
│   ├── test_seal.py                 ← 16 tests (seal creation, verification, tampering, edge cases)
│   ├── test_future.py               ← 27 tests (bulk-scan + stats, including dry-run, source, output, dedup, errors, exit-zero, summary, meta)
│   ├── test_sarif.py               ← 25 tests (SARIF output unit + CLI integration + file write)
│   ├── test_update.py               ← 15 tests (AIDE-style baseline update, path traversal prevention)
│   ├── test_differential.py         ← 26 tests (differential integrity, state round-trip, CLI two-run suppression)
│   ├── test_census_html.py          ← 19 tests (HTML structure, section IDs, XSS, security)
│   ├── test_dogfood_workflow.py     ← 9 tests (dogfood CI workflow structure, permissions, security)
│   └── test_compliance.py           ← 26 tests (NIS2/DORA/ISO templates, HTML/JSON, CLI)
├── scripts/
│   └── test-census.sh               ← Docker smoke test (pre/post publish)
├── deploy.sh                        ← build/test/publish automation
└── .github/workflows/
    ├── test.yml                     ← CI: Python 3.10/3.11/3.12
    ├── publish.yml                  ← PyPI publish on tag v*
    └── dogfood.yml                  ← end-to-end action.yml validation
```

## Reference Material (read-only, do NOT modify)

These files are in the **main CertiSigma repo** and provide context about the API and SDK that Census consumes:

| What | Path | Why you need it |
|------|------|-----------------|
| **Python SDK source** | `/opt/app-certisigma/sdk/python/certisigma/` | Understand the SDK API surface (client.py, types.py, hash.py, exceptions.py, crypto.py, evidence.py) |
| **Python SDK README** | `/opt/app-certisigma/sdk/python/README.md` | Usage examples, public API |
| **SDK docs (HTML)** | `/opt/app-certisigma/developers/site/sdk.html` | Full public documentation |
| **Backend schemas** | `/opt/app-certisigma/backend/app/schemas.py` | API request/response shapes |
| **Backend routes** | `/opt/app-certisigma/backend/app/routes/` | Endpoint behavior (attest.py, verify.py, lookup.py, metadata.py) |
| **Technical debt** | `/opt/app-certisigma/docs/api/TECHNICAL-DEBT.md` | Planned API features Census will need (§10 Bulk, §29 Request IDs, §30 Forensic Sharing, §31 RBAC, §32 Structured Tagging, §33 Derived Lists) |
| **Census commercial doc (v1)** | `/opt/app-certisigma/docs/commercial/CERTISIGMA-CENSUS.md` | Original commercial text (corrected, factually accurate) |
| **Census commercial doc (v2)** | `/opt/app-certisigma/docs/commercial/CERTISIGMA-CENSUS-v2.md` | Expanded vision: derived lists, structured tagging, AI governance, NIS2/DORA/GDPR mapping |
| **Enterprise overview** | `/opt/app-certisigma/docs/commercial/CERTISIGMA-ENTERPRISE-OVERVIEW.md` | Architecture overview, Claims Layer, encryption model |
| **Test harness** | `/opt/app-certisigma/scripts/test-sdk.sh` | Reference for how SDK tests are structured |

**CRITICAL: Never modify files in `/opt/app-certisigma/`.** That is a separate project with its own git repo, deployment pipeline, and versioning. Census only reads from it for reference.

## SDK API Surface (what Census can use)

```python
from certisigma import CertiSigmaClient, hash_file, hash_bytes
from certisigma import CertiSigmaError, AuthenticationError, RateLimitError, QuotaExceededError
from certisigma.exceptions import NotFoundError, ValidationError

# Client (sync)
client = CertiSigmaClient(api_key="cs_...", base_url="https://api.certisigma.ch")
client = CertiSigmaClient()  # no key = public endpoints only

# Core operations
result = client.attest(hash_hex, source="label", extra_data={...})     # → AttestResult
result = client.verify(hash_hex)                                        # → VerifyResult
result = client.batch_attest(hashes_list, source="label")               # → BatchAttestResult
result = client.batch_verify(hashes_list, detailed=False)               # → BatchVerifyResult
result = client.get_evidence(attestation_id)                            # → EvidenceResult
result = client.attest_file(path, source="label")                       # → AttestResult
result = client.status(attestation_id)                                  # → AttestationStatus

# Metadata
result = client.get_metadata(attestation_id)                            # → MetadataResult
result = client.update_metadata(attestation_id, source=..., extra_data=...) # → MetadataPatchResult
result = client.delete_metadata(attestation_id)                         # → MetadataPatchResult

# Sharing (requires `share` scope)
result = client.create_share_token(attestation_ids, expires_in, recipient_label, max_uses) # → ShareTokenResult
tokens = client.list_share_tokens()                                     # → list[ShareTokenInfo]
client.revoke_share_token(token_id)
info = client.get_share_token_info(token_id)                           # → ShareTokenInfo

# Structured Tagging (requires `tags:read`/`tags:write` scopes)
result = client.put_tags(attestation_id, tags={"key": "value"})        # → TagResult
tags = client.get_tags(attestation_id)                                  # → list[Tag]
client.delete_tag(attestation_id, tag_key)
result = client.query_tags(filter={"and": [...]}, limit=100, cursor=None) # → TagQueryResult

# Derived Lists (requires `census` scope)
result = client.create_derived_list(hashes=[], tag_filter=None, label=None, expires_in_hours=None) # → DerivedListResult
items = client.list_derived_lists()                                     # → list[DerivedListInfo]
detail = client.get_derived_list(list_id)                              # → DerivedListDetail
result = client.match_derived_list(list_id, list_key, hashes)          # → MatchResult
sig = client.get_derived_list_signature(list_id)                       # → DerivedListSignature
log = client.get_derived_list_access_log(list_id, limit=50)           # → list[AccessLogEntry]
client.revoke_derived_list(list_id)

# Bulk / Scan (requires `scan` or `batch` scope + org_id)
result = client.scan(hashes)                                            # → ScanResult (up to 50K hashes, 5/hour rate limit)
result = client.get_bulk_stats()                                        # → BulkStatsResult

# Types: ScanMatch (hash_hex, first_seen, source)
#        ScanResult (scan_id, scanned_at, total_scanned, matched, unmatched, match_rate, matches)
#        BulkStatsResult (org_id_hash, total_claims, unique_hashes, first_claim, last_claim, claims_by_month)

# Hashing (standalone, no API call)
h = hash_file("/path/to/file")    # streamed, constant memory
h = hash_bytes(b"raw content")

# Crypto (standalone)
from certisigma.crypto import generate_key, encrypt_metadata, decrypt_metadata, is_encrypted, rotate_key

# Exceptions
# CertiSigmaError (base), AuthenticationError (401), RateLimitError (429),
# QuotaExceededError (403), NotFoundError (404), ValidationError (400/422)
```

## Architecture Rules

1. **Census installs `certisigma` from PyPI.** Never import from the backend source. Never copy SDK files. The published package is the contract.
2. **Content never leaves the client.** Only SHA-256 hashes are sent to the API.
3. **The manifest is local.** It maps hash → filepath and lives on the client filesystem. The server never sees file paths.
4. **No Docker.** Census is a CLI tool installed via `pip install certisigma-census`. No containers, no deploy.sh, no infrastructure.
5. **Batch limit: 100 hashes per batch API call.** Chunk accordingly. The `/scan` endpoint accepts up to 50K hashes per call (5 calls/hour).
6. **Rate limit: 1000 requests/minute** per API key (sliding window). Handle `RateLimitError` with backoff. `/scan` has its own rate limit: 5 calls/hour.

## Git Workflow

```bash
# Work on main branch
git add -A
git commit -m "descriptive message"   # Hook auto-strips "Made-with: Cursor"

# NEVER include in commit messages:
# - References to Cursor, AI, or assistants
# - "Made with Cursor" or similar
```

## Publishing to PyPI

Two publishing paths are supported:

### Option A — Local publish via `deploy.sh` (interactive)

```bash
./deploy.sh publish
# ↳ runs pytest, builds dist/, asks confirmation, then:
#   git tag v<VERSION> && git push origin main --tags && twine upload dist/*

# IMPORTANT: deploy.sh uses `read -rp` for interactive confirmation.
# In non-interactive shells this will hang. Run manually if needed:
git tag -a "v<VERSION>" -m "Release <VERSION>"
git push origin main --tags
twine upload dist/*

# Credentials: twine reads TWINE_USERNAME / TWINE_PASSWORD from env,
# or ~/.pypirc. Never hardcode credentials in scripts or commits.
```

### Option B — GitHub Actions publish on tag push (OIDC)

```bash
git tag -a "v<VERSION>" -m "Release <VERSION>"
git push origin main --tags
# → .github/workflows/publish.yml triggers on v* tags
# → builds + publishes to PyPI via trusted publisher (OIDC)
```

Both paths are valid. Use Option A for local control, Option B for hands-off CI/CD.

**Post-publish smoke test:**
```bash
./scripts/test-census.sh <VERSION>
```

## Current State (what works, what doesn't)

### ✅ Implemented (Fase 1 — CLI Core)
- `census scan <dir>` — walks directory, hashes files, attests in batch, saves manifest
- `census compare <suspect_dir>` — hashes suspects, verifies against registry, reports matches
- `census status <manifest>` — shows inventory summary
- Manifest CRUD with save/load round-trip
- Source labels, dry-run mode, JSON report export

### ✅ Implemented (Fase 2 — Production hardening, v0.2.1)
- Retry with exponential backoff on 429/5xx (uses `RateLimitError.retry_after`)
- Progress bar for scan, attest, and compare operations (`click.progressbar`)
- File filters: `--include`, `--exclude` globs; `--min-size`, `--max-size`
- Resume interrupted scans (`--resume`) with incremental manifest saves
- CSV export: `compare --output report.csv` + `census export manifest.db`
- Structured JSON logging (`--log-format json`) for SIEM integration
- deploy.sh build/test/publish automation script

### ✅ Implemented (Fase 3 — Watch mode + SQLite, v0.3.0)
- SQLite manifest backend (WAL mode, indexed lookups, schema v2)
- Auto-migration of legacy JSON manifests to SQLite
- `census watch <dir>` — continuous filesystem monitoring via `watchdog`
- Producer-consumer architecture: Observer thread → pending dict → main thread
- Per-event-type handling (CREATE, MODIFY, MOVE, DELETE)
- Debounce, batch interval, baseline scan, delete policies
- inotify limit detection with tuning instructions
- PollingObserver fallback for NFS/CIFS mounts
- Graceful shutdown on SIGINT/SIGTERM with final manifest save
- Symlink escape prevention
- Full test suite (106 tests + 17 skipped future markers)

### ✅ Implemented (Phase 3.5 — Forensic Evidence Pipeline, v0.4.0)
- `census verify <hash|--file>` — hash verification with full T0/T1/T2 evidence chain
- `census integrity manifest.db` — tamper detection against manifest baseline
- `census report manifest.db -o report.html|.pdf|.zip` — forensic report generation
- HTML reports (zero deps), PDF reports (fpdf2), evidence bundles (ZIP)
- Evidence bundles: report + OTS proofs + evidence.json + SHA256SUMS
- XSS prevention in HTML, path traversal prevention in ZIP

### ✅ Implemented (Phase 4 — Forensic Diff and CLI Maturity, v0.5.0)
- `census diff base.db target.db` — manifest comparison with AIDE bitmask exit codes
- `census hash <file|dir>` — standalone SHA-256, `--verify` mode
- `census track <att_id>` — attestation status tracking, `--poll` for T2 wait
- `census config show|init|paths` — TOML config with user/project precedence
- `census completion bash|zsh|fish` — shell completions via Click native
- Config integration in `_get_client()` for API key/base URL

### ✅ Implemented (Phase 5 — Enterprise Readiness, v0.6.0)
- `census doctor` — self-diagnostic (API health, config, inotify, deps, manifest integrity)
- `census merge m1.db m2.db -o out.db` — merge manifests for distributed workflows
- `census scan --json` — machine-readable scan output for CI/CD
- `census compare --json` — machine-readable compare output
- `census status --json` — machine-readable manifest status
- Full test suite (261 tests + 17 skipped future markers)

### ✅ Implemented (Phase 5.1 — Operational polish, v0.6.1)
- `census doctor` — HTTP timeout ridotto per `health()` (fail-fast su rete irraggiungibile)
- Watch: `_load_or_create_watch_manifest()` — chiusura garantita manifest in-memory prima del load su disco
- `census compare` — se `--json` e `--output` insieme, nota su stderr (`--output` ignorato)

### ✅ Implemented (Phase 6 — Compliance & Zero-Knowledge, v0.7.0)
- `census audit-log show|verify|clear` — tamper-evident JSONL with SHA-256 hash chain
- Automatic audit logging on `scan` and `compare` commands
- `census snapshot create|list|diff|delete` — named manifest snapshots for compliance baselines
- `census annotate <att_id>` — forensic annotation via `update_metadata()` / `delete_metadata()`
- `census annotate --encrypt` — AES-256-GCM zero-knowledge metadata encryption via `certisigma.crypto`
- `census annotate --delete` — GDPR right-to-erasure support
- Full test suite (321 tests + 17 skipped future markers)

### ✅ Implemented (Phase 7 — Forensic Sharing, Tagging & Derived Lists, v0.8.0)
- `census share create|list|revoke|info` — forensic share tokens (chain of custody)
- `census tag set|get|delete|query` — structured key-value tagging with encryption and cursor pagination
- `census key-rotate <att_id>` — NIST SP 800-57 AES-256 key rotation for metadata + tags
- `census derived-list create|list|info|match|revoke|access-log` — HMAC-SHA256 opaque third-party hash verification
- `census metadata get` — read attestation metadata with optional decryption

### ✅ Implemented (Phase 8 — Forensic Integrity & Performance, v0.9.0)
- `census derived-list signature <list_id>` — ECDSA signature verification for derived list integrity
- `census key-gen` — generate AES-256 encryption key via `certisigma.crypto.generate_key()`
- `census compare --detailed` — enriched batch_verify with source label, T0/T1/T2 level
- `census scan --workers N` / `census compare --workers N` — parallel hashing via `ProcessPoolExecutor`
- `source` field added to `MatchResult` and CSV export
- SDK dependency bumped to `certisigma>=1.7.0`

### ✅ Implemented (Phase 9 — Production Release, v1.0.0)
- `census verify-manifest` — full-chain verification: manifest ↔ registry via batch_verify
- `census verify-manifest --detailed --strict -o report.csv` — enriched output, strict exit codes, CSV/JSON export
- `census hash --stdin` — streaming SHA-256 from stdin for pipes and CI/CD
- `census export --format sha256sum` — GNU coreutils compatible output for forensic interop
- CertiSigmaClient resource management — `close()` registered via `ctx.call_on_close()`
- NotFoundError/ValidationError — specific error messages for 404 and 400/422 API responses
- Centralized `_load_manifest()` helper — all manifest loads go through single try/except
- Full test suite (455 tests + 3 skipped future markers)

### ✅ Implemented (Phase 10 — Forensic Hardening, v1.1.0)
- `census seal <manifest> --key <hex>` — HMAC-SHA256 tamper-evidence seal (Tripwire/AIDE pattern)
- `census verify-seal <manifest> --key <hex>` — seal verification with constant-time HMAC comparison
- `census scan --attest-manifest` — attest the manifest's own SHA-256 after scan (proves manifest existed at scan time)
- `--quiet / -q` global flag — suppress informational output for script/cron automation (errors and `--json` always shown)
- `_echo_info()` helper — centralized quiet-aware informational output
- Classifier updated to `Production/Stable`, TD-004 (multiprocessing) closed
- Full test suite (500 passed, 3 skipped future markers)

### ✅ Implemented (Phase 11 — Bulk Scale & SARIF, v1.2.0)
- `census bulk-scan <dir>` — bulk leak detection: 50K hashes/call via `client.scan()`, auto-chunking, manifest cross-ref
- `census bulk-scan --dry-run` — hash only, no API call, reports file/hash/chunk counts
- `census bulk-scan --source` — audit label for incident tracking
- `census bulk-scan --output` — save results to JSON file
- `census stats` — org-level inventory statistics via `client.get_bulk_stats()` (total claims, unique hashes, monthly breakdown)
- `census compare --format sarif` — SARIF v2.1.0 output for GitHub Security tab, VS Code, Defect Dojo, CI/CD integration
- `census compare --format sarif --output report.sarif` — write SARIF directly to file (CI/CD pattern)
- SARIF: `help.text`/`help.markdown` on rule, `properties.tags` for GitHub categorization, `invocations` for forensic provenance
- `hash_to_paths` tracks ALL file paths per hash (forensic completeness — duplicate-content files are not silently dropped)
- SDK dependency bumped to `certisigma>=1.8.0`
- Full test suite (562 tests, 0 skipped)

### ✅ Implemented (Phase 12 — CI/CD Integration Hardening, v1.3.0)
- `compare --exit-zero` — report-only mode: always exit 0 (CI pipelines that upload SARIF without gating)
- `compare --summary` — counts only, no individual match lines (concise CI logs)
- `bulk-scan --exit-zero` — same report-only pattern for bulk scan
- `bulk-scan --summary` — same summary pattern for bulk scan
- Feature docs: `docs/features/bulk-scan.md`, `docs/features/stats.md`, `docs/features/sarif.md`
- Updated `docs/features/comparison.md` with CI/CD integration section
- Full test suite (562 tests, 0 skipped)

### ✅ Implemented (Phase 13 — Production Hardening + Output Hygiene, v1.3.1)
- Fix: dead code after `sys.exit(1)` in `report_cmd` (success message unreachable)
- Fix: `baseline.close()` resource leak in watcher after `scan_on_start`
- `--no-color` flag + `NO_COLOR` env var (no-color.org standard) on main group
- `census_version` field in all JSON output (scan, compare, bulk-scan) for forensic traceability
- `elapsed_seconds` timing in all JSON output + "Completed in Xs" in text mode
- Helper `_with_meta(data, elapsed)` for consistent JSON metadata enrichment
- Full test suite (562 tests, 0 skipped)

### ✅ Implemented (Phase 14 — Automation & SIEM Integration, v1.4.0)
- `--format jsonl` on compare, bulk-scan, integrity, diff, verify-manifest (one JSON object per result, streamed)
- `--on-match CMD` on compare, bulk-scan — execute command with results as JSON on stdin when matches > 0
- `_emit_jsonl` helper — consistent JSONL output with `_summary` trailer (count, census_version, elapsed_seconds)
- `_run_on_match` helper — best-effort subprocess execution with 30s timeout, warning on failure
- systemd timer templates (`docs/templates/census-integrity.{service,timer}`) for periodic integrity checks
- Cron example script (`docs/templates/cron-example.sh`)
- Full test suite (571 tests, 0 skipped)

### ✅ Implemented (Phase 15 — AIDE-Style Update + Consistency, v1.5.0)
- `census update manifest.db` — accept verified filesystem changes into baseline (AIDE `--update` pattern)
- `update.py` module: `update_manifest_from_integrity()` with `UpdateResult` dataclass
- Interactive confirmation (or `--yes` for CI/cron), `--json` output (implies `--yes`), `--quiet` compliance
- New entries added with `attested=False` — user runs `scan --resume` to attest
- Audit logging (`_audit("update", ...)`) for compliance trail
- Fix: `--format jsonl --output` now warns on stderr (compare, integrity, diff, verify-manifest) — consistent with `--json --output`
- Fix: redundant `import time` in `track_cmd` removed

### ✅ Phase 15 security hardening (v1.5.1)
- Fix: Path traversal prevention in `update.py` — `_resolve_within_root()` with `resolve()` + `is_relative_to()` containment check
- Fix: `except OSError` widened to `except Exception` around `hash_file()` in update loops
- Fix: Progress bar in `update.py` now uses proper context manager (`with` + `nullcontext`)
- Fix: `manifest.save()` wrapped in `try/except` with explicit error handling
- Fix: `verify-manifest --format jsonl --output` now warns on stderr (was inconsistent with other commands)
- Fix: `--json` help text now documents implied `--yes` behavior
- Added security patterns to AGENTS.md: path traversal, exception handling, progress bar, JSONL+output consistency

### ✅ Implemented (Phase 16 — Differential Integrity, v1.6.0)
- `census integrity --since STATE --write-state STATE` — only report NEW findings since last run
- `differential.py` module: `IntegrityState`, `FindingState`, `build_state`, `filter_report`, `load_state`, `save_state` (atomic temp+rename)
- Per-finding `first_seen` (ISO-8601) + `occurrences` counter (forensic metadata, unique vs competitors)
- Auto-sidecar: `--since auto --write-state auto` → `manifest.db.integrity-state`
- JSON/JSONL enrichment: `differential`, `suppressed`, `new_findings`, `state_path`
- State file: versioned JSON (`schema_version: 1`), 50MB size limit, validated on load
- Fix: missing `_audit("integrity", ...)` compliance gap

### ✅ Implemented (Phase 17 — GitHub Action, v1.7.0)
- `certisigma/census-action@v1` composite action for CI/CD
- 11 inputs (command, target, manifest, api-key, source, format, upload-sarif, exit-zero, version, extra-args, python-version)
- 3 outputs (exit-code, output-file, sarif-file)
- SARIF auto-upload to GitHub Security tab, step summary, secret masking
- Shell injection prevention: all inputs via `env:` context
- Full test suite: `tests/test_action.py` (19 tests)

### ✅ Phase 17.1 — GitHub Action hardening (v1.7.1)
- Fail-fast: `format=sarif` rejected unless `command=compare`
- `--exit-zero` guarded by command (compare/bulk-scan only)
- Removed unused `GITHUB_OUTPUT` from install step
- 7 new security patterns in AGENTS.md

### ✅ Implemented (Phase 18 — Developers Page + CI Dogfood + Compliance, v1.8.0)
- `docs/census.html` — standalone developers page (same design system as sdk.html, 14 sections, dark/light, command palette, sidebar TOC)
- `.github/workflows/dogfood.yml` — CI workflow that dogfoods census-action (`uses: ./`) with dry-run manifest + integrity
- `census compliance-report` — NIS2, DORA, ISO 27001 compliance mapping from local manifest data (100% local, no API calls)
- `compliance.py` module: `generate_compliance_report()`, `render_compliance_html()`, `render_compliance_json()`
- Tests: `test_census_html.py` (19), `test_dogfood_workflow.py` (9), `test_compliance.py` (26)
- Full test suite: 700 tests

### ✅ Phase 18.1 — Post-implementation audit (v1.8.0)
- Fix: dead imports (`logging`/`logger`) in compliance.py, key_rotation.py, doctor.py, annotate.py, diff.py
- Fix: dead import (`json`) in compliance.py; dead `MagicMock` in test_compliance.py, test_report.py; dead `json` in test_config.py, test_doctor.py, test_diff.py
- Fix: unused `quiet` variable in `compliance_report_cmd`
- Fix: 70+ unclosed Manifest instances across 9 test files (SQLite connection leak) — converted fixtures to yield+close, added try/finally to all standalone tests
- Fix: `CSS.escape()` on `querySelector` with dynamic IDs in census.html (defense-in-depth)
- Fix: localStorage key aligned to `cs-theme` (cross-page consistency with sdk.html)
- 9 new security patterns in AGENTS.md

### ✅ Implemented (Phase 19 — AI Governance, v1.9.0)
- `census ai-policy init` — generate `.census-ai-policy.toml` template
- `census ai-policy apply <manifest> --policy <toml>` — classify + tag attestations (`ai_training=allowed/excluded`)
- `census ai-policy report <manifest> --policy <toml>` — HTML/JSON compliance report with coverage metrics
- TOML policy format: `[policy]` + `[[rules]]` with glob patterns, size filters, `default_action=exclude` (safety-first)
- Regulatory mapping: EU AI Act Art. 53(1)(c), ISO/IEC 42001, C2PA
- Most-restrictive-wins on shared `attestation_id`
- 100% local for classify/report; only `apply` (without `--dry-run`) makes API calls
- `ai_policy.py` module: load_policy, classify_manifest, tag_classifications, render_ai_report_html/json, generate_template_policy
- 47 new tests (test_ai_policy.py): TOML parsing, rule matching, coverage metrics, XSS safety, CLI integration

### 🔮 Future
- GUI (Tauri/Electron), Nextcloud integration
- Chain of custody documentation (forensic forms)

## Testing

```bash
# Run tests locally
cd /opt/app-certisigma-census
pip install -e ".[dev]"
pytest --tb=short -q

# Tests must NOT call the real API. Mock the SDK client for unit tests.
# Integration tests (real API) should be in a separate test file, clearly marked.
```

## API Environment

- **Production API:** `https://api.certisigma.ch`
- **API key env var:** `CERTISIGMA_API_KEY`
- **Docker exec for API testing (if needed):** `docker exec app-certisigma-notary-api-1 curl -s http://localhost:8080/...`

## Security & Quality Checklist

These patterns have caused bugs in past audits. Check them when writing or reviewing new code. Patterns marked with 🧪 have automated test coverage — the rest require human review.

### HTML Output

- **Every dynamic value** in generated HTML must pass through `html.escape()`. No exceptions. 🧪 `test_xss_prevention_in_paths`, `test_xss_prevention`
- **URLs in `href=`** must use `_safe_href()` (whitelist `https://` only) to block `javascript:` / `data:` schemes from API responses.
- **ZIP filenames** from external data (e.g. attestation IDs) must sanitize `/`, `\`, `..` to prevent path traversal.
- **`querySelector` with dynamic IDs** in JavaScript must use `CSS.escape()` on the ID — prevents selector injection if an ID contains `"`, `]`, or other CSS meta-characters. 🧪 `test_no_eval_calls`
- **`localStorage` keys** must be consistent across pages sharing a domain: use `cs-theme` (not per-page keys like `census-theme`) so theme preference persists across `sdk.html` ↔ `census.html`.

### Input Validation

- **SHA-256 hashes** from user input must be validated with `re.fullmatch(r"[0-9a-f]{64}", value)` before use. 🧪 `test_verify_rejects_invalid_hash`, `test_hash_verify_invalid_hash`
- **`--verify` flag** must reject directories (only single files). 🧪 `test_hash_verify_on_directory_rejected`
- **`resolve_encryption_key()` consumers** must validate hex format before calling SDK crypto functions — a malformed key produces cryptic SDK errors. Modules that validate: `seal.py` (`_parse_key()`), `key_rotation.py` (`_validate_hex_key()`). Cross-module defense: validate at `config.resolve_encryption_key()` level.

### Resource Management

- **`Manifest.load()` must always use `with` or `try/finally`** to close the SQLite connection. `Manifest` implements `__enter__`/`__exit__`.
- **`Manifest._load_from_sqlite()` and `_load_from_json()`** must wrap all post-`connect()` logic in `try/except: conn.close(); raise` — a corrupt database, malformed metadata, or migration failure would otherwise leak the SQLite connection and lock file. Phase 19 audit found both methods lacked this guard.
- **In-memory `Manifest()` then `save()` then `Manifest.load()`**: call `.close()` on the in-memory instance before discarding the reference — otherwise the `:memory:` SQLite connection leaks. 🧪 (watcher create-new path)
- **`manifest.save()` must always have error handling** — bare `manifest.save(path)` can fail (disk full, permissions, WAL corruption) and produce raw tracebacks that leak internal paths. In CLI commands: use `try/except Exception` with styled error message + `sys.exit(1)`. In background loops (watcher): use `try/except Exception` with `logger.error()` to continue processing.
- **File handles** opened conditionally (e.g. `open()` vs `sys.stdout`) must use `try/finally`.
- **Test fixtures that create Manifest must use `yield` + `close()` in `finally`** — `return m` leaks the SQLite connection. Use `yield m; m.close()` or `try/yield/finally` pattern. Tests that create Manifest directly must wrap the body in `try:/finally: m.close()`. Phase 18.1 audit found 70+ leaked instances across 9 test files.
- **Test helpers that return Manifest**: callers must close the returned object. If the helper is only used in tests, document that the caller owns the close() responsibility.
- **NameError risk in test try/finally**: The pattern `m = Manifest(...); try: ...; finally: m.close()` has a theoretical NameError if `Manifest()` raises (variable unbound in finally). Acceptable for in-memory SQLite (never fails in practice). Established codebase convention — do not refactor to `m = None; try: ...` pattern.
- **`NamedTemporaryFile(delete=False)` must wrap creation+use in a single try/finally** — if code between creation and the try block fails (e.g. `f.write()` on disk full), `os.unlink()` in the finally is never reached and the temp file leaks. Correct pattern: `path = None; try: ...; finally: if path and os.path.exists(path): os.unlink(path)`. Phase 20 audit fixed `test_integration.py::test_attest_file`.

### Scripts & Development Tools

- **No `sys.path.insert(0, ...)` in scripts** — prefer `pip install -e .` and import normally. Path manipulation can silently import source-tree code instead of the installed package, masking production-different behavior (e.g. compiled extensions, different `__version__`). Phase 20 audit removed this from `scripts/benchmark.py`.
- **CLI arguments for resource-intensive operations must have upper bounds** — unbounded `--files`, `--entries`, `--count` parameters can exhaust disk space or memory. Use `parser.error()` to reject values above a documented maximum (e.g. `MAX_FILES = 100_000`). Phase 20 audit added limits to `scripts/benchmark.py`.
- **`json.dumps()` in scripts must include `default=str`** — same as the `--json` output rule. Scripts that generate JSON reports, benchmark results, or diagnostic output must not crash on unexpected types.
- **Fragile ID parsing: avoid `int(x.replace("prefix_", ""))` patterns** — if the upstream format changes (e.g. `att_` prefix removed, UUID IDs), the parse silently produces wrong results or crashes. Prefer: (1) store the raw ID from the API and use it directly, (2) if type conversion is needed, use explicit try/except with a clear error. Phase 20 audit fixed `test_integration.py::TestSharingLifecycle`.

### `--json` Output Purity

- Commands with `--json` must **never** write non-JSON text to stdout. Gate all human-readable messages with `if not json_output`. 🧪 `test_integrity_json_with_output_no_stdout_contamination`, `test_diff_json_with_output_no_stdout_contamination`
- **`compare --json` + `--output`:** file report must not be written; warn on **stderr** that `--output` is ignored. 🧪 `test_compare_json_with_output_warns_on_stderr`
- Progress bars must be disabled when `--json` is active.
- Error messages in JSON mode should be JSON objects (`{"error": "..."}`) on stdout, not plain text.
- **ALL `json.dumps()` writing to stdout or files must include `default=str`** — prevents `TypeError` crashes on `datetime`, `Path`, `Decimal`, or other non-JSON-native types that may appear in SDK responses or `asdict()` output. Phase 19 audit found 20+ calls missing this safety net.
- **ALL `--json` output paths must wrap data through `_with_meta(data, elapsed)`** — adds `census_version` and `elapsed_seconds` for forensic traceability. Phase 7+ commands (annotate, share, tag, key-rotate, derived-list, audit-log, snapshot) were identified as missing this pattern.

### Type Safety

- **`Optional[int]` fields** must never be used with format specifiers (`:,`, `:d`) without a null guard: `size = val if val is not None else 0`.
- **`Optional[str]` fields** displayed to the user must use `val or "unknown"`, never bare `{val}` (renders as `None`).

### JSONL & On-Match Hooks

- **JSONL output purity**: `--format jsonl` must emit ONLY JSON lines to stdout — no log messages, no progress bars. Each line must be valid JSON.
- **JSONL `_summary` trailer**: Every JSONL stream must end with a `{"_summary": true, "count": N, "census_version": "..."}` line. Consumers can use `_summary` to detect stream completion.
- **`--on-match` command injection**: The command string is passed to `shell=True` — this is intentional (users need shell features like pipes and env vars). Document this in help text.
- **`--on-match` failure isolation**: `_run_on_match` must never raise or change Census's exit code. Use `try/except Exception` and `logger.warning`.
- **`--on-match` timeout**: 30 seconds default. If the command hangs, Census continues normally.
- **`--format` + `--json` coexistence**: When both are provided, `--format` takes precedence. The `--json` flag is equivalent to `--format json`. Format normalization must happen early in each command.

### Config & Secrets

- **`_DEFAULT_CONFIG`** is deep-copied in `load_config()` — never use `dict()` shallow copy on nested defaults. 🧪 `test_returned_config_does_not_share_references_with_defaults`
- **`_SENSITIVE_KEYS`** defines which config keys are masked in `census config show`. Add new secrets here.
- **API keys**: CLI flag > env var > project config > user config > default. Never log or print unmasked.
- **Key masking** must show only last 4 chars (`...xxxx`), never first+last (`xxxx...xxxx`) — the latter leaks 67-100% of short keys. Use `mask_secret_tail()` from `config.py` for all display paths (`census config show`, doctor, etc.). 🧪 `TestMaskSecretTail`, `TestMaskSensitive`

### Baseline Update

- **`census update` must always run `integrity_check` first** — never accept changes blindly without re-verifying the filesystem state.
- **Confirmation required by default** — `--yes` is opt-in. Interactive mode must show exactly what will change (removed/updated/added counts) before proceeding.
- **`--json` implies `--yes`** — machine-readable output mode auto-accepts changes (no interactive prompt). Document this clearly in `--json` help text.
- **New entries must be `attested=False`** — the update only changes the manifest locally. Attestation requires a separate `census scan --resume`.
- **TOCTOU safety**: Files can change between `integrity_check` and the update loop. `update_manifest_from_integrity` must catch `Exception` per-file and continue (log in `errors` list), not abort the entire update. Use `except Exception`, not `except OSError`, because `hash_file()` may raise non-OS errors (SDK exceptions, `MemoryError`, etc.).
- **Manifest must be saved atomically** — the existing `manifest.save(path)` uses `sqlite3.backup()` which is atomic. Never write directly to the manifest file.
- **Path traversal in update logic**: Any function that resolves paths from an `IntegrityReport` (which could be crafted) **must** call `resolve()` and verify containment with `is_relative_to(root.resolve())`. Never use bare `root / rel_path` without the containment check — `..` segments in `rel_path` escape the root. Use `_resolve_within_root()` from `update.py`.

### Path Traversal Prevention

- **Any `root / user_path` pattern** must validate containment after resolution: `(root / path).resolve().is_relative_to(root.resolve())`. The `/` operator does NOT prevent `..` escape.
- **Absolute paths from external data** must also be containment-checked — an `is_absolute()` branch that uses the path directly is a bypass. Always resolve and check.
- **Reference implementation**: `update._resolve_within_root()` — resolve + `is_relative_to` + `ValueError` on escape.
- **Where to watch**: Any code that takes paths from `IntegrityReport`, manifest entries, or API responses and performs file I/O (`stat()`, `hash_file()`, `open()`).

### Exception Handling Around `hash_file()`

- **ALL `except OSError:` around `hash_file()` must be `except Exception:`** — the SDK's `hash_file()` may raise non-OS exceptions (e.g. `MemoryError`, internal SDK errors). Catching only `OSError` lets unexpected exceptions abort the entire operation. 🧪 `scanner.py` already uses `except OSError` + `except Exception` (two-level pattern — also acceptable).
- **`relative_to()` can raise `ValueError`** (not `OSError`) — when combining `stat()` and `relative_to()` in the same `try` block, catch `(OSError, ValueError)`. Phase 19 audit found `scanner._scan_parallel` catching only `OSError`, which let `ValueError` from `relative_to()` abort the entire scan.
- **Partial-progress save**: If a loop calls `hash_file()` on many files, ensure `manifest.save()` is called even if the loop exits early (via `try/finally`).

### Progress Bar Usage

- **`click.progressbar` is a context manager** — always use `with click.progressbar(...) as bar:`. Calling `.update()` and `.render_finish()` without entering the context is broken (the bar is never initialized/displayed).
- **Use `contextlib.nullcontext(None)` as the no-op alternative** when the progress bar is conditionally disabled. This keeps a single `with` block for both paths.

### JSONL + `--output` Consistency

- **All commands** that support `--format jsonl` and `--output` must warn on stderr and skip the file write when both are active: `"Note: --output is ignored when --format jsonl is set."`. Commands: `compare`, `integrity`, `diff`, `verify-manifest`, `bulk-scan`. 🧪 `test_compare_jsonl_format`

### Differential State Files

- **Schema version check**: `load_state()` must reject unknown `schema_version` values with a clear error. Never silently accept future schemas.
- **Size limit**: Pre-check file size with `os.stat()` before `json.load()` (50MB limit). Prevents DoS from crafted state files.
- **Type validation on deserialized fields**: All JSON fields must be `isinstance`-checked before use. `findings` must be `dict`, each finding entry must be `dict`, `KeyError` on required fields must be converted to `ValueError`. 🧪 `test_rejects_non_dict_finding`, `test_rejects_finding_missing_field`, `test_rejects_findings_as_list`
- **Atomic write**: `save_state()` uses `tempfile.mkstemp()` + `os.replace()` — the state file is never partially written. Temp file is cleaned up on failure.
- **Errors are NOT deduplicated**: `IntegrityReport.errors` always pass through to output unfiltered. Transient I/O issues must always surface.
- **State reflects FULL report, output reflects FILTERED report**: `--write-state` saves state from the complete `integrity_check()` result (pre-filter), while stdout shows only new findings. This prevents missed detections on the next run.
- **`first_seen` carried forward**: When a finding persists with the same fingerprint, `first_seen` is preserved from the original state. Never overwrite it.
- **Auto-sidecar convention**: `manifest.db` → `manifest.db.integrity-state` (follows `seal.py` sidecar pattern). Path must be derived from `resolved.resolve()` to avoid relative-path fragility.
- **`--since` with nonexistent file**: Run full mode (no filtering), no error. The first run creates state for future differential use.
- **`new_files` fingerprint is path-only** (empty string) because `integrity_check()` does not hash new files. A content change in a "new" file won't resurface it. Document this; recommend `census update` to accept new files promptly.
- **`save_state` exception handler** must catch `(OSError, ValueError, TypeError)` — not just `OSError` — because `json.dumps`/`dataclasses.asdict` can raise `ValueError`/`TypeError` on edge-case data.
- **Non-mutating enrichment helpers**: Functions that add metadata to JSON output (like `_diff_enrichment`, `_with_meta`) must return a new dict via `{**d, ...}`, never mutate the input. Prevents latent bugs if callers share dict references.
- **JSONL `_summary` must include differential metadata** when differential mode is active (`differential`, `suppressed`, `new_findings`). JSONL consumers need this to distinguish filtered from full runs. Use `extra_summary` parameter on `_emit_jsonl()`. 🧪 `test_jsonl_summary_includes_differential_meta`

### Imports

- **No dead imports** — check after refactoring. Phase 18.1 audit found dead `import logging`/`logger` in 4 source modules and dead `import json`/`MagicMock` in 5 test files. Common pattern: `import logging; logger = logging.getLogger(__name__)` is added by template but never used. Remove both `import logging` and the `logger =` line together.
- No aliased re-imports (`import csv as csv_mod`) when the module is already top-level.
- Conditional imports (e.g. `from dataclasses import asdict`) should be inside the branch that uses them, not at function top.
- **New CLI commands must use all computed variables** — `quiet = ctx.obj.get("quiet", False)` must be wired to `show_progress` or a conditional. Dead variables are a code smell and confuse reviewers (Phase 18.1: found unused `quiet` in `compliance_report_cmd`).

### Watcher-Specific

- `path.relative_to(root)` can raise `ValueError` on TOCTOU races (file moves outside root between event capture and processing). Always wrap in `try/except ValueError`.
- The `_accept()` filter must resolve symlinks and reject paths outside `self.root` to prevent symlink escape.

### Doctor / Diagnostic Commands

- **`check_api_key` must not make network calls**: The `health()` endpoint is public and doesn't validate API keys — calling it gives false confidence. Format-only checks are honest.
- **Doctor text output must distinguish zero-fail-zero-warn from zero-fail-with-warn** — "All checks passed" is misleading when warnings exist.
- **Diagnostic timeouts**: External API calls in doctor should have shorter timeouts than production calls (SDK default is 30s, doctor should ideally be <10s).

### Audit Log

- **Hash chain genesis** must be `"0" * 64` — never empty string or None.
- **`append_entry()` must never raise exceptions** from the CLI — it's best-effort. Wrap all calls in `_audit()` which catches and logs `Exception`. 🧪 `test_creates_file_and_appends`, `test_detects_tampered_entry`
- **`verify_chain()` must detect both tampered content AND broken links** (prev_hash mismatch vs entry_hash mismatch). 🧪 `test_detects_broken_chain_link`
- **`verify_chain()` must detect corrupt lines** — do NOT use `iter_entries()` (which silently skips corrupt lines). Parse raw lines and report corruption as a chain break. 🧪 `test_detects_corrupt_line`
- **`_last_hash()` must read only the tail** of the file (seek to end - 8 KiB) — never scan the entire file linearly.
- **Every forensically significant CLI command** must call `_audit()` — scan, compare, annotate, annotate_delete, merge, watch_start/watch_stop, share_create/revoke, tag_set/delete, key_rotate, derived_list_create/match/revoke. Omitting a command from audit logging is a compliance gap.

### Snapshots

- **Snapshot names are filesystem-safe**: only `[a-zA-Z0-9_-]`, max 128 chars. Path traversal via `../` or `.` is impossible because they contain illegal characters. 🧪 `test_special_chars_rejected`
- **Snapshot diff reuses existing `diff_manifests()`** — never duplicate diff logic.
- **Snapshot create** must use `sqlite3.backup()` (not `shutil.copy2`) to safely copy SQLite databases with pending WAL data. 🧪 `test_snapshot_is_valid_sqlite`
- **Snapshot create** must fail on duplicate names. 🧪 `test_duplicate_name_raises`
- **CLI commands that call `_validate_name()`** (snapshot delete, snapshot diff, get_snapshot_path) must catch `ValueError` and translate to `click.BadParameter` — otherwise users get raw tracebacks on names like `"../escape"`.

### Forensic Annotation

- **Encryption key validation** must happen before any API call. Missing key → `AnnotationResult(success=False)` with error message. 🧪 `test_missing_key_returns_error`
- **`encrypt_metadata()` errors** (bad key length, invalid hex) must be caught as `(ValueError, TypeError, KeyError)` — `KeyError` is needed because SDK crypto envelope dict access can fail on malformed envelopes. Phase 19 audit found `annotate.py` was missing `KeyError`.
- **`_resolve_encryption_key()`** must check explicit arg first, then config — never environment variables (keys in env vars are a security risk for long-lived processes).
- **`annotate()` catches `CertiSigmaError`** and returns `AnnotationResult(success=False)` — never bubbles SDK exceptions to the CLI. 🧪 `test_error_handling`
- **Encrypted metadata** sets `client_encrypted=True` on the API call so the server knows it's ciphertext. 🧪 `test_encrypted_annotation`

### Retry Logic

- **`retry_after` from `RateLimitError` must be guarded for None/negative/non-numeric** — if the server omits the `Retry-After` header, `exc.retry_after` may be `None`, causing `time.sleep(None)` → `TypeError`. Always use `getattr(exc, "retry_after", None)` with type/sign check and fallback to exponential backoff.
- **`retry_after` must be capped at `MAX_BACKOFF`** — a server returning `retry_after=86400` would block Census for 24h. Apply `min(wait, MAX_BACKOFF)` to both 429 and 5xx wait times.
- **`max_retries` must be non-negative** — a caller passing `max_retries=-1` causes an empty `range()`, `last_exc` stays `None`, and `raise None` produces `TypeError`.

### Key Rotation

- **Partial rotation is a data integrity risk**: If metadata rotation succeeds but tag rotation fails midway, the metadata is encrypted with the new key while some tags remain under the old key. Error messages MUST warn: `"Do NOT discard the old key"`.
- **Tag rotation must be batched** — collect all re-encrypted tags in a list and send via a single `client.put_tags(attestation_id, tags=all_rotated)` call. Per-tag API calls create a window where some tags are under the old key and others under the new key. Phase 19 audit converted per-tag rotation to batch.

### Tagging Performance

- **`_resolve_encryption_key()` and `from certisigma.crypto import encrypt/decrypt_metadata` must be hoisted outside per-tag loops** — resolving the key from TOML config involves disk I/O; importing the module has a one-time cost. Both are constant across loop iterations. Phase 19 audit found both `set_tags` and `get_tags` performing per-tag resolution.

### TOCTOU: stat-before-hash

- **Always `stat()` BEFORE `hash_file()`**, never after. If the file changes between hashing and stat, the manifest records old hash with new size/mtime — a silent inconsistency that survives `--resume`. Found in `update.py` (new-files branch) during Phase 19 audit.
- **`integrity_check()` in `evidence.py`** also has stat-after-hash for `actual_size` — documented as known limitation (forensic context, not manifest update).

### New Command Patterns (v0.6.0+)

- **Merge operations** on Manifest objects must use `try/finally` to close the in-memory merged manifest even on exceptions.
- **Long-running commands with `--resume`** (scan, merge) must close ALL intermediate Manifest objects — both the loaded resume manifest and the scan-directory manifest. When reassigning `manifest = existing_manifest`, close the original first.
- **Dead function parameters** after refactoring: when removing functionality from a function, also remove parameters that are no longer used (e.g. `base_url` after removing network calls).

### Forensic Sharing, Tagging & Derived Lists (v0.8.0+)

- **List functions** (`list_share_tokens`, `get_tags`, `list_derived_lists`) must catch `CertiSigmaError` — return empty list + log error, never bubble raw SDK exceptions to CLI.
- **`match_derived_list`** must validate `list_key` (64 hex chars) before `bytes.fromhex()` — catch `ValueError` and return `MatchResult(success=False)`.
- **CLI `--list-key` validation**: `re.fullmatch(r"[0-9a-fA-F]{64}", ...)` before calling match.
- **`query_tags` error field**: `TagQueryResult` has `error: Optional[str]` — CLI must check and `sys.exit(1)` on error.
- **Secrets never in audit params**: `share_token`, `list_key`, `old_key`, `new_key` must NEVER appear in `_audit()` params.
- **`share_token` and `list_key` are shown exactly once** to the user — this is intentional (one-time display).
- **Key rotation** must validate both keys are 64 hex chars AND different before any API call.

### SDK Type Contract Compliance

- **`create_share_token(attestation_ids: list[int])`**: SDK expects **integer** IDs, not strings. Census receives string IDs from CLI (e.g. `"att_42"`, `"42"`). Always strip `att_` prefix and convert to `int` before passing to SDK. Use `_parse_attestation_id()` from `sharing.py`. 🧪 `TestParseAttestationId`, `test_invalid_attestation_id`
- **`put_tags(tags: list[dict])`**: SDK expects a **list of dicts** (`[{"key": "k", "value": "v"}]`), NOT a flat dict (`{"k": "v"}`). For plain tags: `{"key": k, "value": v}`. For encrypted tags: `{"key": k, "value_enc": envelope["ciphertext"], "value_nonce": envelope["nonce"], "client_encrypted": True}`. Never pass `{"key": k, "value": envelope_dict}` — the backend expects `value_enc`/`value_nonce` as separate fields. 🧪 `test_success` (verifies list format), `test_encrypted_tags` (verifies field mapping), `test_put_tags_sends_list_format`
- **`key_rotation.py` put_tags**: After `rotate_key()` returns `{"ciphertext": ..., "nonce": ..., "v": 1}`, must convert to the `list[dict]` format above before calling `put_tags`. Never pass the raw envelope as a dict value.

### DRY — Shared Utilities

- **`resolve_encryption_key()`** lives in `config.py` — single source of truth. Both `annotate.py` and `tagging.py` delegate to it via thin `_resolve_encryption_key()` wrappers. Never duplicate the config lookup logic in individual modules.

### Error Return Type Consistency

- **`get_derived_list_detail`** returns `DerivedListDetail` on success but `DerivedListCreateResult` on error. **`get_share_token_info`** returns `ShareInfo` on success but `ShareResult` on error. Callers must use `isinstance()` to distinguish success/error. When writing new SDK wrapper functions, prefer a single return type with an `error` field (like `TagQueryResult`) over mixed return types.

### Mock Path Rules for Tests

- **Top-level imports** (`from certisigma import hash_file` at module level in scanner.py, compare.py, evidence.py, watcher.py): mock with `@patch("certisigma_census.<module>.hash_file")` — the name is bound in the module namespace.
- **Late imports** (`from certisigma import CertiSigmaClient` inside a function): mock with `@patch("certisigma.CertiSigmaClient")` — the name is resolved at call time.
- **Crypto imports** (`from certisigma.crypto import encrypt_metadata` inside a function): mock with `@patch("certisigma.crypto.encrypt_metadata")`.
- **Internal helpers** (`certisigma_census.tagging._resolve_encryption_key`): mock at the Census module path.

### Error Messages to stderr

- **All `click.echo` calls that display error or warning messages** must use `err=True` to write to stderr. Only user-facing success output goes to stdout. This prevents error text from contaminating `--json` output or piped workflows.
- **Keywords that trigger `err=True`:** `Error:`, `Cannot`, `Warning:`, `VIOLATION`, `MISMATCH`, `BROKEN`, `fail`, red-styled messages, security advisories ("save this key", "list_key shown only once"). The `doctor` command must branch on `check.status` — WARN/FAIL details go to stderr, OK to stdout.

### Parallel Hashing (v0.9.0+)

- **`ProcessPoolExecutor` cannot serialize mocks.** Tests that use `--workers N > 1` must NOT mock `hash_file` — use real files instead. The child process imports `hash_file` fresh from the module; parent-process mocks are not visible.
- **Workers cap: `MAX_WORKERS = 8`.** CLI validates and caps `--workers` to prevent resource exhaustion. Library functions (`scan_directory`, `compare_directory`) also clamp `workers` defensively.
- **`as_completed()` for progress bar.** Results arrive out of order from the pool. Use `as_completed(futures)` to update the progress bar as each hash completes.
- **TOCTOU: stat BEFORE hash, not after.** In parallel mode, capture `stat()` (size, mtime) before submitting to the pool and carry the values through `futures` dict. Stat-after-hash creates a race where the file changes between worker hashing and main-process stat, causing stale manifest entries that survive `--resume`.
- **Broad exception handling for `future.result()`.** Catch both `OSError` and `Exception` (for `BrokenProcessPool`, `MemoryError`, etc.) so a single worker crash doesn't abort the entire `as_completed` loop.

### batch_verify Mocking (v0.9.0+)

- **`retry_api_call` forwards `**kwargs` to the wrapped function.** Test mocks (lambdas, side_effect functions) for `batch_verify` must accept `**kwargs` because `compare_directory` now passes `detailed=bool`.
- **`batch_verify` result items use `"id"`, not `"attestation_id"`.** The SDK's `BatchVerifyResult.results` dict items use the `"id"` key for attestation ID.

### SDK API Calls — Exception Safety

- **Every `client.XXX()` call MUST be wrapped in `try/except`.** Either via `retry_api_call()` (which handles 429/5xx) or directly with `except CertiSigmaError`. Unprotected calls produce raw tracebacks that leak internal details and break `--json` output purity.
- **`verify_hash()` in `evidence.py` delegates to `retry_api_call`.** Callers (like `verify_cmd`) must STILL wrap the call in try/except — `retry_api_call` re-raises permanent errors (401, 403, 422).
- **`--json` error paths need `sys.exit(1)`.** When `--json` is active and an error occurs, emit `{"error": "..."}` to stdout AND `sys.exit(1)`. A successful JSON output with exit code 0 on error breaks CI/CD pipelines.

### Corrupt Input Handling

- **Use `_load_manifest()` helper for ALL manifest loads in CLI.** The centralized `_load_manifest(path)` in `cli.py` wraps `Manifest.load()` with `try/except Exception` → `click.ClickException`. Never call `Manifest.load()` directly from CLI commands — always go through `_load_manifest()`. This covers corrupt SQLite, file permission errors, and any other DB failure in one place.
- **`hash_file()` on single file must catch `OSError`.** The directory-hashing branch already handles this; the single-file branch (e.g., `census hash <file>`, `census verify --file`) must do the same.

### Resource Leak Patterns

- **`CertiSigmaClient` must be closed.** Two patterns:
  - **CLI commands:** `_get_client()` auto-registers `client.close()` via `ctx.call_on_close()`. Never create `CertiSigmaClient()` directly in CLI code.
  - **Library functions** (sharing.py, tagging.py, etc.): wrap with `try/finally: client.close()` since no Click context is available.
- **`Manifest.load()` must always use a context manager** (`with _load_manifest(...) as m:`) or `try/finally` with `m.close()`. Bare `m = Manifest.load(...)` leaks SQLite connections on exception.
- **`sqlite3.connect()` must always use `try/finally`** with `conn.close()`. Even in tests — if `conn.execute()` raises, the connection leaks.
- **`audit-log show --last N`** must NOT materialize the full log then slice — use `collections.deque(maxlen=N)` to cap memory on large logs.

### Phase 9 — Production / full-chain verification (v1.0.0+)

- **`verify-manifest` / `integrity` / `diff` — `-o` before stdout JSON.** When both `--output`/`-o` and `--json` are used, write the report file **before** emitting success JSON. If the write raises `OSError`, emit `{"error": "Cannot write report: ..."}` to stdout (when `--json`) or a styled message to stderr, then **`sys.exit(1)`**. Never print success JSON and then fail on disk — that breaks CI/CD and forensic automation.
- **Human (non-JSON) + `-o`:** Keep UX: print the full text summary first, then write the file; on `OSError` print to stderr and **`sys.exit(1)`** (user already saw the report on screen).
- **`compare --json` + `--output`:** Output file is **not** written; a note goes to stderr (`--output` ignored). This is intentional — do not duplicate the verify-manifest ordering bug here.
- **`verify-manifest` batch semantics.** `batch_verify` results must record both `exists: true` (verified) and `exists: false` (not attested) in the per-hash map. Omitting `exists: false` rows caused false `"Not in verify response"` errors — always set `verified_set[h] = {"verified": False}` for unattested hashes.
- **`verify_cmd` and `NotFoundError`:** SDK `NotFoundError` is an HTTP **404 on the verify request**, not the normal “hash not in registry” path (that returns `exists=False` without an exception). Tailor user messages accordingly; do not confuse 404 with unattested content.
- **SDK exception imports in CLI:** Import `NotFoundError`, `ValidationError` at **module level** in `cli.py`, not inside `except` blocks — an `ImportError` there would mask the original API error and leak a confusing traceback.
- **`hash --stdin`:** Use chunked reads (e.g. 64 KiB) into `hashlib.sha256()` — bounded memory, same model as `sha256sum`. No stdin size cap by design (operational limits are environment-specific).
- **`export --format sha256sum`:** GNU text format uses two spaces between digest and path. Paths containing **newlines** break one-line-per-record consumers (`sha256sum -c`, naive parsers) — document for integrators; not shell injection when written to a file without `eval`.
- **Library `CertiSigmaClient` lifecycle:** Every function that constructs `CertiSigmaClient(...)` outside `_get_client()` must use `try` / `finally: client.close()` on **all** exit paths (including early `return` after `CertiSigmaError`). 🧪 covered by annotate, sharing, tagging, derived_lists, key_rotation, doctor tests.

### Phase 10 — Forensic Hardening (v1.1.0)

- **HMAC seal must use `hmac.compare_digest()`** — constant-time comparison prevents timing-based attacks on the seal. Never use `==` for HMAC comparison.
- **Seal key validation:** Use `re.fullmatch(r"[0-9a-fA-F]{64}", key_hex)` — not just `len()` + `bytes.fromhex()`. Python's `bytes.fromhex()` silently strips ASCII whitespace, which can produce sub-256-bit keys. Always validate with strict regex first.
- **Seal JSON deserialization:** Always validate the `hmac` field type with `isinstance(stored_mac, str)`. A tampered seal with `"hmac": 12345` or `"hmac": null` would cause `hmac.compare_digest()` to raise `TypeError`.
- **CLI except blocks must catch both `ValueError` and `OSError`** for seal/verify-seal commands. Missing `OSError` produces raw tracebacks on I/O failures.
- **`--attest-manifest` must run after final `manifest.save()`.** The attestation anchors the on-disk bytes, so it must hash the saved file — not the in-memory state.
- **`--attest-manifest` is no-op on `--dry-run`.** Dry-run skips attestation entirely (no client is created), so attempting to attest the manifest would require API access that doesn't exist.
- **`--attest-manifest` failure must not abort the scan.** The broad `except Exception` is intentional — attestation failure is non-critical and only logged.
- **`_echo_info()` must never suppress error messages.** When `kwargs.get("err")` is True, always emit — regardless of `--quiet`. This ensures errors are visible in cron/CI contexts.
- **`_echo_info()` defensive `ctx.obj` check:** Always guard with `ctx and ctx.obj and ctx.obj.get("quiet")` — `ctx.obj` can be `None` in programmatic invocations.
- **`--quiet` only suppresses stdout informational messages.** JSON output (`--json`), error messages (stderr), and exit codes are always preserved. Keep `if not json_output:` guards where they existed — `_echo_info` adds quiet suppression on top, not as a replacement.
- **`show_progress` must include `and not quiet`** in every command that has a progress bar. Audit all `show_progress =` assignments when adding new commands.

### Propagation audit — codebase-wide patterns (v1.1.0)

- **ALL hex key/hash validation must use `re.fullmatch()`** — not `len()` + `bytes.fromhex()` or `int(x, 16)`. This applies to `_validate_list_key` (derived_lists.py), `_validate_hex_key` (key_rotation.py), and `_parse_key` (seal.py). `bytes.fromhex()` silently strips whitespace; `int(x, 16)` accepts underscores and leading/trailing whitespace. Both produce keys shorter than expected.
- **All `except` blocks for SDK crypto operations must include `KeyError`** — `rotate_key()`, `encrypt_metadata()`, and `decrypt_metadata()` return dicts. If the SDK changes field names or the envelope is malformed, subscript access raises `KeyError`. Always catch `(ValueError, TypeError, KeyError)` at minimum; add `AttributeError` for `.get()` on potentially non-dict results.
- **All CLI file-write operations must be wrapped in `try/except OSError`** — every `Path.write_text()`, `Path.write_bytes()`, `open()`, `manifest.save()` in a CLI command must have an OSError handler that: (1) produces valid JSON if `--json` is active, (2) prints a styled error to stderr otherwise, (3) calls `sys.exit(1)`.
- **Deserialized JSON fields must be type-checked before operations** — `entry.get("field")` from user-controlled JSON (audit log, seal sidecar, legacy manifest, snapshot metadata) can return any type. Always `isinstance()` before `str` operations, slicing, `hmac.compare_digest()`, or dict unpacking (`**entry_data`).
- **Legacy manifest `_load_from_json` must validate structure** — `data.get("entries", {})` must be `isinstance(entries, dict)` before `.items()`, and each `entry_data` must be `isinstance(entry_data, dict)` before `ManifestEntry(**entry_data)`.

### Phase 11 — Bulk Scale & SARIF (v1.2.0)

- **`hash_to_paths` must be `dict[str, list[str]]`** — not `dict[str, str]`. Two files with identical content share the same SHA-256 hash. Using `dict[str, str]` silently drops all but the last path — catastrophic for a forensic tool. Use `setdefault(h, []).append(rel)` and expand matches per-path.
- **Sequential hashing must catch `Exception`** — not just `OSError`. Consistency with parallel branch (which catches `Exception` for `BrokenProcessPool`, `MemoryError`, etc.). Otherwise, a non-OSError in sequential mode crashes the command with a raw traceback.
- **`_RULE` dict in sarif.py must be deep-copied** before inclusion in output. Without `copy.deepcopy(_RULE)`, callers who mutate the returned SARIF dict permanently corrupt the module constant for subsequent calls in the same process.
- **SARIF `help.text` and `help.markdown`** must be present on rule definitions — GitHub Code Scanning renders empty alert details without them. Also add `properties.tags` (e.g. `["security", "data-exfiltration"]`) for categorization in security dashboards.
- **SARIF `invocations` array** must include `executionSuccessful: True` — GitHub upload validation uses this field.
- **`compare --format sarif --output PATH`** must write SARIF to file and confirm on stderr — not ignore `--output`. SARIF is a file-oriented format for CI/CD upload; forcing stdout redirect is hostile UX.
- **`bulk-scan --dry-run`** is essential — the `/scan` endpoint has a strict rate limit (5 calls/hour). Users must preview what would be scanned before committing API calls.
- **`stats_cmd` must `_audit()` on error path** — same pattern as `bulk_scan_cmd`. Missing audit on API failure is a compliance gap.
- **`_echo_info` must respect `--quiet` in bulk-scan and stats** — check `show_progress = not verbose and not json_output and not quiet` and gate all info output through `_echo_info`.
- **`bulk-scan --workers` silently clamps** — inconsistent with `compare` which warns. Document this behavior or align with compare's pattern.

### Post-build audit — Phase 11.1 (v1.2.1)

- **`hash_to_paths.get(h, [])` fallback must be `[]`, never `[""]`** — a fallback of `[""]` creates a phantom `MatchResult` with `suspect_path=""` when the API returns a hash not in the local mapping. This is a false positive in the forensic evidence chain — an unfounded accusation with no real suspect file. The empty-path match also corrupts SARIF output (`"."` URI), CSV reports, and inflates match counts. Always use `[]` (skip silently) as the fallback for hash lookups on match expansion.
- **`--source` parameter must be length-capped** — no sanitization on `--source` allows multi-megabyte strings that bloat audit logs, JSON output, and memory. Validate with `_validate_source()` (max `MAX_SOURCE_LEN=256` chars) in every command that accepts `--source`: `scan`, `watch`, `bulk-scan`. The cap applies before any `_audit()` call or output serialization.
- **ALL `except OSError:` around `hash_file()` must be `except Exception:`** — applies uniformly to: `scanner.py:_hash_and_add`, `evidence.py:integrity_check`, `watcher.py:_process_events`, `cli.py:verify_cmd`, `cli.py:hash_cmd` (single-file and directory branches), `compare.py` (sequential branch), `cli.py:bulk_scan_cmd` (sequential branch). `MemoryError`, `BrokenProcessPool`, and unexpected SDK exceptions are not subclasses of `OSError` — catching only `OSError` crashes the command with a raw traceback.
- **`unmatched` field must count unique hashes, not files minus files** — when duplicate-content files exist, `len(all_matches)` (expanded per-file) can exceed `len(all_hashes)` (unique), producing negative `unmatched`. Use `matched_hashes: set[str]` to track matched unique hashes, then `unmatched = len(all_hashes) - len(matched_hashes)`.
- **`report.matched` in `compare.py` counts unique matched hashes** — while `report.matches` contains expanded per-file `MatchResult` entries. These two semantics diverge when duplicate-content files exist. The JSON output includes both `matched` (hashes) and `matches` (files), which is technically correct but `total_suspect` counts files. Document the semantic distinction.
- **`_get_client()` must be deferred after `--dry-run` guard** — creating a `CertiSigmaClient` in `--dry-run` mode wastes resources and confuses the intent. Move `_get_client()` to after the dry-run return.
- **SARIF file-write errors must go to stderr** — `click.echo(click.style(...), err=True)`, not `click.echo(json.dumps({"error": ...}))` to stdout. When `--format sarif --output` is used, the user expects stdout to be completely clean.
- **`copy.deepcopy(_RULE)` is correct for module-level constants** — `json.loads(json.dumps(_RULE))` is an alternative but breaks if non-JSON types are added later. `deepcopy` preserves Python types exactly.

### Propagation audit — unbounded string parameters (v1.2.1)

- **ALL free-text CLI parameters must be length-capped** via `_validate_label(value, "--param")` (max `MAX_LABEL_LEN=256` chars). This applies to: `--source` (scan, watch, bulk-scan, annotate), `--note`, `--tag`, `--case-id` (annotate), `--recipient` (share create), `--label` (derived-list create). Without validation, a multi-megabyte string bloats audit logs (JSONL on disk), JSON output, and API request payloads.
- **Tag keys and values have separate caps** — `MAX_TAG_KEY_LEN=128`, `MAX_TAG_VALUE_LEN=1024` in `tagging.py:parse_tag_pair()`. Values are longer because they may contain structured forensic data (JSON fragments, base64). Keys must be concise identifiers.
- **Module-level mutable collections must be immutable types** — use `frozenset` not `set` for lookup tables like `_VALID_NAME_CHARS` in `snapshot.py`. While the mutation risk is low (no code intentionally modifies them), `frozenset` communicates intent and prevents accidental `.add()`/`.remove()` calls.
- **`dict.get(h, [])` not `dict.get(h, [""])`** — empty-string fallbacks in hash-to-path lookups create phantom data entries. Use `[]` (skip silently) as the fallback for all match-expansion loops. Display-context fallbacks (e.g., `att_id or "unknown"` for messages) are acceptable.
- **`_validate_source()` was renamed to `_validate_label()`** — a single reusable helper for all free-text CLI parameters. The old function was too narrowly named and not applied consistently. Always use `_validate_label(value, param_hint)`.

### Post-build audit — Phase 13.1 (v1.3.1)

- **Dead code after `sys.exit()` — unreachable statements**: Any `click.echo()` or business logic placed after `sys.exit(N)` in the same block is unreachable. Found in `report_cmd` where the success message was inside an `except` block after `sys.exit(1)`. Always verify that success messages are placed *after* the try/except, not inside it.
- **Duplicate success messages on branching write paths**: When a try/except wraps multiple branches (ZIP/PDF/HTML), each branch must NOT print its own success message if a generic one follows outside the try/except. This creates double output and pollutes forensic logs.
- **`NO_COLOR` env var: presence vs truthiness**: The [no-color.org](https://no-color.org) spec says the *mere presence* of `NO_COLOR` (even `NO_COLOR=""`) should disable color. Click's `envvar` for `is_flag` options treats empty strings as falsy. Always add an explicit `os.environ.get("NO_COLOR") is not None` check in addition to the Click flag.
- **Hardcoded version strings in tests**: Never hardcode `"1.2.0"` or `"1.3.1"` in test assertions. Import `__version__` from the package and assert against it. Hardcoded versions break on every bump and obscure the actual failure when they drift.
- **Resource cleanup must use `try/finally`**: When acquiring a resource (`Manifest`, `CertiSigmaClient`) and then performing operations, the `close()` call must be in a `finally` block. Found: `baseline.close()` after `manifest.merge(baseline)` without `finally` — if `merge()` raises, the connection leaks.
- **`_with_meta()` must be applied to ALL JSON output paths**: Including early-return paths (e.g., empty-dir in `bulk-scan`). Inconsistent metadata in JSON output means forensic artifacts lose traceability information for edge cases.
- **In-place dict mutation in helper functions**: `_with_meta(data)` mutates `data` in place. All current callers pass fresh dict literals, but if a shared reference is ever passed, it would corrupt the caller's data. Document the mutation contract or use `{**data, ...}` for safety.

### Propagation audit — Phase 13.2 (v1.3.2)

Systematic propagation of all Phase 13.1 patterns across the entire codebase. Each pattern was re-searched globally; findings below were fixed.

**Resource cleanup (`try/finally`) — 3 new sites fixed:**
- `manifest.py:save()`: `disk_conn.close()` after `self._conn.backup(disk_conn)` — if `backup()` raises, `disk_conn` leaked.
- `cli.py` scan resume path: `manifest.close()` after `existing_manifest.merge(manifest)` — if `merge()` raises, the old manifest leaked (outer `finally` closes `existing_manifest` but not the original `manifest`).
- `watcher.py` cleanup: `manifest.close()` after `manifest.save()` — if `save()` raises, the connection leaked.

**`_with_meta` forensic metadata — propagated to 14 additional commands:**
- `status`, `verify`, `hash`, `track`, `integrity`, `verify-manifest`, `diff`, `stats`, `merge`, `seal`, `verify-seal`, `doctor`, `metadata`, `key-gen` — all now include `census_version` and `elapsed_seconds` in JSON output.
- `hash --json` output format changed from bare list to `{"files": [...], "census_version": "...", "elapsed_seconds": ...}` for consistency with all other commands.

**`_with_meta` made non-mutating:**
- Changed from `data["key"] = value; return data` to `{**data, "key": value}`. Prevents latent bugs if a caller ever passes a shared dict reference.

**`--quiet` compliance — core commands fixed:**
- `report_cmd`: 4 `click.echo()` calls converted to `_echo_info()` (evidence count, integrity check result, save confirmation).
- `track_cmd`: all informational output converted to `_echo_info()` (status display, poll wait messages). Error/timeout output correctly stays on stderr.
- `merge_cmd`: detail and summary messages converted to `_echo_info()`.
- `doctor_cmd`: OK check results converted to `_echo_info()`, warnings/failures stay on stderr.
- `watcher.py`: added `quiet` parameter, replaced all `click.echo()` with quiet-aware `_info()` helper. The `cli.py` watch command now passes `quiet=quiet` through.

**Patterns confirmed clean (no propagation needed):**
- Dead code after `sys.exit()`: 0 new instances
- Duplicate success messages: 0 new instances
- `except OSError` → `except Exception` around `hash_file()`: all 11 sites already correct
- JSON error output (stdout vs stderr): all consistent

### Propagation audit — Phase 16.1 (v1.6.1)

Systematic propagation of all Phase 16 differential-integrity patterns across the full codebase. Each pattern was re-searched globally; all findings fixed.

**Type validation on deserialized JSON — 4 sites fixed:**
- `seal.py:verify_seal()`: `json.loads()` result not checked as `isinstance(seal_data, dict)` — crafted `.seal` file with JSON array/string crashes with `AttributeError`. Added guard before `.get()` access.
- `snapshot.py:list_snapshots()`: `except` tuple missing `TypeError` — corrupt snapshot `.json` with non-dict JSON produces unhandled `TypeError` on `meta["name"]`.
- `audit_log.py:_last_hash()`: `json.loads()` result not guarded — JSON primitive in last audit line crashes `.get()` with `AttributeError`. Added `isinstance(entry, dict)` guard.
- `cli.py:derived_list_cmd`: `json.loads(tag_filter)` not validated as dict — user `--tag-filter '"string"'` passes through to SDK causing confusing error. Added `isinstance(tf, dict)` check.

**`except OSError` too narrow around serialization — 3 sites fixed:**
- `cli.py:_write_compare_report()`: only write helper missing `default=str` on `json.dumps()`. If `asdict(report)` ever produces non-serializable types, `TypeError` escapes uncaught. Added `default=str`.
- `cli.py:_write_compare_report()` caller: `except OSError` widened to `except (OSError, TypeError, ValueError)`.
- `cli.py:bulk_scan_cmd` output write: `json.dumps(result_data)` lacked `default=str`. Added for consistency.
- `seal.py:create_seal()`: `json.dumps(seal_data)` lacked `default=str`. Added for consistency.

**JSONL `_summary` metadata parity — 5 callers enriched:**
- `compare --format jsonl`: JSONL summary lacked `total_suspect`, `matched`, `not_matched`, `errors`. Added via `extra_summary`.
- `integrity --format jsonl`: when NOT in differential mode, summary lacked `total_in_manifest`, `unchanged`, `modified`, `missing`, `new_files`. Now always emits integrity totals.
- `verify-manifest --format jsonl`: summary lacked `total`, `verified`, `not_found`, `errors`. Added.
- `diff --format jsonl`: summary lacked `base_root`, `target_root`, `added`, `removed`, `modified`, `unchanged_count`. Added.
- `bulk-scan --format jsonl`: summary lacked `total_files`, `total_scanned`, `matched_files`, `matched_hashes`, `unmatched_hashes`, `scan_ids`. Added.

**Relative path fragility — 2 sites fixed:**
- `snapshot.py:_resolve_snapshot_dir()`: user-provided override and config paths not `.resolve()`'d — relative paths break if CWD changes between create and list.
- `audit_log.py:_resolve_log_path()`: same pattern. Both now call `.resolve()` on all non-default paths.

**Dead import — 1 site fixed:**
- `watcher.py`: unused `import sys` removed.

**Post-mutation of `_with_meta()` returned dicts — 3 sites fixed:**
- `cli.py` scan: `scan_result["manifest_attestation_id"] = ...` mutated after `_with_meta()`. Refactored to build full dict before enrichment.
- `cli.py` bulk-scan dry-run: `dry_data["hash_errors"] = ...` mutated after `_with_meta()`. Same fix.
- `cli.py` bulk-scan results: `result_data["source"]` and `result_data["hash_errors"]` mutated after `_with_meta()`. Same fix.

### Post-build audit — Phase 16.1 (v1.6.1)

- **ALL file-write `except` blocks wrapping `json.dumps()` + `write_text()` must catch `(OSError, TypeError, ValueError)`** — not just `OSError`. `json.dumps()` raises `TypeError` on non-serializable types and `ValueError` on circular references. Catching only `OSError` lets serialization errors escape as raw tracebacks, breaking `--json` output purity and leaking internal paths. Applies to: integrity report writes (2 branches), verify-manifest report write, diff report writes (2 branches), bulk-scan output write.
- **ALL `json.dumps()` writing to files or stdout must include `default=str`** — defensive consistency against non-serializable types in `asdict()` output. Without it, a future dataclass field of type `datetime`, `Path`, or `Decimal` silently crashes the command. Exception: `audit_log.py` intentionally omits `default=str` to detect data corruption early (all `AuditEntry` fields are primitives by design).
- **`_emit_jsonl` `extra_summary` must not overwrite reserved keys** — `summary.update(extra_summary)` can silently replace `_summary`, `count`, `census_version`, `elapsed_seconds`. Guard with a reserved-key filter: `{k: v for k, v in extra_summary.items() if k not in _RESERVED}`.

### GitHub Action security — Phase 17 (v1.7.0) + audit (v1.7.1)

- **NEVER interpolate `${{ inputs.* }}` in `run:` blocks** — use `env:` context to pass inputs to shell steps. Direct interpolation enables shell injection via crafted input values. This applies to ALL composite actions.
- **CLI flag compatibility matrix** — not all Census commands support the same flags. `--exit-zero` only works on `compare` and `bulk-scan`. `--format sarif` only works on `compare`. The action must guard against passing unsupported flags to commands that don't accept them.
- **Fail fast on invalid `format` + `command` pairs** — if the user sets `format: sarif` with `command` other than `compare`, the action must `::error::` and exit 1. Silent fallback to CLI default format hides misconfiguration and breaks SARIF/upload expectations.
- **Mask secrets before any output** — the `::add-mask::` step must run BEFORE the Census command step. Masking after output is useless.
- **SARIF upload must be conditional** — only upload when the SARIF file was actually generated (`steps.*.outputs.sarif-file != ''`) and `upload-sarif == 'true'`. Unconditional upload on a missing file crashes the workflow.
- **Job permissions for SARIF** — workflows using `upload-sarif: true` need `permissions: security-events: write` (and typically `contents: read`). Document in user-facing docs; missing permissions fail the upload step with a clear GitHub error.
- **Do not append unsanitized multiline strings to `GITHUB_OUTPUT`** — newlines delimit additional key/value pairs and can spoof outputs or break parsing. If a value must be written, strip `\n`/`\r` or use the multiline `<<EOF` syntax deliberately; avoid echoing arbitrary command output into `GITHUB_OUTPUT` unless escaped.
- **`extra-args` is a trust boundary** — word-split expansion (`CMD+=($INPUT_EXTRA_ARGS)`) is intentional for power users. Feeding `extra-args` from untrusted workflow inputs (e.g. PR titles) is command injection. Reusable workflows should only pass static or org-controlled strings.
- **Step summary Markdown** — values from `inputs.*` echoed into `$GITHUB_STEP_SUMMARY` can alter Markdown rendering (backticks, links). Low severity in trusted repos; avoid passing unsanitized external content into summaries in public OSS workflows.

### Propagation audit — Phase 17.1 (GitHub Action patterns)

- **No other composite actions** in this repo — `action.yml` is the only Census-owned GitHub Actions definition. There is nothing to “propagate” the same `run:`/`env:` split to elsewhere in-tree.
- **`.github/workflows/*.yml`** — `test.yml` / `publish.yml` only pass `matrix.python-version` or static values into `actions/*` `with:` blocks, not into shell `run:` via `${{ inputs.*}}`. No shell-injection surface analogous to a composite action.
- **`GITHUB_OUTPUT` / step summary** — only appear in `action.yml` (not in Python sources). CLI and library code must never depend on GitHub-specific env files; keep that boundary.
- **Regression coverage** — `tests/test_action.py` runs in the normal CI pytest job, so edits to `action.yml` that break schema, masking order, SARIF conditions, or install-step hygiene fail the build immediately.

### Compliance Reports — Phase 18 (v1.8.0)

- **ALL dynamic values in compliance HTML must pass through `html.escape()`** — framework name, generated_at, version, root, requirement fields, article, capability, detail. Status CSS class names come from a fixed map, not user input.
- **Template name must be validated against `TEMPLATES` frozenset** before use. `generate_compliance_report()` raises `ValueError` on unknown templates.
- **Compliance reports must not include API keys or secrets** — they use only manifest data (file counts, attestation status) and integrity check results. No API calls, no network access.
- **`render_compliance_json()` uses `dataclasses.asdict()`** — safe for all primitive types in `ComplianceReport`. If future fields include non-serializable types, add `default=str` to the `json.dumps()` call in `cli.py`.

### Developers Page (census.html) — Phase 18 (v1.8.0)

- **TOC and search innerHTML must use `escapeHtml()`** — section titles from `h2.textContent` and section IDs from `sec.id` are hardcoded in the HTML, but defense-in-depth requires escaping before inserting into `innerHTML` (TOC links, search results).
- **External links with `target="_blank"` must include `rel="noopener"`** — prevents reverse tabnabbing. Badges already have this; footer links open in the same tab (no target).
- **No `eval()`, `document.write()`, or dynamic `<script>` injection** — the page is fully static with inline JS.

### AI Governance — Phase 19 (v1.9.0)

- **TOML policy parsing must validate all fields** — unknown actions, non-string patterns, non-integer sizes, missing `[policy]` section must all raise `ValueError` with clear messages. Never accept untrusted TOML blindly.
- **Rule count limit** (`MAX_POLICY_RULES = 500`) — prevents DoS from crafted policy files with thousands of rules.
- **`default_action` must be `"allow"` or `"exclude"`** — no third state. Validated in `load_policy()`.
- **Glob patterns use `fnmatch.fnmatch()`** — safe against path traversal (matches against the manifest's relative path, never constructs filesystem paths from patterns).
- **HTML report uses `html.escape()` on all dynamic values** — policy name, file paths, rule names, attestation IDs. 🧪 `test_html_escapes_dynamic_values`
- **`tag_classifications()` deduplicates by `attestation_id`** — prevents redundant API calls when multiple manifest entries share an attestation.
- **Unattested files cannot be tagged** — files without `attestation_id` are classified in the report but skipped during tagging. 🧪 `test_skips_unattested`
- **Most-restrictive-wins** is applied after classification — if files share an `attestation_id` and one is `exclude`, all are elevated to `exclude`.
- **`apply` without `--dry-run` requires `--api-key`** — enforced in CLI before any API calls.
- **`apply` exits non-zero on tagging errors** — `sys.exit(1)` when `errors > 0`, so CI/CD pipelines detect partial failures. 🧪 `test_apply_requires_api_key_for_tagging`
- **Per-rule pattern limits** (`MAX_PATTERNS_PER_RULE = 1000`, `MAX_PATTERN_LEN = 512`) — prevents memory exhaustion from crafted policy files with millions of patterns or megabyte-long glob strings.
- **Policy file size limit** (`MAX_POLICY_FILE_SIZE = 10 MB`) — pre-checked via `stat()` before `read_bytes()` to prevent loading multi-gigabyte files.
- **Field length caps** (`MAX_FIELD_LEN = 256`) — `[policy].name` capped to prevent bloated HTML reports from malicious policy files.
- **Boolean rejection in size fields** — Python `bool` is a subclass of `int`; TOML `min_size = true` would silently become `1`. Explicit `isinstance(x, bool)` check before `isinstance(x, int)`.
- **Negative size rejection** — `min_size` and `max_size` must be non-negative; `min_size > max_size` raises `ValueError`.
- **Dedup guard uses action values, not tag values** — `tag_classifications()` compares `existing == "exclude"` (action), not `"excluded"` (tag value). Prevents allow-overwrites-exclude when `_apply_most_restrictive` is bypassed.
- **Tagging exceptions are logged** — `except Exception as exc` in per-attestation loop logs `logger.warning()` with attestation ID and error. Silent swallow is a compliance gap.
- **TOCTOU-safe file creation** — `ai_policy_init` uses `os.O_CREAT | os.O_EXCL` (atomic exclusive create) instead of check-then-write.
- **UTF-8 BOM handling** — `load_policy()` uses `utf-8-sig` codec to transparently strip Windows BOM.
- **`--quiet` compliance in Phase 7+ commands** — all confirmation messages (snapshot, annotate, share, tag, key-rotate, derived-list, audit-log clear) use `_echo_info()` instead of bare `click.echo()`.
- **SARIF `json.dumps` must include `default=str`** — prevents `TypeError` if SARIF data contains non-serializable types from `asdict()`.
- **`merged.save()` exception width** — `Manifest.save()` uses `sqlite3.backup()` which raises `sqlite3.DatabaseError` (not `OSError`). Except clause must be `except Exception`.

### Propagation audit — Phase 19.2 (v1.9.2)

Systematic propagation of all Phase 19.1 patterns across the entire codebase. Each pattern was re-searched globally; findings below were fixed.

- **`scan_cmd` exits non-zero on complete attestation failure** — when `attested == 0` and `hashes_to_attest > 0` (not dry-run), the command now emits a red error on stderr and `sys.exit(1)`. Previously exited 0, hiding total API failures from CI/CD.
- **`update_cmd` exits non-zero on update errors** — when `result.errors` is non-empty, `sys.exit(1)` ensures pipelines detect partial failures.
- **`bulk_scan_cmd` hash errors are now logged** — three bare `except Exception:` blocks (parallel with progress, parallel without, sequential) now capture `exc` and call `logger.warning("Cannot hash %s: %s", fp, exc)`. Previously only incremented a counter, losing the reason for hash failure (permissions, corruption, etc.).
- **`report_cmd` except clause widened** — `except OSError` around HTML/PDF generation now catches `(OSError, TypeError, ValueError)`. Generators use `html.escape()`, `fpdf2`, and string formatting that can raise non-OS errors.
- **`audit_log_cmd` except clauses widened** — `except OSError` around `iter_entries()` and `verify_chain()` now catches `(OSError, ValueError)`. `iter_entries()` parses JSONL via `json.loads()` which raises `json.JSONDecodeError` (subclass of `ValueError`). A corrupt audit log would escape the handler.
- **`snapshot create` except clause widened** — `except OSError` now `except Exception`, catching `sqlite3.DatabaseError` from `sqlite3.backup()` on corrupt manifests.
- **`compliance-report` except clause widened** — `except OSError` around `render_compliance_html()` write now catches `(OSError, TypeError, ValueError)`.
- **`json.dumps` defense-in-depth** — three calls displaying decrypted annotation data and metadata (`data['decrypted_extra_data']`, `data['extra_data']`, decrypted plaintext) now include `default=str`.
- **`html.escape()` defense-in-depth in compliance.py** — `i_class` and `i_status` (fixed ternaries) now escaped for consistency with the requirements loop which escapes `cls`.
- **`html.escape()` defense-in-depth in report.py** — `status_class` and `status_text` (fixed ternaries) now escaped for consistency.
- **TOCTOU-safe `config init`** — `init_config()` now uses `os.O_CREAT | os.O_EXCL` (same pattern as `ai_policy_init`). CLI caller catches `FileExistsError` instead of check-then-write.
- **Config file size limit** — `_load_toml()` now checks `path.stat().st_size` against `MAX_CONFIG_FILE_SIZE = 1 MB` before parsing. Prevents memory exhaustion from crafted config files (same pattern as `ai_policy.py MAX_POLICY_FILE_SIZE`).
- **`--quiet` compliance: derived-list create** — confirmation line "Derived list created:" now uses `_echo_info()`.
- **`--quiet` compliance: config init** — "Config already exists" and "Config template created" now use `_echo_info()`.

**Patterns confirmed clean (no propagation needed):**
- **Action/value string mismatch** — all comparisons in `ai_policy.py`, `compliance.py`, `doctor.py`, `differential.py`, `cli.py` are within their correct domain. No cross-domain confusion found.
- **CSS class names from user input** — all `class=` attributes in HTML generators use fixed ternaries or fixed-key dict lookups. No user-controlled values reach CSS class names.
- **`audit_log.py` json.dumps** — intentionally omits `default=str` for deterministic hashing (documented exception in AGENTS.md).
- **`seal.py` key validation** — regex `{64}` inherently bounds length. `verify_seal` except is `OSError`-only, which is correct for pure file I/O + HMAC.
- **`snapshot.py` name validation** — 128-char limit + character whitelist. Clean.
- **`tagging.py` key/value limits** — 128/1024 char caps. Clean.

### Phase 20 — Technical Consolidation (v1.10.0)

- **Branch coverage enforced in CI** — `pytest-cov` with `branch=true`, `fail_under=79%`. Coverage reported as XML for machine consumption and term-missing for CI logs.
- **Integration tests** (`tests/test_integration.py`) — 18 tests against real CertiSigma API: public endpoints, attestation lifecycle, metadata, tagging, sharing, derived lists, error handling. Gated by `@pytest.mark.integration` and `CERTISIGMA_API_KEY` env var. Run via `pytest -m integration`.
- **Performance regression guards** (`tests/test_performance.py`) — 3 tests with generous thresholds (hash 10K files < 30s, manifest 10K inserts < 10s, save/load < 5s). Run in every CI build (no marker). Catch catastrophic regressions, not micro-optimizations.
- **Benchmark script** (`scripts/benchmark.py`) — standalone tool for precise measurement: hash throughput, walk speed, scan sequential/parallel, manifest add/query/save-load. JSON output with platform info.
- **Integration CI workflow** (`.github/workflows/integration.yml`) — manual `workflow_dispatch` using `secrets.CERTISIGMA_API_KEY`. Never auto-triggered; requires explicit maintainer action.
- **Test count**: 760+ (unit) + 18 (integration, gated) + 3 (performance).

### Phase 20.1 — Post-implementation audit (v1.10.0 → v1.10.1)

Security audit and propagation of Phase 20 patterns:

- **5 issues fixed in Phase 20 code**: resource exhaustion in benchmark (MEDIUM), `json.dumps` without `default=str` in benchmark (MEDIUM), `sys.path.insert` antipattern (LOW), fragile ID parsing in integration tests (LOW), temp file leak (LOW).
- **22 `json.dumps` fixed in cli.py**: all error-path dicts lacked `default=str`, now consistent.
- **3 CLI integer bounds hardened**: `--poll-interval` and `--timeout` now use `click.IntRange(min=1)` (prevents tight loop / useless timeout); `--expires` uses `click.IntRange(min=1, max=2160)` (enforces documented 90-day max).
- **5 new patterns added** to Security & Quality Checklist: temp file cleanup, sys.path manipulation, CLI arg upper bounds, json.dumps in scripts, fragile ID parsing.
- `sharing._parse_attestation_id` reviewed — uses safe `removeprefix` + `try/except ValueError`. No fix needed.
- `audit_log.py` lines 50/128: intentional `json.dumps` without `default=str` for deterministic hashing (confirmed correct).

## What NOT To Do

- Do NOT modify any file in `/opt/app-certisigma/` **except** `/opt/app-certisigma/developers/site/` and `/opt/app-certisigma/developers/nginx.conf` (Census docs deployment — see §Developers Site Deployment below)
- Do NOT add Docker infrastructure to Census
- Do NOT import from `backend.app.*` or any internal module
- Do NOT hardcode API keys
- Do NOT store sensitive data in the manifest (it's a local SQLite file, not encrypted)
- Do NOT add dependencies without checking they're actively maintained and necessary
- Do NOT skip tests

## Developers Site Deployment

Census deploys `docs/census.html` to the CertiSigma developers site (`developers.certisigma.ch/census`). This requires writing to files **outside** the Census repo:

- `/opt/app-certisigma/developers/site/census.html` — the page itself (copy from `docs/census.html`)
- `/opt/app-certisigma/developers/site/sitemap.xml` — add `/census` URL entry
- `/opt/app-certisigma/developers/site/llms.txt` — add Census CLI section
- `/opt/app-certisigma/developers/nginx.conf` — add `/census` location block (same pattern as `/sdk`)

After updating these files, rebuild and redeploy:
```bash
cd /opt/app-certisigma
docker compose -f docker-compose.developers.yml build
docker compose -f docker-compose.developers.yml up -d
```

**Design system**: `census.html` must share the same CSS variables, fonts, layout, nav bar, sidebar TOC, command palette, and theme toggle as `sdk.html`. localStorage key for theme is `cs-theme` (shared across pages).

**Cross-page navigation**: Both `sdk.html` and `census.html` nav bars must link to each other (API Reference, SDKs, Census).
