# CertiSigma Census — Roadmap

**Version:** 1.10.1
**Date:** 2026-03-21
**Status:** Phase 20.1 — Post-implementation audit

---

## Visione

Census è un prodotto applicativo che utilizza l'infrastruttura CertiSigma (API + SDK) per il censimento crittografico di asset informativi, il rilevamento di esfiltrazioni e la cooperazione selettiva con soggetti terzi (analisti forensi, operatori threat intelligence).

Census è un **client** dell'API, non un'estensione del backend. Il valore di Census non si esaurisce nell'attestare l'esistenza di un contenuto: rende quel contenuto classificabile e operativamente verificabile da terzi senza trasferirne il dato né il significato.

**Documento di riferimento:** `/opt/app-certisigma/docs/commercial/CERTISIGMA-CENSUS-v2.md`

---

## Fase 1 — CLI Core (v0.1.0)

**Obiettivo:** Tool a linea di comando funzionante per scan, attest e compare.

| Feature | Dipendenza API/SDK | Status API | Status Census |
|---------|-------------------|------------|---------------|
| Scan directory + SHA-256 | `hash_file()` | ✅ SDK v1.5.0 | ✅ Implementato |
| Manifest locale (JSON→SQLite v2) | Nessuna (locale) | N/A | ✅ Implementato |
| Batch attest (100/call) | `POST /attest/batch` | ✅ API operativo | ✅ Implementato |
| Batch verify (100/call) | `POST /verify/batch` | ✅ API operativo | ✅ Implementato |
| Source label | campo `source` | ✅ API operativo | ✅ Implementato |
| Dry run mode | Nessuna | N/A | ✅ Implementato |
| Compare report (JSON) | `batch_verify()` | ✅ SDK v1.5.0 | ✅ Implementato |
| Manifest status | Nessuna (locale) | N/A | ✅ Implementato |
| CLI via `click` | Nessuna | N/A | ✅ Implementato |

**Stima:** Pronto per test e raffinamento.

---

## Fase 2 — Produzione e robustezza (v0.2.1)

**Obiettivo:** Hardening per uso enterprise reale.

| Feature | Dipendenza API/SDK | Status API | Status Census |
|---------|-------------------|------------|---------------|
| Ripresa scansione interrotta (`--resume`) | Nessuna (locale) | N/A | ✅ Implementato |
| Retry con backoff su 429/5xx | `RateLimitError`, `Retry-After` | ✅ SDK v1.5.0 | ✅ Implementato |
| Progress bar (file grandi) | Nessuna | N/A | ✅ Implementato |
| Filtri per estensione/dimensione | Nessuna (locale) | N/A | ✅ Implementato |
| Esclusione pattern (glob) | Nessuna (locale) | N/A | ✅ Implementato |
| Report CSV export | Nessuna (locale) | N/A | ✅ Implementato |
| Logging strutturato (JSON) | Nessuna | N/A | ✅ Implementato |
| Test suite + CI (106 test + 17 markers) | Nessuna | N/A | ✅ Implementato |
| Documentazione features (`docs/features/`) | Nessuna | N/A | ✅ Implementato |
| Pubblicazione PyPI | Nessuna | N/A | ✅ Implementato |

**Completata** il 2026-03-18.

---

## Fase 3 — Watch mode + SQLite manifest (v0.3.0)

**Obiettivo:** Monitoraggio continuo filesystem, manifest scalabile, attestazione automatica.

| Feature | Dipendenza API/SDK | Status API | Status Census |
|---------|-------------------|------------|---------------|
| Manifest SQLite (WAL, schema v2) | Nessuna (locale) | N/A | ✅ Implementato |
| Migrazione automatica JSON→SQLite | Nessuna (locale) | N/A | ✅ Implementato |
| `census watch /path` (inotify/fsevents) | `attest()` | ✅ SDK v1.5.0 | ✅ Implementato |
| Debounce su modifiche rapide | Nessuna | N/A | ✅ Implementato |
| Aggiornamento incrementale manifest | Nessuna (locale) | N/A | ✅ Implementato |
| Per-event handling (CREATE/MODIFY/MOVE/DELETE) | Nessuna | N/A | ✅ Implementato |
| Baseline scan on start | Nessuna | N/A | ✅ Implementato |
| PollingObserver per NFS/CIFS | Nessuna | N/A | ✅ Implementato |
| Rilevamento limite inotify | Nessuna | N/A | ✅ Implementato |
| Shutdown graceful (SIGINT/SIGTERM) | Nessuna | N/A | ✅ Implementato |
| Prevenzione symlink escape | Nessuna | N/A | ✅ Implementato |
| Delete policies (ignore/mark/remove) | Nessuna | N/A | ✅ Implementato |
| Template systemd / launchd | Nessuna | N/A | ✅ Documentato |

**Completata** il 2026-03-19.
**Dipendenze:** `watchdog>=4.0.0` (opzionale, `pip install certisigma-census[watch]`).

**Debito tecnico correlato:** vedi `docs/TECHNICAL-DEBT.md` per eBPF (TD-001) e auditd/who-data (TD-002).

---

## Phase 3.5 — Forensic Evidence Pipeline (v0.4.0)

**Obiettivo:** Verifica hash, integrity check e reportistica forense.

| Feature | Dipendenza API/SDK | Status API | Status Census |
|---------|-------------------|------------|---------------|
| `census verify <hash>` (T0/T1/T2 chain) | `verify()`, `get_evidence()` | ✅ API operativo | ✅ Implementato |
| `census verify --file` (hash then verify) | `hash_file()` + `verify()` | ✅ SDK v1.5.0 | ✅ Implementato |
| `census verify --save-ots` (OTS export) | `save_ots_proof()` | ✅ SDK v1.5.0 | ✅ Implementato |
| `census integrity manifest.db` | Nessuna (locale) | N/A | ✅ Implementato |
| `census integrity --strict` (exit code 1) | Nessuna (locale) | N/A | ✅ Implementato |
| Report HTML (zero deps) | Nessuna | N/A | ✅ Implementato |
| Report PDF (fpdf2) | `fpdf2>=2.8.0` (opzionale) | N/A | ✅ Implementato |
| Evidence bundle (ZIP + OTS + SHA256SUMS) | `save_ots_proof()` | ✅ SDK v1.5.0 | ✅ Implementato |
| XSS prevention in HTML | Nessuna | N/A | ✅ Implementato |
| Path traversal prevention in ZIP | Nessuna | N/A | ✅ Implementato |
| Test suite (149 test + 17 markers) | Nessuna | N/A | ✅ Implementato |

**Completata** il 2026-03-18.
**Dipendenze:** `fpdf2>=2.8.0` (opzionale, `pip install certisigma-census[report]`).

---

## Phase 4 — Forensic Diff and CLI Maturity (v0.5.0)

**Obiettivo:** Manifest diffing, standalone hashing, attestation tracking, config file, shell completions.

| Feature | Dipendenza API/SDK | Status API | Status Census |
|---------|-------------------|------------|---------------|
| `census diff m1.db m2.db` (manifest comparison) | Nessuna (locale) | N/A | ✅ Implementato |
| Diff HTML report (color-coded added/removed/modified) | Nessuna (locale) | N/A | ✅ Implementato |
| Diff bitmask exit codes (AIDE-style: 1=added, 2=removed, 4=modified) | Nessuna | N/A | ✅ Implementato |
| `census hash <file\|dir>` (standalone SHA-256) | `hash_file()` | ✅ SDK v1.5.0 | ✅ Implementato |
| `census hash --verify <hash>` (hash and compare) | `hash_file()` | ✅ SDK v1.5.0 | ✅ Implementato |
| `census track <att_id>` (attestation status) | `client.status()` | ✅ API operativo | ✅ Implementato |
| `census track --poll` (wait for T2 anchoring) | `client.status()` | ✅ API operativo | ✅ Implementato |
| Config file (TOML) con precedenza CLI > env > project > user | Nessuna | N/A | ✅ Implementato |
| `census config show\|init\|paths` | Nessuna | N/A | ✅ Implementato |
| `census completion bash\|zsh\|fish` (shell completions) | Click built-in | N/A | ✅ Implementato |

**Completata** il 2026-03-19.
**Dipendenze:** `tomli>=2.0; python_version < "3.11"` (per config TOML su Python 3.10).

---

## Phase 5 — Enterprise Readiness (v0.6.0)

**Obiettivo:** Self-diagnostics, distributed workflows, e automation-ready JSON output per tutti i comandi.

| Feature | Dipendenza API/SDK | Status API | Status Census |
|---------|-------------------|------------|---------------|
| `census doctor` (connectivity, config, inotify, deps) | `client.health()` | ✅ SDK v1.5.0 | ✅ Implementato |
| `census merge m1.db m2.db -o out.db` (merge manifests) | Nessuna (locale) | N/A | ✅ Implementato |
| `census scan --json` (structured output) | Nessuna | N/A | ✅ Implementato |
| `census compare --json` (structured output) | Nessuna | N/A | ✅ Implementato |
| `census status --json` (structured output) | Nessuna | N/A | ✅ Implementato |

**Completata** il 2026-03-19.
**Dipendenze:** Nessuna.

---

## Phase 5.1 — Operational polish (v0.6.1)

**Obiettivo:** Chiudere il cerchio su hardening post-audit, UX automation e test di regressione.

| Feature | Dipendenza API/SDK | Status API | Status Census |
|---------|-------------------|------------|---------------|
| `census doctor` — HTTP timeout breve per health check | `CertiSigmaClient(timeout=…)` | ✅ SDK v1.5.0 | ✅ Implementato |
| Watch manifest bootstrap — `save()` poi `load()` senza leak `:memory:` | Nessuna | N/A | ✅ Implementato |
| `census compare --json` + `--output` — avviso su stderr, nessun file scritto | Nessuna | N/A | ✅ Implementato |
| Suite test: watcher manifest init, doctor timeout, compare UX | Nessuna | N/A | ✅ Implementato |

**Completata** il 2026-03-19.

---

## Phase 6 — Compliance & Zero-Knowledge (v0.7.0)

**Obiettivo:** Audit trail tamper-evident, snapshot per compliance baseline, annotazione forense con cifratura zero-knowledge.

| Feature | Dipendenza API/SDK | Status API | Status Census |
|---------|-------------------|------------|---------------|
| `census audit-log show\|verify\|clear` (hash chain JSONL) | Nessuna (locale) | N/A | ✅ Implementato |
| Audit automatico su scan/compare | Nessuna (locale) | N/A | ✅ Implementato |
| `census snapshot create\|list\|diff\|delete` | Nessuna (locale) | N/A | ✅ Implementato |
| `census annotate <att_id> --note/--tag/--case-id` | `update_metadata()` | ✅ SDK v1.5.0 | ✅ Implementato |
| `census annotate --delete` (GDPR erasure) | `delete_metadata()` | ✅ SDK v1.5.0 | ✅ Implementato |
| `census annotate --encrypt` (AES-256-GCM zero-knowledge) | `certisigma.crypto` | ✅ SDK v1.5.0 | ✅ Implementato |

**Completata** il 2026-03-19.
**Dipendenze:** `cryptography` (transitive via certisigma SDK).

---

## Phase 7 — Forensic Sharing, Tagging & Derived Lists (v0.8.0)

**Obiettivo:** Forensic sharing, structured tagging, key rotation, derived lists — tutte le feature sbloccate da SDK v1.7.0.

| Feature | Dipendenza API/SDK | Status API | Status Census |
|---------|-------------------|------------|---------------|
| `census share` (create/list/revoke/info) | `create_share_token()` + §30 | ✅ SDK v1.6.0 | ✅ Implementato |
| `census tag` (set/get/delete/query) | `put_tags()`, `query_tags()` + §32 | ✅ SDK v1.6.0 | ✅ Implementato |
| `census key-rotate` (AES-256 rotation) | `certisigma.crypto.rotate_key()` | ✅ SDK v1.6.0 | ✅ Implementato |
| `census derived-list` (create/list/info/match/revoke/access-log) | `create_derived_list()` + §33 | ✅ SDK v1.7.0 | ✅ Implementato |
| `census metadata get` | `get_metadata()` | ✅ SDK v1.6.0 | ✅ Implementato |

**Completata** il 2026-03-20.
**Dipendenze:** `certisigma>=1.7.0`.

---

## Phase 8 — Forensic Integrity & Performance (v0.9.0)

**Obiettivo:** Verifica crittografica delle derived list, generazione chiavi, compare enriched, parallel hashing.

| Feature | Dipendenza API/SDK | Status API | Status Census |
|---------|-------------------|------------|---------------|
| `census derived-list signature <list_id>` | `get_derived_list_signature()` | ✅ SDK v1.7.0 | ✅ Implementato |
| `census key-gen` (AES-256 key generation) | `certisigma.crypto.generate_key()` | ✅ SDK v1.6.0 | ✅ Implementato |
| `census compare --detailed` (enriched batch_verify) | `batch_verify(detailed=True)` | ✅ SDK v1.5.0 | ✅ Implementato |
| Parallel hashing `--workers N` (scan + compare) | Nessuna (locale) | N/A | ✅ Implementato |
| `source` field in MatchResult and CSV export | `batch_verify(detailed=True)` | ✅ SDK v1.5.0 | ✅ Implementato |
| Cleanup test_future.py (§30/§32/§33 skip markers removed) | N/A | N/A | ✅ Implementato |

**Completata** il 2026-03-20.
**Dipendenze:** Nessuna nuova.

---

## Phase 9 — Production Release (v1.0.0)

**Obiettivo:** Full-chain verification, resource management, forensic interop, production polish.

| Feature | Dipendenza API/SDK | Status API | Status Census |
|---------|-------------------|------------|---------------|
| `census verify-manifest` (full chain verification) | `batch_verify()` | ✅ SDK v1.5.0 | ✅ Implementato |
| `census verify-manifest --detailed --strict` | `batch_verify(detailed=True)` | ✅ SDK v1.5.0 | ✅ Implementato |
| `census verify-manifest -o report.csv` | Nessuna (locale) | N/A | ✅ Implementato |
| CertiSigmaClient resource management (`close()`) | `CertiSigmaClient.__exit__` | ✅ SDK v1.7.0 | ✅ Implementato |
| NotFoundError/ValidationError handling | `certisigma.exceptions` | ✅ SDK v1.7.0 | ✅ Implementato |
| `census hash --stdin` (streaming SHA-256) | `hashlib` (stdlib) | N/A | ✅ Implementato |
| `census export --format sha256sum` (GNU format) | Nessuna (locale) | N/A | ✅ Implementato |

**Completata** il 2026-03-20.
**Dipendenze:** Nessuna nuova.

---

## Phase 10 — Forensic Hardening (v1.1.0)

**Obiettivo:** Manifest integrity seal, manifest self-attestation, quiet mode per automation.

| Feature | Dipendenza API/SDK | Status API | Status Census |
|---------|-------------------|------------|---------------|
| `census seal <manifest>` (HMAC-SHA256 tamper seal) | Nessuna (locale) | N/A | ✅ Implementato |
| `census verify-seal <manifest>` (seal verification) | Nessuna (locale) | N/A | ✅ Implementato |
| `census scan --attest-manifest` (self-attestation) | `client.attest()` | ✅ SDK v1.5.0 | ✅ Implementato |
| `--quiet / -q` global flag (script/cron mode) | Nessuna | N/A | ✅ Implementato |
| Classifier updated to Production/Stable | Nessuna | N/A | ✅ Implementato |
| TD-004 closed (parallel hashing shipped v0.9.0) | N/A | N/A | ✅ Chiuso |

**Completata** il 2026-03-20.
**Dipendenze:** Nessuna nuova.

---

## Fase 11 — Bulk Scale & SARIF (v1.2.0)

**Obiettivo:** Bulk leak detection via `/scan` endpoint, org-level stats via `/bulk/stats`, SARIF v2.1.0 output per CI/CD integration.

| Feature | Dipendenza API/SDK | Status API | Status Census |
|---------|-------------------|------------|---------------|
| `census bulk-scan <dir>` (50K hashes/call) | `client.scan()` | ✅ SDK v1.8.0 | ✅ Implementato |
| `census stats` (org-level inventory) | `client.get_bulk_stats()` | ✅ SDK v1.8.0 | ✅ Implementato |
| `census compare --format sarif` (SARIF v2.1.0) | Nessuna (locale) | N/A | ✅ Implementato |
| Chunking automatico per >50K hashes | Nessuna (locale) | N/A | ✅ Implementato |
| Cross-reference con manifest locale | Nessuna (locale) | N/A | ✅ Implementato |
| Full test suite (544 tests, 0 skipped) | N/A | N/A | ✅ Implementato |

**Completata** il 2026-03-20.
**Dipendenze:** `certisigma>=1.8.0`.

---

## Phase 14 — Automation & SIEM Integration (v1.4.0)

**Obiettivo:** Log streaming per SIEM/SOC, notification hooks per incident response, templates per deploy schedulato.

| Feature | Dipendenza API/SDK | Status API | Status Census |
|---------|-------------------|------------|---------------|
| `--format jsonl` (compare, bulk-scan, integrity, diff, verify-manifest) | Nessuna (locale) | N/A | ✅ Implementato |
| `--on-match CMD` (compare, bulk-scan) | Nessuna (locale) | N/A | ✅ Implementato |
| `_emit_jsonl` helper (summary trailer) | Nessuna | N/A | ✅ Implementato |
| `_run_on_match` helper (30s timeout, best-effort) | Nessuna | N/A | ✅ Implementato |
| systemd timer templates | Nessuna | N/A | ✅ Documentato |
| Cron example script | Nessuna | N/A | ✅ Documentato |
| Test suite (571 tests) | N/A | N/A | ✅ Implementato |

**Completata** il 2026-03-18.
**Dipendenze:** Nessuna nuova.

---

## Phase 15 — AIDE-Style Update (v1.5.0)

**Obiettivo:** Accettare le modifiche verificate nel baseline del manifest, completando il workflow FIM: detect → review → accept.

| Feature | Dipendenza API/SDK | Status API | Status Census |
|---------|-------------------|------------|---------------|
| `census update manifest.db` (baseline update) | Nessuna (locale) | N/A | ✅ Implementato |
| `--yes` (non-interactive mode) | Nessuna | N/A | ✅ Implementato |
| `--json` output | Nessuna | N/A | ✅ Implementato |
| `UpdateResult` dataclass | Nessuna | N/A | ✅ Implementato |
| Audit logging | Nessuna | N/A | ✅ Implementato |
| Test suite (584 tests) | N/A | N/A | ✅ Implementato |

**Completata** il 2026-03-20.
**Dipendenze:** Nessuna nuova.

---

## Fase 12 — Integrazioni (v2.x)

**Obiettivo:** Canali di distribuzione, integrazioni e casi d'uso avanzati.

| Feature | Dipendenza | Priorità | Note |
|---------|-----------|----------|------|
| Nextcloud app | Plugin PHP/app esterna | P3 | Solo se mercato PMI lo richiede |
| GUI desktop (Tauri/Electron) | Census CLI come backend | P3 | Versione consumer, settimane di lavoro |
| Webhook su match trovato | Webhook API | ✅ Già disponibile | Census come consumer di webhook |
| ~~AI governance: allow/deny lists~~ | §32 + §33 | ~~P2~~ | ✅ Implementato in v1.9.0 (`census ai-policy`) |
| RBAC scoped keys | §31 | ✅ Implementato | API key read-only per analisti terzi con audit trail |

---

## Feature API/Backend da cui Census dipenderà in futuro

Le seguenti feature backend impattano (o impatteranno) Census:

| Feature backend | §Tech Debt | Status | Priorità | Impatto su Census |
|----------------|-----------|--------|----------|-------------------|
| **Request/Correlation IDs** | §29 | ✅ Implementato | — | Trasparente via SDK, nessun codice Census da toccare |
| **Key rotation automatica** | Esistente | ✅ Implementato | — | Trasparente via SDK |
| **Endpoint bulk (10K+)** | §10 | ✅ Implementato | — | `census bulk-scan` e `census stats` in v1.2.0 |
| **Forensic metadata sharing** | §30 | ✅ Implementato | — | `census share` implementato in v0.8.0 |
| **RBAC / Scoped read-only keys** | §31 | ✅ Implementato | — | API key per analisti con audit trail |
| **Structured Tagging** | §32 | ✅ Implementato | — | `census tag` implementato in v0.8.0 |
| **Derived Lists (HMAC)** | §33 | ✅ Implementato | — | `census derived-list` implementato in v0.8.0, signature in v0.9.0 |

Census è sempre downstream: consuma l'SDK pubblicato, non anticipa API inesistenti.

---

## Readiness checklist (cosa fare quando arrivano le API)

Test-marcatori in `tests/test_future.py` (skippati, visibili con `pytest -rs`).

### §30 Forensic Sharing — COMPLETATA (v0.8.0)
- [x] `census share create/list/revoke/info` implementato
- [x] Time-limited, use-limited tokens con audit trail
- [x] Test suite: `test_sharing.py`, `test_cli.py::TestShareCommand`

### §32 Structured Tagging — COMPLETATA (v0.8.0)
- [x] `census tag set/get/delete/query` implementato
- [x] Client-side encryption per tag values (AES-256-GCM)
- [x] Test suite: `test_tagging.py`, `test_cli.py::TestTagCommand`

### §33 Derived Lists — COMPLETATA (v0.8.0, signature in v0.9.0)
- [x] `census derived-list create/list/info/match/revoke/access-log/signature`
- [x] HMAC-SHA256 derivation con test vector validation
- [x] ECDSA signature verification per integrity check
- [x] Test suite: `test_derived_lists.py`, `test_cli.py::TestDerivedListCommand`

### §10 Bulk Endpoints — COMPLETATA (v1.2.0)
- [x] `census bulk-scan` — 50K hashes/call via `client.scan()`
- [x] `census bulk-scan --dry-run` — hash and count, no API call
- [x] `census bulk-scan --source` — audit label for incident tracking
- [x] `census bulk-scan --output` — save results to JSON file
- [x] `census stats` — org-level inventory stats via `client.get_bulk_stats()`
- [x] Chunking automatico per >50K hashes
- [x] Cross-reference con manifest locale (hash_to_paths: all paths per hash)
- [x] `census compare --format sarif` — SARIF v2.1.0 with help, tags, invocations
- [x] `census compare --format sarif --output` — write SARIF directly to file
- [x] Full test suite: 19 bulk-scan/stats tests + 25 SARIF tests (544 total)

### Phase 12 — CI/CD Integration Hardening (v1.3.0)
- [x] `compare --exit-zero` — report-only mode (always exit 0, CI pipelines)
- [x] `compare --summary` — counts only, no individual match details
- [x] `bulk-scan --exit-zero` — same pattern for bulk scan
- [x] `bulk-scan --summary` — same pattern for bulk scan
- [x] Feature docs: `bulk-scan.md`, `stats.md`, `sarif.md`
- [x] Updated `comparison.md` with CI/CD integration section
- [x] Full test suite: 556 tests (12 new CI/CD integration tests)

### Phase 13 — Production Hardening + Output Hygiene (v1.3.1)
- [x] Fix dead code after `sys.exit(1)` in `report_cmd`
- [x] Fix `baseline.close()` resource leak in watcher
- [x] `--no-color` + `NO_COLOR` env var support (no-color.org)
- [x] `census_version` in all JSON output for forensic traceability
- [x] `elapsed_seconds` timing in scan/compare/bulk-scan JSON + text output
- [x] Full test suite: 562 tests (6 new production hardening tests)

### Phase 13.2 — Propagation Audit (v1.3.2)
- [x] Resource cleanup (`try/finally`): 3 new leak sites fixed (manifest.save, scan resume, watcher cleanup)
- [x] `_with_meta` propagated to 14 additional commands (status, verify, hash, track, integrity, verify-manifest, diff, stats, merge, seal, verify-seal, doctor, metadata, key-gen)
- [x] `_with_meta` made non-mutating (`{**data, ...}` pattern)
- [x] `hash --json` output normalized to dict format (`{"files": [...]}`) for consistency
- [x] `--quiet` compliance: core commands (report, track, merge, doctor, watcher) now use `_echo_info`/quiet-aware helpers
- [x] watcher.py: new `quiet` parameter, all informational output gated
- [x] Full test suite: 562 tests, 0 failures

### Phase 14 — Automation & SIEM Integration (v1.4.0)
- [x] `--format jsonl` on compare, bulk-scan, integrity, diff, verify-manifest — one JSON object per result, streamed
- [x] `--on-match CMD` on compare, bulk-scan — execute command with results as JSON on stdin when matches > 0
- [x] `_emit_jsonl` helper — consistent JSONL output with `_summary` trailer (count, census_version, elapsed_seconds)
- [x] `_run_on_match` helper — best-effort subprocess execution with 30s timeout, warning on failure
- [x] systemd timer templates: `census-integrity.service` + `.timer` for periodic integrity checks
- [x] Cron example script: `docs/templates/cron-example.sh`
- [x] Format normalization: `--format` + `--json` coexist cleanly across all commands
- [x] Full test suite: 571 tests (9 new: JSONL, on-match, Phase 13.2 meta coverage), 0 failures

### Phase 15 — AIDE-Style Update + Consistency Hardening (v1.5.0)
- [x] `census update manifest.db` — accept verified filesystem changes into baseline
- [x] `update.py` module with `update_manifest_from_integrity()` + `UpdateResult` dataclass
- [x] Interactive confirmation (or `--yes`), `--json`, `--quiet` compliance
- [x] New entries added with `attested=False`
- [x] Audit logging for compliance trail
- [x] Fix: `--format jsonl --output` warns on stderr in compare, integrity, diff
- [x] Fix: redundant `import time` in track_cmd
- [x] Full test suite: 584 tests (13 new), 0 failures

### Phase 15.1 — Security Hardening Audit (v1.5.1)
- [x] Fix: Path traversal in `update.py` — `_resolve_within_root()` with `resolve()` + `is_relative_to()` containment
- [x] Fix: `except OSError` → `except Exception` around `hash_file()` in update loops
- [x] Fix: Progress bar in `update.py` now uses proper context manager (`with` + `nullcontext`)
- [x] Fix: `manifest.save()` error handling in update_cmd
- [x] Fix: `verify-manifest --format jsonl --output` now warns on stderr (was inconsistent)
- [x] Fix: `--json` help text documents implied `--yes` behavior
- [x] Added 7 security patterns to AGENTS.md: path traversal, exception handling, progress bar, JSONL+output

### Phase 16 — Differential Integrity (v1.6.0)
- [x] `census integrity --since STATE --write-state STATE` — only report NEW findings
- [x] `differential.py` module: `IntegrityState`, `FindingState`, `build_state`, `filter_report`, `load_state`, `save_state` (atomic)
- [x] Per-finding `first_seen` (ISO-8601) + `occurrences` counter (Wazuh-inspired, unique vs all competitors)
- [x] Auto-sidecar: `--since auto --write-state auto` → `manifest.db.integrity-state`
- [x] JSON/JSONL enrichment: `differential`, `suppressed`, `new_findings`, `state_path`
- [x] State file: versioned JSON (`schema_version: 1`), 50MB size limit
- [x] Fix: missing `_audit("integrity", ...)` — compliance gap closed
- [x] Full test suite: 612 tests (26 new), 0 failures

### Phase 16.1 — Propagation Audit (v1.6.1)
- [x] Type validation on deserialized JSON: `seal.py`, `snapshot.py`, `audit_log.py`, `cli.py` (4 sites)
- [x] `except OSError` too narrow around serialization: widened + `default=str` added (4 sites)
- [x] JSONL `_summary` metadata parity: `compare`, `integrity`, `verify-manifest`, `diff`, `bulk-scan` (5 commands)
- [x] Relative path fragility: `.resolve()` added to `snapshot.py`, `audit_log.py` path overrides
- [x] Post-mutation of `_with_meta()` dicts: refactored `scan`, `bulk-scan` (3 sites)
- [x] Dead import: `watcher.py` `import sys` removed
- [x] `snapshot.py:list_snapshots` except clause: added `sqlite3.Error` for corrupt DB files
- [x] Full test suite: 627 tests (10 new), 0 failures

### Phase 17 — GitHub Action (v1.7.0)
- [x] `action.yml` composite action: `certisigma/census-action@v1`
- [x] Commands: `scan`, `integrity`, `compare`, `bulk-scan`
- [x] 11 inputs: command, target, manifest, api-key, source, format, upload-sarif, exit-zero, version, extra-args, python-version
- [x] 3 outputs: exit-code, output-file, sarif-file
- [x] SARIF auto-upload to GitHub Security tab via `github/codeql-action/upload-sarif@v3`
- [x] Step summary: Markdown table written to `$GITHUB_STEP_SUMMARY`
- [x] Secret masking: API key masked via `::add-mask::`
- [x] Shell injection prevention: all inputs passed via `env:` context (never interpolated in `run:`)
- [x] Version pinning: `inputs.version` pins Census PyPI version for reproducibility
- [x] `docs/features/github-action.md` with 6 workflow examples
- [x] action.yml schema validation test + YAML lint test
- [x] Full test suite: 644 tests (17 new), 0 failures

### Phase 17.1 — GitHub Action audit (v1.7.1)
- [x] Fail-fast: `format=sarif` rejected unless `command=compare` (no silent wrong-format run)
- [x] Removed unused `GITHUB_OUTPUT` append from Install step (delimiter-injection hygiene)
- [x] AGENTS.md: extended GitHub Action patterns (permissions, GITHUB_OUTPUT, extra-args trust, step summary)
- [x] AGENTS.md: propagation audit — no other composite actions; CI workflows clean; `test_action` in CI
- [x] `docs/features/github-action.md`: maintainer release checklist (pytest + `./scripts/test-census.sh X.Y.Z`)
- [x] `tests/test_action.py`: +2 tests (19 total for action.yml)
- [x] Docs: `github-action.md` Security + SARIF sections corrected
- [x] Full test suite: 646 tests (2 new), 0 failures
- [x] Release: tag `v1.7.1` pushed to origin; PyPI via `.github/workflows/publish.yml` on tag (OIDC) — local `twine` optional if token configured

### Phase 18 — Developers Page + CI Dogfood + Compliance Templates (v1.8.0)
- [x] `docs/census.html` — standalone developers page (same design system as sdk.html)
  - 14 sections, dark/light toggle, sidebar TOC, command palette search (Ctrl+K)
  - Nav: API Reference, SDKs, Census (active)
- [x] `.github/workflows/dogfood.yml` — CI end-to-end validation of `action.yml`
  - Triggers on push/PR touching `action.yml`
  - Creates test manifest with `--dry-run`, runs `uses: ./` with `command: integrity`
  - No API key, no secrets — 100% local
- [x] `census compliance-report` — NIS2/DORA/ISO 27001 compliance mapping
  - `compliance.py` module: `generate_compliance_report()`, `render_compliance_html()`, `render_compliance_json()`
  - Templates: nis2 (10 requirements), dora (8), iso27001 (9)
  - `--integrity` flag to include integrity check results
  - 100% local, no API calls
- [x] AGENTS.md: GHA moved to Implemented, Publishing section fixed (twine + OIDC)
- [x] README.md: compliance-report in feature table, developers page link
- [x] Full test suite: 700 tests (54 new), 0 failures
  - `test_census_html.py`: 19 tests (structure, section IDs, XSS, noopener)
  - `test_dogfood_workflow.py`: 9 tests (structure, permissions, no secrets)
  - `test_compliance.py`: 26 tests (NIS2/DORA/ISO templates, HTML/JSON, CLI)

### Phase 18.1 — Post-implementation audit (v1.8.0 → v1.8.1)
- [x] Dead imports removed: `compliance.py`, `key_rotation.py`, `doctor.py`, `annotate.py`, `diff.py`, 5 test files
- [x] Manifest resource leaks: 70+ `Manifest` objects in tests converted to `yield` + `close()` pattern
- [x] `CSS.escape()` in `census.html` for `querySelector` with dynamic IDs
- [x] `localStorage` key aligned: `census-theme` → `cs-theme` for cross-page consistency
- [x] Unused `quiet` variable removed from `compliance_report_cmd`
- [x] `html.escape()` added to TOC/search `innerHTML` in `census.html` (DOM XSS prevention)
- [x] AGENTS.md: 5 new security patterns documented

### Phase 19 — Security audit + AI Governance (v1.8.1 → v1.9.0)
- [x] `manifest.py`: connection leak guard in `_load_from_sqlite` / `_load_from_json` (`try/except: conn.close()`)
- [x] `retry.py`: `retry_after` guarded for None/negative/non-numeric, capped at `MAX_BACKOFF`
- [x] `scanner.py`: `ValueError` from `relative_to()` caught alongside `OSError` in `_scan_parallel`
- [x] `annotate.py`: `KeyError` added to crypto except clause for malformed SDK envelopes
- [x] `tagging.py`: `_resolve_encryption_key()` + crypto import hoisted out of per-tag loops
- [x] `update.py`: stat-before-hash reorder to fix TOCTOU race
- [x] `key_rotation.py`: batch tag rotation + partial rotation warning
- [x] `cli.py`: `default=str` added to 20+ `json.dumps()` calls
- [x] AGENTS.md: 7 new vulnerability patterns documented
- [x] Full test suite: 700 tests, 0 failures
- [x] `ai_policy.py`: TOML policy engine — load_policy, classify_manifest, tag_classifications, render reports
- [x] CLI: `census ai-policy init/apply/report` with `--dry-run`, `--json`, `--quiet` compliance
- [x] Policy format: `.census-ai-policy.toml` with `[policy]` + `[[rules]]` (glob patterns, size filters)
- [x] Regulatory mapping: EU AI Act, ISO/IEC 42001, C2PA
- [x] Most-restrictive-wins on shared `attestation_id`
- [x] Tag key: `ai_training=allowed/excluded` via existing `set_tags()` infrastructure
- [x] HTML/JSON report generation (same style as compliance.py)
- [x] 47 new tests (test_ai_policy.py): TOML parsing, rule matching, coverage, XSS, CLI
- [x] Full test suite: 747 tests, 0 failures

### Phase 19.1 — Post-implementation audit (v1.9.0 → v1.9.1)
- [x] Fix: `"excluded"` vs `"exclude"` comparison in tag dedup guard (HIGH — latent logic bug)
- [x] Fix: per-rule pattern limits (MAX_PATTERNS_PER_RULE=1000, MAX_PATTERN_LEN=512)
- [x] Fix: policy file size limit (10 MB pre-check)
- [x] Fix: field length cap (MAX_FIELD_LEN=256 on policy name)
- [x] Fix: bool rejection in TOML size fields (`true` is `int` in Python)
- [x] Fix: negative size and min>max validation
- [x] Fix: logging in tag exception handler (silent swallow → `logger.warning()`)
- [x] Fix: UTF-8 BOM handling (`utf-8-sig` codec)
- [x] Fix: TOCTOU in `ai-policy init` (O_CREAT|O_EXCL atomic create)
- [x] Fix: exit non-zero on tagging errors in `ai-policy apply`
- [x] Fix: SARIF `json.dumps` missing `default=str`
- [x] Fix: `merged.save()` except clause widened for `sqlite3.DatabaseError`
- [x] Fix: 14 Phase 7+ commands converted from `click.echo` to `_echo_info` for `--quiet` compliance
- [x] AGENTS.md: 15 new vulnerability patterns documented
- [x] 10 new tests: bool/negative/inverted sizes, pattern limits, unicode, size=0, all-excluded, dedup correctness
- [x] Full test suite: 757 tests, 0 failures

### Phase 19.2 — Propagation audit (v1.9.1 → v1.9.2)
- [x] Fix: `scan_cmd` exits non-zero on complete attestation failure (HIGH)
- [x] Fix: `update_cmd` exits non-zero on update errors (HIGH)
- [x] Fix: `bulk_scan_cmd` hash errors now logged (`logger.warning`) instead of bare `except Exception:` (MEDIUM)
- [x] Fix: `report_cmd` except clause widened to `(OSError, TypeError, ValueError)` (MEDIUM)
- [x] Fix: `audit_log_cmd` except clauses widened to `(OSError, ValueError)` for JSONL parsing (MEDIUM)
- [x] Fix: `snapshot create` except clause widened for `sqlite3.DatabaseError` (MEDIUM)
- [x] Fix: `compliance-report` except clause widened to `(OSError, TypeError, ValueError)` (MEDIUM)
- [x] Fix: 3 `json.dumps` calls for decrypted data now include `default=str` (MEDIUM)
- [x] Fix: `html.escape()` defense-in-depth in `compliance.py` and `report.py` integrity sections (LOW)
- [x] Fix: TOCTOU-safe `config init` with `O_CREAT | O_EXCL` (LOW)
- [x] Fix: config file size limit `MAX_CONFIG_FILE_SIZE = 1 MB` (LOW)
- [x] Fix: `--quiet` compliance for `derived-list create` and `config init` (LOW)
- [x] Test fixes: `test_scan_workers_flag` uses `--dry-run`, `test_bulk_scan_hash_errors` mocks logger
- [x] 14 new propagation patterns in AGENTS.md
- [x] Full test suite: 757 tests, 0 failures

### Phase 20 — Technical Consolidation (v1.9.2 → v1.10.0)
- [x] Branch coverage enforcement: `pytest-cov` with `branch=true`, `fail_under=79%`
- [x] Coverage config in `pyproject.toml` (`[tool.coverage.run/report/html]`)
- [x] CI updated: `pytest --cov --cov-report=xml --cov-report=term-missing`
- [x] Integration tests: `tests/test_integration.py` (18 tests against real API)
  - Public endpoints: health, verify, batch_verify
  - Authenticated lifecycle: attest, batch_attest, attest_file, evidence, status
  - Metadata, tagging, sharing, derived lists CRUD
  - Error handling: AuthenticationError, CertiSigmaError
  - Gated by `@pytest.mark.integration` + `CERTISIGMA_API_KEY` env var
- [x] Performance regression guards: `tests/test_performance.py` (3 tests in every CI build)
  - Hash 10K files < 30s, manifest 10K inserts < 10s, save/load < 5s
- [x] Benchmark script: `scripts/benchmark.py` with JSON output
  - Hash throughput, walk speed, scan sequential/parallel, manifest CRUD
- [x] Integration CI: `.github/workflows/integration.yml` (manual workflow_dispatch)
- [x] README: badges (CI, PyPI, Python, coverage), testing section
- [x] Deferred: TD-007 (distribution packaging), TD-008 (signed PDFs), TD-009 (app integrations)
- [x] Full test suite: 760 passed (unit) + 18 gated (integration), baseline 81% branch coverage

### Phase 20.1 — Post-implementation audit (v1.10.0 → v1.10.1)
- [x] Fixed 5 issues in Phase 20 code (resource exhaustion, json.dumps, sys.path, ID parsing, temp leak)
- [x] Fixed 22 json.dumps in cli.py error paths (added default=str)
- [x] Hardened 3 CLI options: --poll-interval, --timeout (IntRange min=1), --expires (IntRange 1–2160)
- [x] Added 5 new patterns to AGENTS.md Security & Quality Checklist
- [x] Propagation audit: all patterns verified clean across full codebase

---

## Out of scope (deferred)

- **TD-002 (auditd/who-data)** — only on explicit customer request or compliance requirement
- **TD-003 (manifest encryption)** — only on explicit customer request
- **Webhooks integration** — deferred to when `census track --poll` replacement is prioritized
- **TD-007 (distribution packaging)** — man pages, .deb/.rpm, Docker scanner image
- **TD-008 (signed PDF reports)** — PDF forensic reports with ECDSA proof chain
- **TD-009 (application integrations)** — Nextcloud/ownCloud, web dashboard (`census serve`)

---

## Principi architetturali

1. **Census è un client, non un'estensione del backend.** Usa solo il SDK pubblicato (`pip install certisigma`). Zero dipendenze source-level dal backend.
2. **Il contenuto non lascia mai il client.** Solo gli hash SHA-256 vengono trasmessi.
3. **Il manifest è locale.** La mappa hash→filepath vive sul filesystem del cliente. CertiSigma non la vede.
4. **Ogni feature API nuova che Census richiede va nel core**, perché è utile per tutti i clienti, non solo per Census.

---

**Author:** Ten Sigma Sagl
