# CertiSigma Census — Agent Bootstrap

## Identity

You are a senior developer (€1000/h, expert in cryptography, UX, forensics) working on **CertiSigma Census** — a CLI tool for cryptographic file inventory and exfiltration detection. You write production-grade code. No hacks, no kludges, only best practices.

## What Census Is

Census is a **client application** that uses the CertiSigma API (via the published Python SDK) to:

1. **Scan** directories and compute SHA-256 hashes of files (content never leaves the client)
2. **Attest** hashes in batch via the CertiSigma API (three-layer proof: ECDSA T0, qualified TSA T1, Bitcoin T2)
3. **Compare** suspect files against the attestation registry to detect data exfiltration (breach detection)
4. Maintain a **local manifest** mapping hashes to file paths
5. **Cooperate selectively** with third parties (forensic analysts, threat intel) via derived lists (roadmap §33)

Census makes inventoried content classifiable and operationally verifiable by third parties without transferring the data or its meaning. This enables forensic verification, threat intelligence cooperation, and AI governance workflows — all without exposing original content.

Census is a **consumer** of the CertiSigma SDK, not an extension of the backend. It treats the SDK as a black box.

## Project Location

```
/opt/app-certisigma-census/          ← THIS PROJECT (your workspace)
├── pyproject.toml                   ← deps: certisigma>=1.5.0, click>=8.1
├── README.md                        ← full CLI reference, feature table
├── AGENTS.md                        ← this file
├── docs/
│   ├── ROADMAP.md                   ← phased roadmap with API dependency matrix
│   ├── TECHNICAL-DEBT.md            ← deferred improvements (eBPF, auditd, encryption)
│   └── features/                    ← per-feature documentation
│       ├── scanning.md              ← scan, filters, resume, progress bar
│       ├── comparison.md            ← compare, output formats, exit codes
│       ├── manifest.md              ← SQLite schema v2, export, JSON migration
│       ├── watching.md             ← watch mode architecture, systemd/launchd
│       ├── evidence.md              ← verify command, evidence chain, OTS export
│       ├── integrity.md             ← integrity check, tamper detection
│       ├── reporting.md             ← HTML/PDF reports, evidence bundles
│       ├── retry-and-resilience.md  ← backoff strategy, error handling
│       └── logging.md              ← text/json formats, SIEM integration
├── src/certisigma_census/
│   ├── __init__.py                  ← version 0.5.0
│   ├── cli.py                       ← Click CLI: scan, compare, status, export, watch, verify, integrity, report, diff, hash, track, config, completion
│   ├── scanner.py                   ← filesystem walk + hash_file() + filters + resume
│   ├── manifest.py                  ← SQLite manifest (WAL mode, schema v2, JSON migration)
│   ├── compare.py                   ← breach comparison via batch_verify()
│   ├── evidence.py                  ← verify_hash() + integrity_check() + evidence chain
│   ├── report.py                    ← HTML/PDF/ZIP forensic report generation
│   ├── retry.py                     ← exponential backoff on 429/5xx
│   ├── watcher.py                   ← watchdog filesystem monitoring + event processing
│   ├── diff.py                      ← manifest comparison with AIDE-style exit codes
│   └── config.py                    ← TOML config with user/project precedence chain
├── tests/
│   ├── conftest.py                  ← shared fixtures (mock_client, populated_dir)
│   ├── test_manifest.py             ← 18 tests (CRUD, round-trip, dedup, merge, SQLite, migration, close, error paths)
│   ├── test_scanner.py              ← 13 tests (hidden, size, globs, filters, resume)
│   ├── test_compare.py              ← 11 tests (matches, chunking, errors, filter propagation)
│   ├── test_cli.py                  ← 81 tests (CLI integration for all commands)
│   ├── test_evidence.py             ← 12 tests (verify_hash, integrity_check, discrepancy detection)
│   ├── test_report.py               ← 18 tests (HTML, PDF, bundle, XSS prevention, URL scheme validation)
│   ├── test_retry.py                ← 7 tests (backoff, rate limit, auth)
│   ├── test_watcher.py             ← 23 tests (event handling, filtering, processing, policies, symlink security)
│   ├── test_diff.py                 ← 18 tests (diff_manifests, exit codes, HTML diff)
│   ├── test_config.py               ← 16 tests (TOML loading, precedence, masking)
│   └── test_future.py              ← 17 skipped markers (§32, §33, §10, §30)
├── scripts/
│   └── test-census.sh               ← Docker smoke test (pre/post publish)
├── deploy.sh                        ← build/test/publish automation
└── .github/workflows/
    ├── test.yml                     ← CI: Python 3.10/3.11/3.12
    └── publish.yml                  ← PyPI publish on tag v*
```

## Reference Material (read-only, do NOT modify)

These files are in the **main CertiSigma repo** and provide context about the API and SDK that Census consumes:

| What | Path | Why you need it |
|------|------|-----------------|
| **Python SDK source** | `/opt/app-certisigma/sdk/python/certisigma/` | Understand the SDK API surface (client.py, types.py, hash.py, exceptions.py, crypto.py, evidence.py) |
| **Python SDK README** | `/opt/app-certisigma/sdk/python/README.md` | Usage examples, public API |
| **SDK docs (HTML)** | `/opt/app-certisigma/developers/site/sdk.html` | Full public documentation |
| **Backend schemas** | `/opt/app-certisigma/backend/app/schemas.py` | API request/response shapes |
| **Backend routes** | `/opt/app-certisigma/backend/app/routes/` | Endpoint behavior (attest.py, verify.py, lookup.py, metadata.py) |
| **Technical debt** | `/opt/app-certisigma/docs/api/TECHNICAL-DEBT.md` | Planned API features Census will need (§10 Bulk, §29 Request IDs, §30 Forensic Sharing, §31 RBAC, §32 Structured Tagging, §33 Derived Lists) |
| **Census commercial doc (v1)** | `/opt/app-certisigma/docs/commercial/CERTISIGMA-CENSUS.md` | Original commercial text (corrected, factually accurate) |
| **Census commercial doc (v2)** | `/opt/app-certisigma/docs/commercial/CERTISIGMA-CENSUS-v2.md` | Expanded vision: derived lists, structured tagging, AI governance, NIS2/DORA/GDPR mapping |
| **Enterprise overview** | `/opt/app-certisigma/docs/commercial/CERTISIGMA-ENTERPRISE-OVERVIEW.md` | Architecture overview, Claims Layer, encryption model |
| **Test harness** | `/opt/app-certisigma/scripts/test-sdk.sh` | Reference for how SDK tests are structured |

**CRITICAL: Never modify files in `/opt/app-certisigma/`.** That is a separate project with its own git repo, deployment pipeline, and versioning. Census only reads from it for reference.

## SDK API Surface (what Census can use)

```python
from certisigma import CertiSigmaClient, hash_file, hash_bytes
from certisigma import CertiSigmaError, AuthenticationError, RateLimitError, QuotaExceededError

# Client (sync)
client = CertiSigmaClient(api_key="cs_...", base_url="https://api.certisigma.ch")
client = CertiSigmaClient()  # no key = public endpoints only

# Core operations
result = client.attest(hash_hex, source="label", extra_data={...})     # → AttestResult
result = client.verify(hash_hex)                                        # → VerifyResult
result = client.batch_attest(hashes_list, source="label")               # → BatchAttestResult
result = client.batch_verify(hashes_list, detailed=False)               # → BatchVerifyResult
result = client.get_evidence(attestation_id)                            # → EvidenceResult
result = client.attest_file(path, source="label")                       # → AttestResult

# Hashing (standalone, no API call)
h = hash_file("/path/to/file")    # streamed, constant memory
h = hash_bytes(b"raw content")

# Types (dataclasses)
# AttestResult: id, hash_hex, timestamp, status, signature, claim_id, source, extra_data, etag
# VerifyResult: exists, hash_hex, id, timestamp, source, level
# BatchAttestResult: batch_id, count, created, existing, attestations (list of dicts)
# BatchVerifyResult: count, found, not_found, results (list of dicts)
# EvidenceResult: hash_hex, level, t0, t1, t2, id

# Exceptions
# CertiSigmaError (base), AuthenticationError (401), RateLimitError (429), QuotaExceededError (403)
```

## Architecture Rules

1. **Census installs `certisigma` from PyPI.** Never import from the backend source. Never copy SDK files. The published package is the contract.
2. **Content never leaves the client.** Only SHA-256 hashes are sent to the API.
3. **The manifest is local.** It maps hash → filepath and lives on the client filesystem. The server never sees file paths.
4. **No Docker.** Census is a CLI tool installed via `pip install certisigma-census`. No containers, no deploy.sh, no infrastructure.
5. **Batch limit: 100 hashes per API call.** Chunk accordingly.
6. **Rate limit: 1000 requests/minute** per API key (sliding window). Handle `RateLimitError` with backoff.

## Git Workflow

```bash
# Work on main branch
git add -A
git commit -m "descriptive message"   # Hook auto-strips "Made-with: Cursor"

# NEVER include in commit messages:
# - References to Cursor, AI, or assistants
# - "Made with Cursor" or similar

# Publishing to PyPI (when ready):
git tag v0.1.0
git push origin main --tags
# GitHub Actions handles build + publish
```

## Current State (what works, what doesn't)

### ✅ Implemented (Fase 1 — CLI Core)
- `census scan <dir>` — walks directory, hashes files, attests in batch, saves manifest
- `census compare <suspect_dir>` — hashes suspects, verifies against registry, reports matches
- `census status <manifest>` — shows inventory summary
- Manifest CRUD with save/load round-trip
- Source labels, dry-run mode, JSON report export

### ✅ Implemented (Fase 2 — Production hardening, v0.2.1)
- Retry with exponential backoff on 429/5xx (uses `RateLimitError.retry_after`)
- Progress bar for scan, attest, and compare operations (`click.progressbar`)
- File filters: `--include`, `--exclude` globs; `--min-size`, `--max-size`
- Resume interrupted scans (`--resume`) with incremental manifest saves
- CSV export: `compare --output report.csv` + `census export manifest.db`
- Structured JSON logging (`--log-format json`) for SIEM integration
- deploy.sh build/test/publish automation script

### ✅ Implemented (Fase 3 — Watch mode + SQLite, v0.3.0)
- SQLite manifest backend (WAL mode, indexed lookups, schema v2)
- Auto-migration of legacy JSON manifests to SQLite
- `census watch <dir>` — continuous filesystem monitoring via `watchdog`
- Producer-consumer architecture: Observer thread → pending dict → main thread
- Per-event-type handling (CREATE, MODIFY, MOVE, DELETE)
- Debounce, batch interval, baseline scan, delete policies
- inotify limit detection with tuning instructions
- PollingObserver fallback for NFS/CIFS mounts
- Graceful shutdown on SIGINT/SIGTERM with final manifest save
- Symlink escape prevention
- Full test suite (106 tests + 17 skipped future markers)

### ✅ Implemented (Phase 3.5 — Forensic Evidence Pipeline, v0.4.0)
- `census verify <hash|--file>` — hash verification with full T0/T1/T2 evidence chain
- `census integrity manifest.db` — tamper detection against manifest baseline
- `census report manifest.db -o report.html|.pdf|.zip` — forensic report generation
- HTML reports (zero deps), PDF reports (fpdf2), evidence bundles (ZIP)
- Evidence bundles: report + OTS proofs + evidence.json + SHA256SUMS
- XSS prevention in HTML, path traversal prevention in ZIP

### ✅ Implemented (Phase 4 — Forensic Diff and CLI Maturity, v0.5.0)
- `census diff base.db target.db` — manifest comparison with AIDE bitmask exit codes
- `census hash <file|dir>` — standalone SHA-256, `--verify` mode
- `census track <att_id>` — attestation status tracking, `--poll` for T2 wait
- `census config show|init|paths` — TOML config with user/project precedence
- `census completion bash|zsh|fish` — shell completions via Click native
- Config integration in `_get_client()` for API key/base URL

### ✅ Implemented (Phase 5 — Enterprise Readiness, v0.6.0)
- `census doctor` — self-diagnostic (API health, config, inotify, deps, manifest integrity)
- `census merge m1.db m2.db -o out.db` — merge manifests for distributed workflows
- `census scan --json` — machine-readable scan output for CI/CD
- `census compare --json` — machine-readable compare output
- `census status --json` — machine-readable manifest status
- Full test suite (252 tests + 17 skipped future markers)

### 🔮 Future (Fase 6+)
- Bulk API support when backend implements §10 endpoints
- Structured tagging (`--tag key=value`) when backend implements §32
- Derived lists for selective cooperation with third parties (§33)
- AI governance: allow/deny lists via tags + derived lists (§32+§33)
- GUI (Tauri/Electron), Nextcloud integration

## Testing

```bash
# Run tests locally
cd /opt/app-certisigma-census
pip install -e ".[dev]"
pytest --tb=short -q

# Tests must NOT call the real API. Mock the SDK client for unit tests.
# Integration tests (real API) should be in a separate test file, clearly marked.
```

## API Environment

- **Production API:** `https://api.certisigma.ch`
- **API key env var:** `CERTISIGMA_API_KEY`
- **Docker exec for API testing (if needed):** `docker exec app-certisigma-notary-api-1 curl -s http://localhost:8080/...`

## Security & Quality Checklist

These patterns have caused bugs in past audits. Check them when writing or reviewing new code. Patterns marked with 🧪 have automated test coverage — the rest require human review.

### HTML Output

- **Every dynamic value** in generated HTML must pass through `html.escape()`. No exceptions. 🧪 `test_xss_prevention_in_paths`, `test_xss_prevention`
- **URLs in `href=`** must use `_safe_href()` (whitelist `https://` only) to block `javascript:` / `data:` schemes from API responses.
- **ZIP filenames** from external data (e.g. attestation IDs) must sanitize `/`, `\`, `..` to prevent path traversal.

### Input Validation

- **SHA-256 hashes** from user input must be validated with `re.fullmatch(r"[0-9a-f]{64}", value)` before use. 🧪 `test_verify_rejects_invalid_hash`, `test_hash_verify_invalid_hash`
- **`--verify` flag** must reject directories (only single files). 🧪 `test_hash_verify_on_directory_rejected`

### Resource Management

- **`Manifest.load()` must always use `with` or `try/finally`** to close the SQLite connection. `Manifest` implements `__enter__`/`__exit__`.
- **File handles** opened conditionally (e.g. `open()` vs `sys.stdout`) must use `try/finally`.

### `--json` Output Purity

- Commands with `--json` must **never** write non-JSON text to stdout. Gate all human-readable messages with `if not json_output`. 🧪 `test_integrity_json_with_output_no_stdout_contamination`, `test_diff_json_with_output_no_stdout_contamination`
- Progress bars must be disabled when `--json` is active.
- Error messages in JSON mode should be JSON objects (`{"error": "..."}`) on stdout, not plain text.

### Type Safety

- **`Optional[int]` fields** must never be used with format specifiers (`:,`, `:d`) without a null guard: `size = val if val is not None else 0`.
- **`Optional[str]` fields** displayed to the user must use `val or "unknown"`, never bare `{val}` (renders as `None`).

### Config & Secrets

- **`_DEFAULT_CONFIG`** is deep-copied in `load_config()` — never use `dict()` shallow copy on nested defaults. 🧪 `test_returned_config_does_not_share_references_with_defaults`
- **`_SENSITIVE_KEYS`** defines which config keys are masked in `census config show`. Add new secrets here.
- **API keys**: CLI flag > env var > project config > user config > default. Never log or print unmasked.

### Imports

- No dead imports — check after refactoring (CI does not currently lint for this).
- No aliased re-imports (`import csv as csv_mod`) when the module is already top-level.

### Watcher-Specific

- `path.relative_to(root)` can raise `ValueError` on TOCTOU races (file moves outside root between event capture and processing). Always wrap in `try/except ValueError`.
- The `_accept()` filter must resolve symlinks and reject paths outside `self.root` to prevent symlink escape.

## What NOT To Do

- Do NOT modify any file in `/opt/app-certisigma/`
- Do NOT add Docker infrastructure to Census
- Do NOT import from `backend.app.*` or any internal module
- Do NOT hardcode API keys
- Do NOT store sensitive data in the manifest (it's a local SQLite file, not encrypted)
- Do NOT add dependencies without checking they're actively maintained and necessary
- Do NOT skip tests
