# Scanning

Census scans directories, computes SHA-256 hashes for each file, and builds a local manifest. File content **never leaves the client** — only hashes are sent to the CertiSigma API.

## Basic usage

```bash
# Scan and attest all files
census scan /path/to/files --source inventory-hr

# Hash only, no API calls
census scan /path/to/files --dry-run
```

## File filters

Filters control which files are included in the scan. All filters apply to both `scan` and `compare` commands.

### Include / exclude globs

```bash
# Only scan PDF and DOCX files
census scan /data --include "*.pdf" --include "*.docx"

# Exclude logs and temporary files
census scan /data --exclude "*.log" --exclude "*.tmp" --exclude "*.bak"

# Combine: scan only PDFs, but skip drafts
census scan /data --include "*.pdf" --exclude "draft-*"
```

- `--include` patterns are matched against the **filename only** (not the full path).
- If any `--include` is specified, a file must match **at least one** pattern to be included.
- `--exclude` patterns are evaluated **after** include: a file matching any exclude pattern is skipped.
- Glob syntax uses Python's `fnmatch`: `*` matches anything, `?` matches one character, `[seq]` matches characters in seq.

### Size filters

```bash
# Skip files smaller than 1 KB
census scan /data --min-size 1K

# Skip files larger than 100 MB
census scan /data --max-size 100M

# Combine: only files between 1 KB and 500 MB
census scan /data --min-size 1K --max-size 500M
```

Accepted size units: `B`, `K`/`KB`, `M`/`MB`, `G`/`GB`, `T`/`TB`. Fractional values are supported (e.g. `1.5G`). Without a unit, the value is in bytes.

The default maximum file size is **5 GB**. Files exceeding this are skipped with a warning.

### Default skip rules

Census automatically skips:
- **Hidden files and directories** (names starting with `.`) — disable with source code modification
- **Known non-content directories**: `.git`, `.svn`, `.hg`, `__pycache__`, `node_modules`, `.venv`, `venv`, `.env`, `.DS_Store`, `Thumbs.db`

## Resume interrupted scans

Large scans can be interrupted (Ctrl+C, crash, network failure). The `--resume` flag avoids re-hashing files that haven't changed.

```bash
# First scan (may be interrupted)
census scan /data/500k-files --source quarterly --manifest inventory.db

# Resume — skips unchanged files, hashes only new/modified ones
census scan /data/500k-files --source quarterly --manifest inventory.db --resume
```

### How resume works

1. The existing manifest is loaded from the `--manifest` path.
2. For each file on disk, Census checks if it's already in the manifest with the **same size and mtime**.
3. Unchanged files reuse the cached hash (no disk I/O for hashing).
4. New or modified files are hashed normally.
5. The result is merged into the existing manifest, preserving attestation state.

### Incremental saves

During attestation, the manifest is saved **after every batch** (100 hashes). If the process is interrupted mid-attestation, progress since the last batch is preserved.

## Parallel hashing (`--workers`)

For large directories, Census can hash files in parallel using multiple CPU cores:

```bash
# Use 4 worker processes for hashing
census scan /data/500k-files --workers 4

# Auto-capped at MAX_WORKERS=8 for resource safety
census scan /data --workers 16  # → silently capped to 8
```

- Workers are `ProcessPoolExecutor` child processes — true parallelism, no GIL.
- File metadata (`stat()`) is captured **before** hashing to prevent TOCTOU race conditions.
- Default: `--workers 1` (sequential). Use `--workers N` for directories with many files.
- Workers cap: `MAX_WORKERS = 8` (enforced by both CLI and library).

## Progress bar

For non-verbose output, Census displays a progress bar during:
- The hashing phase (file count)
- The attestation phase (batch count)

Disable with `-v` (verbose mode replaces the progress bar with debug-level log lines).

## Manifest output

The scan produces a `.census-manifest.db` file (default location: inside the scanned directory). Override with `--manifest /path/to/output.db`.

See [manifest.md](manifest.md) for the manifest format specification.

## JSON output

```bash
census scan /data --json
```

Machine-readable JSON output on stdout, suitable for CI/CD pipelines and scripting.
