# Formalizer — Gap Analysis & Remediation Plan

Identifies gaps between the `PROJECT.md` design vision and the current implementation, and documents how each will be closed.

## Gap 1: `lib.typ` not bundled for PyPI distribution

**Problem.** `codegen.py` resolves the engine via a relative filesystem path three levels above the installed package:

```python
_ENGINE_LIB = Path(__file__).resolve().parent.parent.parent / "formalizer-engine" / "lib.typ"
```

This works during development but breaks when installed from PyPI — `formalizer-engine/` is outside the wheel and not declared as package data.

**Plan.** Keep `lib.typ` in its canonical home (`formalizer-engine/lib.typ`) and expose it to the Python package via a resource interface:

1. Add a `resources/` directory inside `src/formalizer/` and symlink or copy `lib.typ` there at build time. Alternatively, declare `formalizer-engine/lib.typ` as a shared data file in `pyproject.toml`.
2. In `codegen.py`, load the engine using `importlib.resources` (Python 3.10+):
   ```python
   from importlib.resources import files
   _ENGINE_LIB = files("formalizer") / "resources" / "lib.typ"
   ```
3. Update `pyproject.toml` to include the resource in the wheel:
   ```toml
   [tool.hatch.build.targets.wheel]
   packages = ["src/formalizer"]
   # Ensure non-Python files are included
   [tool.hatch.build.targets.wheel.force-include]
   "formalizer-engine/lib.typ" = "formalizer/resources/lib.typ"
   ```
4. This keeps the engine's source of truth in `formalizer-engine/` (for independent engine development and testing) while making it available inside the installed package.

**Open question.** Hatch's `force-include` copies at build time, so edits to `formalizer-engine/lib.typ` won't reflect in `src/formalizer/resources/` during development without a rebuild. An editable install (`pip install -e .`) should be tested to ensure the resource resolves correctly in both editable and installed modes. If the editable path doesn't resolve cleanly, a fallback to the in-tree path (the current behaviour) can be kept for development only:

```python
def _resolve_engine_lib() -> Path:
    """Locate lib.typ — installed resource first, in-tree fallback for dev."""
    try:
        ref = files("formalizer") / "resources" / "lib.typ"
        with as_file(ref) as p:
            if p.exists():
                return p
    except Exception:
        pass
    # Development fallback
    dev = Path(__file__).resolve().parent.parent.parent / "formalizer-engine" / "lib.typ"
    if dev.exists():
        return dev
    raise FileNotFoundError("Engine lib.typ not found")
```

---

## Gap 2: Dead code in `lib.typ`

**Problem.** `build-radio-groups()` is called in `render-form` and assigned to `radio-groups`, but the variable is never referenced. The actual radio selection logic uses `radio-counters` and `radio-button-selected` directly.

**Plan.** Remove the `build-radio-groups` function and the `let radio-groups = ...` assignment from `render-form`. The `radio-counters` / `radio-button-selected` approach is correct and sufficient.

---

## Gap 3: Example schema JSON files use stale format

**Problem.** Two schema JSON files were generated by the ad-hoc `designs/reference/example/extraction/extract.py` script and contain fields not in `FIELDS_SCHEMA.md`:

| File | Issues |
|---|---|
| `formalizer-engine/example/FIELDS.json` | `"type": "Text"` (capitalised), extra keys: `label`, `value`, `flags`, `max_len` |
| `designs/reference/example/all_fields_sample.pdf.json` | Same extra keys, raw PyMuPDF types |
| `designs/reference/example/af4141-fillable.pdf.json` | Same extra keys, raw PyMuPDF types |

The production `extract.py` correctly normalises types to lowercase and omits the extra fields. The engine handles capitalised types via `lower()`, so this isn't a runtime bug — but the example data is misleading as a reference.

**Plan.** Regenerate all three JSON files by running the canonical `extract.py` against each source PDF:

```sh
# from repo root
python -c "
from formalizer.extract import extract
from pathlib import Path
import tempfile, shutil, json

for pdf, out_json in [
    ('designs/reference/example/all_fields_sample.pdf', 'designs/reference/example/all_fields_sample.pdf.json'),
    ('designs/reference/example/af4141-fillable.pdf',   'designs/reference/example/af4141-fillable.pdf.json'),
]:
    with tempfile.TemporaryDirectory() as tmp:
        schema = extract(pdf, Path(tmp))
        Path(out_json).write_text(json.dumps(schema, indent=2))
"
```

For `formalizer-engine/example/FIELDS.json`, regenerate from the af4141 PDF (which is its source) and verify that the engine tests still pass.

---

## Gap 4: Sample PDF files should be checked in

**Problem.** The `tests/conftest.py` fixtures skip all integration tests if the reference PDFs are absent. If these aren't tracked in git, CI and fresh checkouts get vacuously passing test suites.

**Status.** Both `all_fields_sample.pdf` and `af4141-fillable.pdf` are already tracked in git under `designs/reference/example/`. The `.gitignore` does not exclude them. This gap is **already closed** — no action needed.

---

## Gap 5: Generated `typst.toml` should use `"formalizer"` as author

**Problem.** The codegen produces `authors = []` in generated packages. The reference template uses `authors = ["nibsbin"]`. Neither is ideal for end-user packages.

**Plan.** Change the `typst.toml` template in `codegen.py` to:

```toml
[package]
name = "<name>"
version = "0.1.0"
entrypoint = "lib.typ"
authors = ["formalizer"]
license = "Apache-2.0"
description = "Fillable form package generated by formalizer"
```

This makes the author consistently `"formalizer"` (the tool that produced the package) and gives a meaningful default description. Users can edit the generated `typst.toml` if they want different attribution.

Also update the in-tree `formalizer-engine/typst.toml` to match:

```toml
authors = ["formalizer"]
description = "Renders pixel-perfect PDF form replicas from a PyMuPDF-extracted schema."
```

---

## Gap 6: `_typst_ident` can produce identifier collisions

**Problem.** Field names like `"first-name"` and `"first.name"` both sanitise to `first_name` via `_typst_ident`. If a PDF contains field names that collide after sanitisation, `form.typ` will emit duplicate parameter names and fail to compile.

**Plan.** Add collision detection with numeric suffix deduplication in `codegen.py`:

```python
def _unique_ident(name: str, seen_idents: set[str]) -> str:
    """Return a unique Typst identifier, appending _2, _3, etc. on collision."""
    base = _typst_ident(name)
    ident = base
    n = 2
    while ident in seen_idents:
        ident = f"{base}_{n}"
        n += 1
    seen_idents.add(ident)
    return ident
```

This is applied during `codegen()` when building the `params` and `values_entries` lists:

```python
seen_idents: set[str] = set()
for field_name, f in seen.items():
    ident = _unique_ident(field_name, seen_idents)
    ...
```

A comment is emitted in `form.typ` when a rename occurs (e.g., `// renamed from "first.name"`) so the user can identify the mapping.

---

## Gap 7: Radio fields — mark as experimental

**Problem.** Radio button support has several rough edges: dummy values are always omitted from `example.typ`, `export_value` extraction via `button_states()` is an approximation, and the selection logic has untested corner cases.

**Plan.** Mark radio support as experimental rather than attempting a full fix now:

1. Add an `// experimental` suffix to the radio field comment in generated `form.typ`:
   ```typst
   choice: none,  // radio (experimental)
   ```
2. Keep the existing rendering logic (it works for common cases).
3. Defer improvements to radio dummy values, export-value extraction, and edge-case handling to a future iteration.

---

## Gap 8: Improve dummy value generation for `example.typ`

**Problem.** The current `_typst_dummy` for text fields returns the raw PDF field name (e.g., `"commonforms_text_p1_1"`), which produces ugly and uninformative previews. A smarter generator would produce content that looks realistic in the compiled PDF.

**Plan.** Implement a `_typst_dummy` replacement that generates contextually appropriate values:

| Field type | Strategy | Example |
|---|---|---|
| `text` | Humanise the field name: strip common prefixes, replace underscores/camelCase with spaces, title-case the result. Fall back to `"Sample text"` if the name is purely machine-generated (e.g., all digits after stripping). | `"first_name"` → `"First Name"`, `"commonforms_text_p1_1"` → `"Sample Text"` |
| `checkbox` | Keep `true` (shows the checkmark in preview). | `true` |
| `radio` | `none` — experimental, omit from example. | *(omitted)* |
| `combobox` | First option's export value (current behaviour is correct). | `"M"` |
| `listbox` | First option's export value (current behaviour is correct). | `"aaa"` |

The humanisation logic:

```python
def _humanise_field_name(name: str) -> str:
    """Convert a PDF field name into a human-readable dummy value."""
    # Strip common machine prefixes (e.g., "commonforms_text_p1_")
    name = re.sub(r"^[a-z]+forms?_", "", name, flags=re.IGNORECASE)
    name = re.sub(r"_p\d+_\d+$", "", name)       # trailing page/index
    name = re.sub(r"\d+$", "", name)               # trailing digits
    # camelCase → spaces
    name = re.sub(r"([a-z])([A-Z])", r"\1 \2", name)
    # underscores/hyphens → spaces
    name = re.sub(r"[_\-]+", " ", name)
    name = name.strip()
    if not name or name.isspace():
        return "Sample Text"
    return name.title()
```

This produces reasonable previews without requiring semantic knowledge of the form.

---

## Open Question: Accessibility via field labels

**Context.** The ad-hoc extraction script captures `widget.field_label` (the PDF field's tooltip/description). The canonical `extract.py` does not include this in `FIELDS.json`, and `FIELDS_SCHEMA.md` does not define a `label` field.

**Decision.** Labels are **not** added to the schema at this time. However, they could improve accessibility in a future iteration:

- **Typst accessibility.** Typst does not currently support PDF accessibility tags (`/Alt`, `/ActualText`, etc.), so there's no immediate rendering use for labels.
- **Dummy value improvement.** Field labels could serve as higher-quality dummy values for `example.typ` (e.g., a field named `"tf_1"` with label `"First Name"` would produce a much better preview). This could be added as an optional `label` field in a future schema revision.
- **Future-proofing.** If/when Typst gains accessibility support, the labels would be ready for use. At that point, `FIELDS_SCHEMA.md` would be extended with an optional `label` field and the engine could emit accessible annotations.

**Recommendation.** Revisit after Typst ships accessibility features. In the meantime, the humanised field-name approach (Gap 8) provides a good-enough approximation.
