# Decisions Log

## 2026-03-02 — Phase structure and test batching
**Context:** Determining how to break a complex project (static analysis + LLM pipeline + TUI + theming) into reviewable phases.
**Decision:** 14 phases with 3 batched test phases (5, 8, 13). Data layer tests included in Phase 5 despite being optional per iterative-build rules.
**Rationale:** The repository layer involves multiple junction tables and complex queries — testing it early catches schema issues before the static analysis pipeline depends on it. Batching tests keeps the total phase count manageable while still enforcing context separation (new sessions for test phases).

## 2026-03-02 — create_many() id retrieval relies on uniqueness assumption (Phase 5 test note)
**Context:** After bulk insert via `executemany()`, SQLite's `lastrowid` only returns the last inserted id. To return all entities with ids, `ModuleRepository.create_many()`, `MissionRepository.create_many()`, and `ChainRepository.create_many()` each re-fetch all entities for the campaign and match by a unique-ish field (`path`, `title_plain`, `name_plain` respectively).
**Risk:** If two entities in the same campaign share the same match field (e.g., two modules with the same path), the id assignment silently returns wrong results. There are no DB UNIQUE constraints on those fields in those tables.
**Phase 5 action:** Test `create_many()` with duplicate match-field values to confirm and document the failure mode. Consider whether a UNIQUE constraint should be added to `module.path` within a campaign (likely yes), or whether the application layer guarantee is sufficient.

## 2026-03-02 — Source layout: src/ directory
**Context:** Python package structure for a pip-installable CLI tool.
**Decision:** Use `src/unfamiliar/` layout with pyproject.toml.
**Rationale:** The src layout prevents accidental imports from the working directory during development. It's the standard for pip-installable packages and what the requirements describe (`pip install unfamiliar`).

## 2026-03-05 — Category hints rely on naming conventions (document for README)
**Context:** The `_infer_category_hints()` function in `clustering.py` detects investigation categories by matching file paths against keyword patterns (e.g., `test_` → TESTING, `auth` → DOMAIN_BUSINESS_LOGIC). This is a heuristic, not a guarantee.
**Assumptions made:**
- Tests follow naming conventions (`test_`, `_test.py`, `tests/`, `spec/`)
- Business logic files have descriptive names (`service`, `handler`, `processor`)
- Security-relevant code mentions auth/token/credential in filenames
- Entry points are named conventionally (`main.py`, `app.py`, `server.py`)
- Configuration files are named conventionally (`config`, `settings`, `constants`)
**What slips through cracks:**
- Tests in unconventional directories (e.g., `src/checks/`) → no TESTING hint
- Generic filenames like `utils.py`, `helpers.py` → no category hints
- False positives: `todo_list.py` triggers TECHNICAL_DEBT because "todo" is in the name
**Action for v1.0 README:**
- Add "Limitations & Assumptions" section documenting that category hints are convention-based
- Note that unconventional project structures will receive weaker or no hints
- This is acceptable for MVP — hints are suggestions to the LLM, not definitive classifications

## 2026-03-05 — Mutmut configuration and Hypothesis compatibility issue
**Context:** Running mutation testing with `mutmut` to identify test coverage gaps. Encountered `RuntimeError: context has already been set` when mutmut runs tests that involve Hypothesis property-based tests.

**Issue:** Mutmut and Hypothesis both use Python's `contextvars` internally. When mutmut runs tests in isolation, this causes a conflict. Tests pass under normal `pytest` but fail under `mutmut run`.

**Workaround for now:**
- Accept that mutation testing on files with Hypothesis tests will fail
- Focus mutation testing on files without property-based tests
- String mutations in `scoring.py` (e.g., `"test file"` → `"XXtest fileXX"`) are acknowledged as low-value noise

**How to ignore specific mutations (when mutmut works):**
```python
# Add pragma comment to prevent mutation of a line
reasons.append("test file")  # pragma: no mutate
```

**How to run mutmut on specific files:**
```bash
# Pass filename directly as positional argument (not a flag)
.venv/bin/python -m mutmut run "src/unfamiliar/analysis/scoring.py"

# Or use wildcard pattern
.venv/bin/python -m mutmut run "scoring*"
```

**Configuration in pyproject.toml:**
```toml
[tool.mutmut]
paths_to_mutate = ["src/unfamiliar/analysis/scoring.py"]
tests_dir = ["tests/"]
```

**Exclude files from mutation in setup.cfg:**
```ini
[mutmut]
do_not_mutate=
    *__tests.py
```

**Action for later:**
- Investigate mutmut + Hypothesis version compatibility
- Consider isolating Hypothesis tests in separate files that mutmut can skip
- Revisit after major test suite changes

---

## 2026-03-06 — Clustering produces too many singletons on sparse import graphs
**Context:** First real-world test of `analyze_repository()` on an external codebase (134 files). The clustering algorithm produced 103 modules, with 97 being singletons.

**Findings:**
- 97 singleton modules (files with no import connections)
- Only 54 import edges across 134 files (0.40 edges per file)
- 1 meaningful cluster of 21 files, plus 2 clusters of 3-6 files
- Label propagation works correctly but can't group files that don't import each other

**Root cause:** The algorithm relies entirely on import graph connectivity. Isolated files (no imports, or only external imports like `import os`) have no neighbors to merge with, so they remain singletons.

**Impact for MVP:**
- Not a bug — the algorithm is correctly detecting sparse structure
- Singletons as "modules" are noise for LLM consumption
- Top scored files were correct: `conftest.py`, `app.py`, `database.py` (entry points and hubs)

**Short-term fix (MVP):** Filter output to only expose clusters with 3+ files to LLM:
```python
real_modules = [m for m in result.modules if m.file_count >= 3]
```

**Future improvements (v1.1+):**
- Add directory-based fallback clustering for isolated files (group `backend/handlers/*.py` even without imports)
- Consider file naming patterns as weak clustering signal
- Expose "cohesion score" (edges per file) in `AnalysisResult` so LLM can contextualize module quality
- Make `min_module_size` configurable per-repo

**Action:** Document in v1.0 README that module detection is import-based and works best on cohesive codebases with internal dependencies.

---

## 2026-03-07 — Language adapter naming conventions
**Context:** The project supports multiple languages via adapter pattern. Each adapter implements `AdapterProtocol` and provides parsed file structure.
**Decision:**
- **File naming:** `{language}_adapter.py` (e.g., `python_adapter.py`, `javascript_adapter.py`, `typescript_adapter.py`)
- **Internal helpers:** `_treesitter_utils.py` for tree-sitter utilities shared by JS/TS adapters
- **Parser selection:**
  - Python → stdlib `ast` module (no external dependency)
  - JavaScript/TypeScript → tree-sitter (handles ES modules, CommonJS, JSX patterns that stdlib `ast` can't parse)
**Rationale:**
- Python's stdlib `ast` is mature and requires no external dependencies
- JS/TS have complex module syntax (ES6, CommonJS, JSX) that benefits from tree-sitter's robust parsing
- Underscore prefix on `_treesitter_utils.py` signals "internal use" — it to import from adapters but not from user code
**Adding new languages:**
Create new `{language}_adapter.py` file. If using tree-sitter, add shared utilities to `_treesitter_utils.py` or create new `_{language}_utils.py` for language-specific helpers.
**Action for README:** Document supported languages and their parser choices. Note that Python uses stdlib while JS/TS require tree-sitter grammars.

---

## 2026-03-04 — Security scanning perception risk (flagged for v1.0 release)
**Context:** The app includes secret detection (`detect-secrets`) and CVE scanning (`pip-audit`/`npm audit`) as part of static analysis. This is a defensive feature to help users find exposed credentials in their own codebases.
**Risk:** For a free open-source tool, "this tool scans for secrets" can sound suspicious. Users may worry about API key harvesting even though all analysis runs locally and nothing is transmitted.
**Action needed before public release:**
- Add clear documentation that all analysis is local-only
- Consider making security scanning opt-in or adding `--skip-security` flag
- Add a note in README explicitly stating no data leaves the user's machine
**Phase to address:** v1.0 polish/release phase (after MVP complete)

---

## 2026-03-07 — _BASE_DIR removed; config functions accept injectable base_dir
**Context:** Phase 5.5 technical debt cleanup. `_BASE_DIR = Path.home() / ".unfamiliar"` was a module-level constant evaluated at import time, making `load_config()` and `get_campaign_dir()` untestable without patching `Path.home()`.
**Decision:** Remove `_BASE_DIR`. Add `base_dir: Path | None = None` parameter to both functions. Default computed at call time: `Path.home() / ".unfamiliar"`.
**Rationale:** Additive change — all existing callers pass no `base_dir` and get identical behavior. Tests can now pass `tmp_path` directly instead of monkeypatching module internals. Deferred the more structural split (Issue #4, extracting `campaign_storage.py`) to a future phase.
**Impact:** `test_config.py` updated to use `base_dir=` param instead of `monkeypatch.setattr("unfamiliar.config._BASE_DIR", ...)`. No CLI changes needed.

---

## 2026-03-07 — Parser caching strategy for tree-sitter adapters
**Context:** Phase 5.5 performance improvement. `Parser(language)` was re-created on every `parse_file()` call in JS and TS adapters.
**Decision:** Cache parsers as instance attributes in `__init__()`. JS adapter caches one parser. TS adapter caches two (one for `.ts`, one for `.tsx` since they use different Language objects).
**Rationale:** Tree-sitter `Parser` is stateless between `parse()` calls and safe to reuse. Instance-level caching (not module-level) keeps parsers tied to adapter lifecycle and avoids global state.
**Impact:** No behavioral change. Reduces object allocation for large repositories.

---

## 2026-03-10 — Python version upgraded from 3.10 to 3.11 (post-Phase 7)
**Context:** After completing Phase 7, the project's minimum Python version was upgraded from 3.10 to 3.11.
**Changes made:**
- `pyproject.toml`: Updated `requires-python`, `target-version` (ruff), `python_version` (mypy)
- `mutants/pyproject.toml`: Same updates, removed conditional `tomli` dependency
- `src/unfamiliar/constants.py`: Removed StrEnum backport, now uses stdlib `from enum import StrEnum`
- `src/unfamiliar/config.py`: Removed tomli fallback, now uses stdlib `import tomllib`
**Rationale:**
- StrEnum is in stdlib as of 3.11 (no need for backport)
- tomllib is in stdlib as of 3.11 (no need for tomli conditional dependency)
- Simplifies codebase, removes version checks
**Documentation still referencing 3.10:** README.md, `_planning/codebase.md`, `_planning/requirements.md`, and phase 1 planning docs. These are historical/documentation only and don't affect runtime.

---

## 2026-03-11 — LLM context budget enforcement and verbosity reduction
**Context:** Testing against a small personal project (134 files) produced a context of ~54,000 tokens, exceeding the 50,000 token budget. The root cause was that `build_briefing_context()` only applied the budget to file contents — other sections (modules, tests, security) were unconstrained.

**Problem:**
- Token budget was only applied to file contents section
- All other sections were added without size limits
- Total context exceeded budget, causing API failures on rate-limited providers (GLM)

**Changes made to `src/unfamiliar/llm/context.py`:**

1. **Global budget enforcement** — Hard cap now applied to entire context, not just file contents:
   ```python
   if len(result) > max_chars and max_chars > 500:
       result = result[:max_chars] + "\n\n... [context truncated to fit token budget]"
   ```

2. **Max file size limit** — Files larger than 10,000 chars (~2,500 tokens) are truncated:
   ```python
   MAX_FILE_CHARS = 10_000
   if len(content) > MAX_FILE_CHARS:
       content = content[:MAX_FILE_CHARS] + "\n... [file truncated]"
   ```

3. **Reduced listing limits:**
   - Module file listings: 10 → 5 files per module
   - Test file listings: 15 → 8 files
   - Security findings: 10 → 5 per category
   - Removed verbose description lines from secrets output

4. **Dynamic budget allocation** — File contents section now gets remaining budget after other sections:
   ```python
   current_chars = sum(len(s) for s in sections)
   remaining_budget = max(0, token_budget - (current_chars // CHARS_PER_TOKEN) - 500)
   ```

**Rationale:**
- LLM providers have varying context limits; respecting budget ensures compatibility
- Smaller contexts reduce API costs and rate limiting
- Key information (repo overview, top files) preserved; verbose listings truncated
- For larger codebases, users can increase `DEFAULT_TOKEN_BUDGET` in constants.py

**Impact on providers:**
- Anthropic (200k context): Works well at default budget
- GLM/Zhipu: May need lower budget or paid tier for large codebases

**Tests updated:** 4 tests in `test_llm_context.py` updated to reflect new limits (5 files instead of 10, 8 test files instead of 15, 5 security findings instead of 10).

---

## 2026-03-17 — Tech debt bulk fix and multi-call pipeline wiring (outside phase process)

**Context:** After Phase 12 review, 6 high-severity tech debt items (CLEANUP-001, QUALITY-001 through QUALITY-005) were fixed in commits `92d2e8a` through `8eb9a09`. These changes were made outside the formal `/plan:phase` → `/plan:build` process.

**Changes made (across 4 commits):**
1. **CLEANUP-001:** `pipeline.py` now imports and calls `generate_category_missions_multi` instead of `generate_category_missions`. Multi-call generator exported from `missions/__init__.py`.
2. **QUALITY-001/005:** All deferred imports moved to module level (`create_structured_chain` in generator.py, `get_retry_delay` in retry.py, `resolve_api_key`/`ChatZhipuAI` in factory.py, `SqliteSaver`/`sqlite3` in pipeline.py).
3. **QUALITY-002:** Mutable default arg closure replaced with `functools.partial(chain.invoke, input_data)`.
4. **QUALITY-003:** Pipeline category loop narrowed from `except Exception` to three specific types. Checkpoint save narrowed to `OSError | sqlite3.Error`.
5. **QUALITY-004:** Duplicate `:param` block in pipeline docstring deleted.
6. **Pipeline enhancement:** `run_mission_pipeline()` now accepts `categories` parameter for processing a subset of categories in canonical order.

**Rationale:** These were mechanical fixes that could be verified by the existing test suite (890 tests, all passing). Formal phase planning was unnecessary overhead for straightforward debt cleanup.

**Impact:** Test count unchanged (890). All fixes are behavioral improvements (narrower exceptions, correct call path) or code hygiene (import organization).

---

## 2026-03-17 — GLM provider config overhaul (unstaged)

**Context:** Research into GLM/Zhipu rate limiting revealed they use concurrency-based limits, not RPM/TPM. User has a paid account, so the free-tier constraints were unnecessarily restrictive.

**Changes (unstaged in factory.py, test_llm_provider.py):**
1. Default model: `glm-4` → `glm-4-flash` (user overrides to `glm-4.7-Flash` via env var)
2. Retry base delay: 30.0s → 2.0s (sequential calls don't need long backoff)
3. Max token budget cap: 8,000 removed (was free-tier constraint)
4. `_get_default_max_tokens()` now accepts `model_name` — Flash→4095, standard→4096
5. New helper `_max_tokens_for_glm_model()` for model-family detection

**Rationale:** The 30s retry delay made test runs take 7+ minutes per failed category with no improvement in success rate. The 8K token budget cap was a free-tier artifact; user has paid account. Model-family-aware max_tokens prevents sending 4096 to Flash models (which hard-cap at 4095).

**Note:** These changes are unstaged and need to be committed.
