# Phase Shift: Analysis Filtering (New Phase 9)

> **Inserted:** 2026-03-13
> **Trigger:** Pipeline inefficiency from excessive non-code files
> **Insertion Point:** Before original Phase 9 (Campaign orchestration)

---

## Trigger

The analysis pipeline processes too many files:

- **Test repo:** 135 files discovered
- **Non-code files:** 81 files (docs, data, config, etc.)
- **Result:** Clustering produces 98 singleton modules
- **Impact:** Token bloat, wasted GLM API calls, 429 rate limiting on paid tier

User approved **Option E** from `test_pipeline_issue.md` — aggressive filtering at discovery phase.

---

## Scope

### Core Functionality

1. **Default exclude patterns** — Aggressive filtering of data/docs/config files
2. **Always-include list** — Hardcoded exceptions for dependency manifests
3. **.gitignore respect** — Files ignored by VCS are ignored by analysis
4. **Per-repo config** — User can add include/exclude patterns via `~/.unfamiliar/repos/<hash>/config.yaml`
5. **Excluded count tracking** — `AnalysisResult.excluded_count: int` for reporting

### Files to Modify

| File | Change |
|------|--------|
| `pyproject.toml` | Add `pathspec` dependency |
| `src/unfamiliar/config.py` | Load filter config from `~/.unfamiliar/repos/<hash>/config.yaml` |
| `src/unfamiliar/analysis/discovery.py` | Apply filter during file discovery |
| `src/unfamiliar/analysis/models.py` | Add `excluded_count: int` to `AnalysisResult` |
| `src/unfamiliar/analysis/orchestrator.py` | Wire filter into pipeline |

### New Files

| File | Purpose |
|------|---------|
| `src/unfamiliar/analysis/filters/__init__.py` | Package exports |
| `src/unfamiliar/analysis/filters/patterns.py` | Default patterns, glob matching, always-include logic |
| `src/unfamiliar/analysis/filters/gitignore.py` | `.gitignore` parsing via `pathspec` |

---

## Configuration Schema

### Location

```
~/.unfamiliar/repos/<repo-hash>/config.yaml
```

Where `<repo-hash>` is `sha256(repo_path)[:8]`.

### Schema

```yaml
# Patterns to ADD BACK to analysis (override default_exclude)
include_patterns:
  - "*.md"
  - "data/**/*.csv"

# Patterns to EXCLUDE in addition to defaults
exclude_patterns:
  - "tests/fixtures/**"
```

### Hardcoded Patterns

```python
# Always exclude (not overridable)
ALWAYS_EXCLUDE = [
    ".git/", ".hg/", ".svn/",
    "__pycache__/", "node_modules/", ".venv/",
    "*.pyc", "*.pyo",
]

# Default exclude (overridable by include_patterns)
DEFAULT_EXCLUDE = [
    "*.csv", "*.json", "*.sql", "*.xml",
    "*.yaml", "*.yml", "*.toml", "*.ini",
    "*.md", "*.txt", "*.rst", "*.lock",
    "data/", "_planning/", "docs/",
]

# Always include (overrides DEFAULT_EXCLUDE, not ALWAYS_EXCLUDE)
ALWAYS_INCLUDE = [
    "package.json", "package-lock.json",
    "requirements.txt", "pyproject.toml", ".gitignore",
]
```

### Filter Application Order

1. `ALWAYS_EXCLUDE` — applied first, not overridable
2. `.gitignore` — files ignored by VCS are excluded
3. `DEFAULT_EXCLUDE` — default patterns
4. `include_patterns` (from config) — add back files
5. `exclude_patterns` (from config) — additional exclusions

---

## Affected Phases

| Old # | New # | Phase Name |
|-------|-------|------------|
| 9 | 10 | Campaign orchestration and CLI commands |
| 10 | 11 | TUI — layout, widgets, and command routing |
| 11 | 12 | TUI — screens and interaction flows |
| 12 | 13 | Film Noir theme and narrative |
| 13 | 14 | Tests for phases 9–13 |
| 14 | 15 | Polish — error handling, accessibility, and edge cases |

---

## Testing

Testing for this phase will be batched into **Phase 14** (Tests for phases 9–13).

---

## Ready for Planning

All requirements resolved. Ready for `/plan:phase 9`.
