<!-- STATUS: COMPLETE -->

# Phase 9: Analysis Filtering

## Goal

Implement aggressive file filtering at the discovery phase to reduce token bloat and prevent 429 errors from non-code files. The pipeline currently processes all tracked files (135 in test repo, 81 non-code), producing 98 singleton modules and wasting API calls.

**Target:** Filter data/docs/config files at discovery, respect .gitignore patterns, support per-repo configuration, and track excluded file count for reporting.

## Dependencies

- **Phase 8.5** — Test fixes complete, pipeline stable
- **Phase 6** — LLM context builder (will benefit from smaller inputs)

---

## Research Notes

### pathspec Library

The `pathspec` library provides gitignore-compatible pattern matching:

```python
from pathspec import PathSpec
from pathspec.gitignore import GitIgnoreSpec

# Load from .gitignore file
with open('.gitignore') as f:
    spec = GitIgnoreSpec.from_lines(f)

# Check if file matches
if spec.match_file('path/to/file.py'):
    # File is ignored
```

- `GitIgnoreSpec` properly handles negation patterns (`!pattern`)
- `match_file()` returns True if the file should be ignored
- Works with relative paths from repo root

### Filter Application Order

Per phase_shift_requirements, the filter order is:

1. `ALWAYS_EXCLUDE` — not overridable (`.git/`, `__pycache__/`, etc.)
2. `.gitignore` — files ignored by VCS are excluded
3. `DEFAULT_EXCLUDE` — default patterns (overridable)
4. `include_patterns` (from per-repo config) — add back files
5. `exclude_patterns` (from per-repo config) — additional exclusions

### Per-Repo Config Location

```
~/.unfamiliar/repos/<repo-hash>/config.yaml
```

Where `<repo-hash>` is `sha256(repo_path)[:8]` (consistent with existing campaign hash scheme).

---

## Tasks

### Task 1: Add pathspec dependency — DONE

**File:** `pyproject.toml`

**Action:** Add `pathspec>=0.12` to dependencies list.

**Verify:** `pip show pathspec` confirms installation.

**Done when:** Dependency added, pyproject.toml syntax valid.

---

### Task 2: Create filter package structure — DONE

**Files:**
- `src/unfamiliar/analysis/filters/__init__.py`
- `src/unfamiliar/analysis/filters/patterns.py`

**Action:**
1. Create `filters/` directory under `analysis/`
2. In `patterns.py`, define the three pattern lists:
   - `ALWAYS_EXCLUDE` — hard exclusions (`.git/`, `__pycache__/`, `node_modules/`, etc.)
   - `DEFAULT_EXCLUDE` — default data/docs/config patterns
   - `ALWAYS_INCLUDE` — dependency manifests that override DEFAULT_EXCLUDE
3. Create `FileFilter` class with:
   - Constructor accepting optional `include_patterns` and `exclude_patterns`
   - `should_exclude(path: str) -> bool` method implementing filter order
   - `load_gitignore(repo_path: Path) -> None` to load .gitignore patterns

**Verify:** Module imports without error, `FileFilter` class instantiable.

**Done when:** Package structure created, pattern constants defined, `FileFilter` class skeleton complete.

---

### Task 3: Implement gitignore loading — DONE

**File:** `src/unfamiliar/analysis/filters/gitignore.py`

**Action:**
1. Create `load_gitignore_spec(repo_path: Path) -> GitIgnoreSpec | None`
2. Read `.gitignore` file from repo root
3. Parse with `GitIgnoreSpec.from_lines()`
4. Return None if no .gitignore exists
5. Handle encoding errors gracefully

**Verify:** Function loads a sample .gitignore and correctly identifies ignored files.

**Done when:** Function returns working GitIgnoreSpec, handles missing file.

---

### Task 4: Implement FileFilter.should_exclude() — DONE

**File:** `src/unfamiliar/analysis/filters/patterns.py`

**Action:**
1. Compile ALWAYS_EXCLUDE into a PathSpec at construction
2. Compile DEFAULT_EXCLUDE into a PathSpec
3. Compile user-provided include/exclude patterns into PathSpecs
4. Implement filter order in `should_exclude()`:
   - If ALWAYS_EXCLUDE matches → exclude
   - If gitignore spec matches → exclude
   - If DEFAULT_EXCLUDE matches AND not in ALWAYS_INCLUDE → exclude
   - If include_patterns matches → include (override)
   - If exclude_patterns matches → exclude

**Verify:** Unit test with sample paths confirms correct exclusion behavior.

**Done when:** Method correctly applies all filter layers in order.

---

### Task 5: Add per-repo config loading — DONE

**File:** `src/unfamiliar/config.py`

**Action:**
1. Add `load_repo_filter_config(repo_path: Path, base_dir: Path | None = None) -> RepoFilterConfig`
2. Create `RepoFilterConfig` Pydantic model with:
   - `include_patterns: list[str] = []`
   - `exclude_patterns: list[str] = []`
3. Config file path: `~/.unfamiliar/repos/<hash>/config.yaml`
4. Return empty config if file doesn't exist
5. Use `yaml` safe_load (add `pyyaml` dependency if needed, or use tomllib)

**Verify:** Function loads existing config, returns defaults for missing file.

**Done when:** Config loading works, Pydantic model validates correctly.

---

### Task 6: Add excluded_count to AnalysisResult — DONE

**File:** `src/unfamiliar/analysis/models.py`

**Action:**
1. Add `excluded_count: int = 0` to `AnalysisResult` dataclass
2. Document in docstring: count of files excluded by filter (for reporting)

**Verify:** Model instantiates with new field, default is 0.

**Done when:** Field added, no breaking changes to existing code.

---

### Task 7: Wire filter into discovery — DONE

**File:** `src/unfamiliar/analysis/discovery.py`

**Action:**
1. Add `FileFilter` import
2. Modify `discover_files()` signature:
   - Add `file_filter: FileFilter | None = None` parameter
3. After getting tracked files, apply filter before processing:
   - Track excluded count
   - Return `(file_infos, excluded_count)` tuple OR add excluded_count to a wrapper
4. Alternative: Keep return signature, filter in orchestrator instead

**Design decision:** Filter in discovery (simpler) vs orchestrator (cleaner separation). Recommend filtering in discovery for single-pass efficiency.

**Verify:** Filtered files list is smaller, excluded_count matches difference.

**Done when:** discover_files returns filtered list and count.

---

### Task 8: Wire filter into orchestrator — DONE

**File:** `src/unfamiliar/analysis/orchestrator.py`

**Action:**
1. Import filter components
2. Load per-repo filter config
3. Create FileFilter with gitignore + user patterns
4. Pass filter to discover_files()
5. Store excluded_count in AnalysisResult
6. Log filtered count for observability

**Verify:** Pipeline runs with filtering, AnalysisResult.excluded_count populated.

**Done when:** End-to-end filtering works, logs show excluded count.

---

### Task 9: Update analysis package exports — DONE

**File:** `src/unfamiliar/analysis/__init__.py`

**Action:** Export FileFilter and filter utilities for potential CLI use.

**Verify:** `from unfamiliar.analysis import FileFilter` works.

**Done when:** Public API updated.

---

## Interface Contracts

### FileFilter

```python
class FileFilter:
    """Filter files based on exclude/include patterns and gitignore."""

    def __init__(
        self,
        include_patterns: list[str] | None = None,
        exclude_patterns: list[str] | None = None,
    ) -> None:
        """Initialize filter with optional user patterns.

        Args:
            include_patterns: Patterns to add back (override DEFAULT_EXCLUDE).
            exclude_patterns: Additional patterns to exclude.
        """

    def load_gitignore(self, repo_path: Path) -> None:
        """Load .gitignore patterns from repository.

        Args:
            repo_path: Path to repository root.

        Does nothing if .gitignore doesn't exist.
        """

    def should_exclude(self, path: str) -> bool:
        """Check if a file path should be excluded from analysis.

        Args:
            path: Relative path from repo root.

        Returns:
            True if file should be excluded, False to include.
        """
```

**Invariants:**
- ALWAYS_EXCLUDE patterns are never overridden
- ALWAYS_INCLUDE overrides DEFAULT_EXCLUDE but not ALWAYS_EXCLUDE
- gitignore patterns apply after ALWAYS_EXCLUDE

**Valid inputs:**
- `path`: Non-empty string, relative path without leading slash

**Edge cases:**
- Empty path → `True` (exclude)
- Path starting with `/` → `True` (exclude, invalid format)
- .gitignore not found → filter still works with default patterns

---

### RepoFilterConfig

```python
class RepoFilterConfig(BaseModel):
    """Per-repository filter configuration."""

    include_patterns: list[str] = Field(default_factory=list)
    exclude_patterns: list[str] = Field(default_factory=list)
```

**Invariants:**
- Empty lists are valid defaults
- Patterns are gitignore-style glob patterns

**Edge cases:**
- Config file missing → return empty config
- Config file invalid YAML → raise ConfigError
- Empty pattern list → no effect on filtering

---

### discover_files (modified)

```python
def discover_files(
    repo_path: Path,
    registry: AdapterRegistry,
    file_filter: FileFilter | None = None,
) -> tuple[list[FileInfo], int]:
    """Discover source files, optionally filtering non-code files.

    Returns:
        Tuple of (filtered files, excluded count).
    """
```

**Invariants:**
- If `file_filter` is None, no filtering occurs (backward compatible)
- Excluded count is 0 if no filter provided
- Files are still sorted by path

**Edge cases:**
- All files excluded → returns empty list, count = total discovered
- No files excluded → returns all files, count = 0
- Filter is None → behaves exactly as before

---

## Size Check

9 tasks — within the ~10 task limit. No split needed.

---

## Testing Notes

Testing for this phase is batched into **Phase 14** (Tests for phases 9–13).

Key test scenarios:
1. Default patterns filter common non-code files
2. .gitignore patterns are respected
3. ALWAYS_INCLUDE overrides DEFAULT_EXCLUDE (e.g., `package.json` kept)
4. ALWAYS_EXCLUDE is never overridden
5. Per-repo config adds custom include/exclude patterns
6. Filter application order is correct
7. excluded_count is accurate
