# Phase 9 Testing Guide: Analysis Filtering

## Summary

Phase 9 implements aggressive file filtering at the discovery phase to reduce token bloat. The filtering system:
- Excludes non-code files (docs, data, config) by default
- Respects `.gitignore` patterns
- Supports per-repo configuration for custom include/exclude patterns
- Tracks excluded file count for reporting

## Automated Testing

All 723 tests pass, including:
- `tests/test_discovery.py` — Discovery tests updated to handle tuple return
- `tests/test_orchestrator.py` — Orchestrator tests updated for filtering integration

Run tests:
```bash
pytest tests/ -v
```

## Manual Testing Steps

### 1. Verify Filter Module Imports

```python
from unfamiliar.analysis import FileFilter, analyze_repository
from unfamiliar.analysis.filters import ALWAYS_EXCLUDE, DEFAULT_EXCLUDE, ALWAYS_INCLUDE
from unfamiliar.config import load_repo_filter_config, RepoFilterConfig

# Verify constants exist
print(f"ALWAYS_EXCLUDE: {len(ALWAYS_EXCLUDE)} patterns")
print(f"DEFAULT_EXCLUDE: {len(DEFAULT_EXCLUDE)} patterns")
print(f"ALWAYS_INCLUDE: {len(ALWAYS_INCLUDE)} patterns")
```

**Expected:** All imports succeed, pattern counts displayed.

### 2. Test Basic FileFilter

```python
from unfamiliar.analysis.filters.patterns import FileFilter

# Create filter with defaults
f = FileFilter()

# Test ALWAYS_EXCLUDE patterns (cannot be overridden)
assert f.should_exclude(".git/config") == True
assert f.should_exclude("node_modules/package/index.js") == True
assert f.should_exclude("__pycache__/module.pyc") == True

# Test DEFAULT_EXCLUDE patterns
assert f.should_exclude("README.md") == True
assert f.should_exclude("docs/guide.rst") == True
assert f.should_exclude("config.json") == True

# Test ALWAYS_INCLUDE overrides
assert f.should_exclude("package.json") == False  # JS deps
assert f.should_exclude("pyproject.toml") == False  # Python deps
assert f.should_exclude("requirements.txt") == False

# Test source files are NOT excluded
assert f.should_exclude("src/main.py") == False
assert f.should_exclude("app.ts") == False
```

**Expected:** All assertions pass.

### 3. Test gitignore Integration

```bash
# Create temp repo with .gitignore
mkdir -p /tmp/test-filter
cd /tmp/test-filter
git init
echo "*.log" > .gitignore
echo "build/" >> .gitignore
```

```python
from pathlib import Path
from unfamiliar.analysis.filters.patterns import FileFilter

f = FileFilter()
f.load_gitignore(Path("/tmp/test-filter"))

# gitignore patterns should be respected
assert f.should_exclude("debug.log") == True
assert f.should_exclude("build/output.js") == True
assert f.should_exclude("src/main.py") == False
```

**Expected:** gitignore patterns are respected.

### 4. Test Per-Repo Config

Per-repo config is stored at `~/.unfamiliar/repos/<hash>/config.yaml` where `<hash>` is `sha256(repo_path)[:8]`.

```python
from pathlib import Path
from unfamiliar.config import (
    get_repo_config_path,
    load_repo_filter_config,
    RepoFilterConfig,
)
from unfamiliar.analysis.filters.patterns import FileFilter
import yaml

# Get config path for a repo
repo_path = Path("/tmp/test-filter")
config_dir = get_repo_config_path(repo_path)
print(f"Config directory: {config_dir}")

# Create the config directory and file
config_dir.mkdir(parents=True, exist_ok=True)
config_file = config_dir / "config.yaml"

# Write a custom config
config_data = {
    "include_patterns": ["*.md", "docs/"],
    "exclude_patterns": ["legacy/", "*.bak"],
}
with open(config_file, "w") as f:
    yaml.dump(config_data, f)

# Load the config
config = load_repo_filter_config(repo_path)
print(f"include_patterns: {config.include_patterns}")
print(f"exclude_patterns: {config.exclude_patterns}")

# Create filter with loaded patterns
f = FileFilter(
    include_patterns=config.include_patterns,
    exclude_patterns=config.exclude_patterns,
)

# Custom include overrides default
assert f.should_exclude("README.md") == False  # *.md is included

# Custom exclude adds new exclusion
assert f.should_exclude("legacy/old_code.py") == True  # legacy/ excluded
```

**Expected:** Custom patterns loaded from YAML, filter behaves accordingly.

### 5. Test Full Pipeline

```python
from pathlib import Path
from unfamiliar.analysis import analyze_repository

# Run analysis on this repository (use the actual project path)
# In Docker container: Path("/project")
# Local development: Path(__file__).parent.parent.parent.parent
project_path = Path.cwd()  # Assuming you're in the project root

result = analyze_repository(project_path)

print(f"Files discovered: {len(result.files)}")
print(f"Files excluded: {result.excluded_count}")
print(f"Languages: {result.languages}")

# Verify excluded_count is populated
assert result.excluded_count >= 0
assert isinstance(result.excluded_count, int)
```

**Expected:** Analysis runs, excluded_count is populated, non-code files filtered out.

### 6. Test Filter Application Order

The filter order is:
1. ALWAYS_EXCLUDE (cannot be overridden)
2. .gitignore
3. DEFAULT_EXCLUDE (can be overridden)
4. include_patterns
5. exclude_patterns

```python
from unfamiliar.analysis.filters.patterns import FileFilter

f = FileFilter(include_patterns=["*.json"])

# ALWAYS_EXCLUDE cannot be overridden - node_modules always excluded
assert f.should_exclude("node_modules/package.json") == True

# DEFAULT_EXCLUDE can be overridden - this .json is kept
assert f.should_exclude("data/config.json") == False

# Regular code files are kept
assert f.should_exclude("src/main.py") == False
```

**Expected:** Filter order is correct.

## Edge Cases to Check

1. **Empty path:** `f.should_exclude("")` → `True`
2. **Absolute path:** `f.should_exclude("/etc/passwd")` → `True`
3. **Missing .gitignore:** Filter works with defaults
4. **All files excluded:** Returns empty list, count = total
5. **No files excluded:** Returns all files, count = 0

## Security Checklist

- [x] File paths are relative (no absolute path traversal)
- [x] No secrets in config files are logged
- [x] Invalid YAML in per-repo config raises ConfigError (doesn't crash silently)
- [x] Filter patterns don't expose sensitive file contents

## Phase Completion Checklist

- [x] pathspec dependency added to pyproject.toml
- [x] filters/ package created with patterns.py and gitignore.py
- [x] FileFilter class implements 5-layer filter order
- [x] RepoFilterConfig Pydantic model added to config.py
- [x] excluded_count field added to AnalysisResult
- [x] discover_files() returns tuple (files, excluded_count)
- [x] analyze_repository() creates filter and passes to discovery
- [x] FileFilter exported from analysis/__init__.py
- [x] All 723 tests pass
- [x] Linting passes (ruff check)
