# Configuration System Testing Strategy

## Overview
This document outlines the comprehensive testing strategy for the unified configuration system to ensure it works correctly and prevent regressions.

## Testing Approaches

### 1. Unit Tests (test_unified_config_system.py)
Created comprehensive unit tests covering:
- ✅ Precedence order verification (CLI > env > file > defaults)
- ✅ All initialization contexts (CLI, MCP, direct)
- ✅ Deep merging behavior
- ✅ Error handling and validation
- ✅ Project detection and local config
- ✅ Backward compatibility

### 2. Integration Tests
Need to create tests that verify:
- Config works correctly with actual CLI commands
- MCP servers load config properly in real scenarios
- File watchers and periodic indexers use consistent config
- Database initialization respects config settings

### 3. Property-Based Testing
Use hypothesis to test:
- Any valid combination of config sources produces valid config
- Merging is associative (order of operations doesn't matter)
- No data loss during merging
- Type safety is maintained

### 4. End-to-End Tests
Verify complete workflows:
- `chunkhound index` with various config combinations
- `chunkhound mcp stdio` with environment setup
- `chunkhound search` with local config files
- Config file migration scenarios

### 5. Performance Tests
Ensure config loading doesn't regress:
- Measure config initialization time
- Test with large config files
- Verify no repeated file I/O
- Check memory usage

### 6. Security Tests
Verify security aspects:
- API keys are properly masked in logs
- Sensitive data isn't exposed in error messages
- Config files with bad permissions are handled
- Path traversal attacks are prevented

## Critical Test Scenarios

### Scenario 1: MCP Server Launch
```bash
# Test that MCP server gets config from environment
export CHUNKHOUND_DATABASE__PATH=/tmp/test.db
export CHUNKHOUND_EMBEDDING__PROVIDER=openai
export CHUNKHOUND_EMBEDDING__API_KEY=test-key
chunkhound mcp stdio
```

### Scenario 2: CLI with Local Config
```bash
# Test that CLI respects local .chunkhound.json
cd /project/with/config
cat > .chunkhound.json << EOF
{
  "database": {"provider": "lancedb"},
  "embedding": {"provider": "ollama"}
}
EOF
chunkhound index .
```

### Scenario 3: Override Chain
```bash
# Test complete override chain
export CHUNKHOUND_DEBUG=false
cat > config.json << EOF
{"debug": true}
EOF
chunkhound --config config.json --debug false index .
# Should result in debug=false (CLI wins)
```

### Scenario 4: Nested Config Merging
```bash
# Test deep merging of nested configs
export CHUNKHOUND_DATABASE__PATH=/env/path
cat > .chunkhound.json << EOF
{
  "database": {
    "provider": "lancedb",
    "lancedb_index_type": "IVF_PQ"
  }
}
EOF
chunkhound --db /cli/path index .
# Should have provider=lancedb, path=/cli/path, index_type=IVF_PQ
```

## Regression Prevention

### 1. Smoke Tests
Add to test_smoke.py:
```python
def test_config_import_no_side_effects():
    """Ensure config import has no side effects."""
    import chunkhound.core.config.config
    # Should not raise or modify environment

def test_all_cli_commands_config_compatible():
    """Ensure all CLI commands can initialize config."""
    commands = ["index", "search", "run", "mcp"]
    for cmd in commands:
        # Should be able to create config for each
```

### 2. Contract Tests
Define contracts that must always be true:
- Config() with no args must always work
- Config(args=x) must respect all args fields
- Environment variables must map correctly
- Local config must be found if in project root

### 3. Mutation Testing
Use mutmut to ensure tests catch changes:
```bash
mutmut run --paths-to-mutate chunkhound/core/config/config.py
```

### 4. Coverage Requirements
- Line coverage: 100% for config.py
- Branch coverage: 100% for precedence logic
- Path coverage: All config source combinations

## Continuous Monitoring

### 1. Performance Benchmarks
```python
# Add to benchmarks/
def benchmark_config_initialization(benchmark):
    benchmark(Config)

def benchmark_config_with_all_sources(benchmark, tmp_path):
    # Set up env, files, args
    benchmark(Config, args=args)
```

### 2. Memory Profiling
```python
# Check for memory leaks in config loading
@profile
def test_config_memory_usage():
    for _ in range(1000):
        Config()
        reset_config()
```

### 3. Real-World Testing
- Test with actual user config files
- Test with production environment variables
- Test with complex project structures

## Test Data Management

### 1. Config File Fixtures
Create fixtures for common scenarios:
```python
@pytest.fixture
def minimal_config():
    return {"database": {"path": "/tmp/test.db"}}

@pytest.fixture
def full_config():
    return {
        "database": {...},
        "embedding": {...},
        "mcp": {...},
        "indexing": {...}
    }

@pytest.fixture
def invalid_configs():
    return [
        "not json",
        {"database": "not an object"},
        {"invalid_key": "value"}
    ]
```

### 2. Environment Fixtures
```python
@pytest.fixture
def clean_env(monkeypatch):
    """Remove all CHUNKHOUND_ env vars."""
    for key in list(os.environ.keys()):
        if key.startswith("CHUNKHOUND_"):
            monkeypatch.delenv(key)

@pytest.fixture
def full_env(monkeypatch):
    """Set all possible env vars."""
    vars = {
        "CHUNKHOUND_DEBUG": "true",
        "CHUNKHOUND_DATABASE__PATH": "/env/db",
        # ... all variables
    }
    for k, v in vars.items():
        monkeypatch.setenv(k, v)
```

## Error Injection Testing

Test resilience to:
1. Corrupted config files
2. Missing directories
3. Permission errors
4. Network timeouts (for embeddings)
5. Invalid type conversions
6. Circular config references

## Documentation Tests

Ensure examples in documentation work:
```python
def test_readme_examples():
    """Test that README config examples work."""
    # Extract code blocks from README
    # Execute and verify they work

def test_docstring_examples():
    """Test that docstring examples work."""
    import doctest
    doctest.testmod(chunkhound.core.config.config)
```

## Migration Testing

Test upgrade paths:
1. Old env vars → new env vars
2. Old config format → new format
3. Missing config → defaults
4. v2.7 configs → v2.8 configs

## Observability

Add logging to track:
- Which config sources were used
- What values were overridden
- Performance metrics
- Validation failures

```python
# Add debug logging
logger.debug(f"Config sources used: {sources}")
logger.debug(f"Final precedence: {precedence_chain}")
logger.debug(f"Load time: {load_time}ms")
```

## Success Criteria

The configuration system is considered properly tested when:
1. ✅ 100% code coverage on config modules
2. ✅ All precedence combinations tested
3. ✅ All error paths tested
4. ✅ Performance benchmarks established
5. ✅ Real-world scenarios validated
6. ✅ No regressions in 10 releases
7. ✅ Migration from old patterns complete
8. ✅ Documentation examples verified
9. ✅ Security review passed
10. ✅ Mutation testing score > 90%