# Step 03: Create Sanitizer Tests

## Status Reference
- **Current Status**: Pending
- **Dependencies**: Steps 01-02 should be complete
- **Target Modules**: 
  - `storm_logger/utils/sanitizer.py` (15.15% coverage)
  - `storm_logger/utils/sanitizer_v2.py` (15.64% coverage)
- **Target Coverage**: 100% for both modules

## Objectives
1. Create comprehensive test suites for both sanitizer modules
2. Test all PII detection and sanitization patterns
3. Verify security and compliance features
4. Ensure thread safety and performance

## Coverage Gap Analysis

### sanitizer.py - Missing Coverage
Lines missing coverage: 97-108, 120-184, 188-191, 203-216, 228-234, 245-251, 260-266, 275, 284, 300-302, 318-323

Key areas:
- Pattern compilation and caching
- Sanitization methods for different data types
- Compliance mode implementations
- Custom pattern support
- Whitelist management

### sanitizer_v2.py - Missing Coverage  
Lines missing coverage: 49-57, 62-98, 104, 109, 121-145, 150-173, 177-181, 193-208, 220-226, 236, 245, 249, 253, 257-258, 262, 276-284, 296-324

Key areas:
- Enhanced pattern matching
- Performance optimizations
- Batch sanitization
- Advanced compliance features
- Metrics and reporting

## Test Suite Design

### Test File: `test_sanitizer_comprehensive.py`

#### 1. Basic Sanitization Tests
```python
class TestBasicSanitization:
    - test_email_sanitization
    - test_phone_number_sanitization
    - test_credit_card_sanitization
    - test_ssn_sanitization
    - test_ip_address_sanitization
    - test_mixed_pii_sanitization
```

#### 2. Pattern Management Tests
```python
class TestPatternManagement:
    - test_add_custom_pattern
    - test_remove_pattern
    - test_pattern_priority
    - test_pattern_caching
    - test_invalid_pattern_handling
```

#### 3. Compliance Mode Tests
```python
class TestComplianceModes:
    - test_gdpr_compliance
    - test_hipaa_compliance
    - test_pci_dss_compliance
    - test_custom_compliance
    - test_compliance_switching
```

#### 4. Whitelist Tests
```python
class TestWhitelisting:
    - test_whitelist_addition
    - test_whitelist_removal
    - test_whitelist_patterns
    - test_whitelist_precedence
    - test_whitelist_persistence
```

#### 5. Performance Tests
```python
class TestPerformance:
    - test_large_text_sanitization
    - test_batch_processing
    - test_pattern_compilation_caching
    - test_memory_usage
    - test_concurrent_sanitization
```

### Test File: `test_sanitizer_v2_comprehensive.py`

#### 1. Enhanced Features Tests
```python
class TestEnhancedFeatures:
    - test_contextual_sanitization
    - test_smart_detection
    - test_partial_masking
    - test_format_preservation
    - test_reversible_sanitization
```

#### 2. Advanced Patterns Tests
```python
class TestAdvancedPatterns:
    - test_nested_pii_detection
    - test_international_formats
    - test_custom_entity_detection
    - test_ml_based_detection
    - test_pattern_learning
```

#### 3. Integration Tests
```python
class TestSanitizerIntegration:
    - test_with_logger_integration
    - test_with_metrics_collector
    - test_with_config_system
    - test_with_console_output
    - test_end_to_end_flow
```

## Implementation Strategy

### Phase 1: Core Functionality (2 hours)
1. Create test files with proper structure
2. Implement basic sanitization tests
3. Test all built-in PII patterns
4. Verify masking and replacement logic

### Phase 2: Advanced Features (1 hour)
1. Test compliance modes thoroughly
2. Implement whitelist functionality tests
3. Test custom pattern management
4. Verify thread safety

### Phase 3: Edge Cases and Performance (30 minutes)
1. Test edge cases (empty input, None, special chars)
2. Performance benchmarks
3. Memory usage tests
4. Concurrent access tests

## Test Data Preparation

### Sample PII Data
```python
TEST_DATA = {
    "emails": ["test@example.com", "user.name@company.org"],
    "phones": ["123-456-7890", "+1 (555) 123-4567"],
    "credit_cards": ["4111111111111111", "5500 0000 0000 0004"],
    "ssns": ["123-45-6789", "987654321"],
    "ips": ["192.168.1.1", "2001:0db8:85a3::8a2e:0370:7334"]
}
```

### Compliance Test Scenarios
- GDPR: Full PII removal
- HIPAA: Medical record protection
- PCI DSS: Payment card masking
- SOC2: Audit trail preservation

## Success Criteria
- 100% coverage for both sanitizer modules
- All security patterns working correctly
- Compliance modes fully tested
- Performance benchmarks established
- No security vulnerabilities

## Next Steps
1. Create test files
2. Implement test classes systematically
3. Run coverage after each class
4. Update status.json with progress
5. Move to Step 04 after achieving 100% coverage

## Automation Notes
- Use parameterized tests for pattern variations
- Create fixtures for test data
- Mock external dependencies
- Test in isolation before integration
- Document any security considerations