# Step 07: Pattern System Tests

## Status Reference
- **Current Status**: Pending
- **Dependencies**: Steps 01-06 should be complete
- **Target Modules**:
  - `storm_logger/utils/patterns/base.py` (30.88% coverage)
  - `storm_logger/utils/patterns/payment.py` (19.30% coverage)
  - `storm_logger/utils/patterns/registry.py` (17.43% coverage)
  - `storm_logger/utils/patterns/sensitive_data.py` (44.44% coverage)
- **Target Coverage**: 100% for all pattern modules

## Objectives
1. Test pattern matching system comprehensively
2. Verify payment pattern detection
3. Test pattern registry functionality
4. Ensure sensitive data patterns work correctly
5. Test pattern priority and conflicts
6. Verify performance with complex patterns

## Coverage Gap Analysis

### base.py - Missing Coverage
Missing lines: 32-33, 48-49, 68-71, 90-100, 112-138, 150-161, 173, 177, 189-199, 211-215, 226-230, 242-244, 248

Key areas:
- Pattern compilation and caching
- Match finding algorithms
- Pattern priority handling
- Replacement strategies
- Pattern validation

### payment.py - Missing Coverage
Missing lines: 18-20, 24, 28, 89-102, 114-136, 146-187

Key areas:
- Credit card pattern variations
- IBAN validation
- Routing number patterns
- Payment provider patterns
- International formats

### registry.py - Missing Coverage
Missing lines: 21-22, 32-33, 42-44, 56, 71-109, 125-132, 145, 154-155, 167-168, 175-177, 181-183, 193-195, 199-202, 212-221

Key areas:
- Pattern registration
- Pattern lookup
- Category management
- Pattern conflicts
- Registry persistence

### sensitive_data.py - Missing Coverage
Missing lines: 16-18, 22, 26, 138-173

Key areas:
- Additional sensitive patterns
- Pattern combinations
- Edge cases

## Test Suite Design

### Test File: `test_pattern_base_comprehensive.py`

#### 1. Pattern Creation Tests
```python
class TestPatternCreation:
    - test_create_simple_pattern
    - test_create_complex_pattern
    - test_pattern_with_groups
    - test_pattern_with_flags
    - test_invalid_pattern_handling
    - test_pattern_compilation_caching
```

#### 2. Pattern Matching Tests
```python
class TestPatternMatching:
    - test_find_single_match
    - test_find_multiple_matches
    - test_no_matches
    - test_overlapping_matches
    - test_case_sensitivity
    - test_multiline_matching
    - test_unicode_matching
```

#### 3. Replacement Tests
```python
class TestPatternReplacement:
    - test_simple_replacement
    - test_group_replacement
    - test_callback_replacement
    - test_conditional_replacement
    - test_preserve_format_replacement
```

#### 4. Priority Tests
```python
class TestPatternPriority:
    - test_priority_ordering
    - test_same_priority_handling
    - test_priority_with_overlaps
    - test_dynamic_priority_adjustment
```

### Test File: `test_payment_patterns_comprehensive.py`

#### 1. Credit Card Tests
```python
class TestCreditCardPatterns:
    - test_visa_detection
    - test_mastercard_detection
    - test_amex_detection
    - test_discover_detection
    - test_invalid_card_numbers
    - test_card_number_formatting
    - test_partial_card_numbers
```

#### 2. IBAN Tests
```python
class TestIBANPatterns:
    - test_valid_iban_formats
    - test_country_specific_ibans
    - test_invalid_ibans
    - test_iban_checksum_validation
    - test_partial_iban_detection
```

#### 3. Routing Number Tests
```python
class TestRoutingNumbers:
    - test_us_routing_numbers
    - test_canadian_routing_numbers
    - test_invalid_routing_numbers
    - test_routing_number_formats
```

#### 4. Payment Provider Tests
```python
class TestPaymentProviders:
    - test_paypal_patterns
    - test_stripe_patterns
    - test_square_patterns
    - test_bitcoin_addresses
    - test_ethereum_addresses
```

### Test File: `test_pattern_registry_comprehensive.py`

#### 1. Registration Tests
```python
class TestPatternRegistration:
    - test_register_pattern
    - test_register_duplicate
    - test_unregister_pattern
    - test_bulk_registration
    - test_category_registration
```

#### 2. Lookup Tests
```python
class TestPatternLookup:
    - test_lookup_by_name
    - test_lookup_by_category
    - test_lookup_by_priority
    - test_fuzzy_lookup
    - test_lookup_performance
```

#### 3. Conflict Resolution Tests
```python
class TestConflictResolution:
    - test_overlapping_patterns
    - test_subset_patterns
    - test_priority_conflicts
    - test_automatic_resolution
    - test_manual_resolution
```

#### 4. Persistence Tests
```python
class TestRegistryPersistence:
    - test_save_registry
    - test_load_registry
    - test_export_import
    - test_versioning
    - test_migration
```

### Test File: `test_sensitive_data_patterns.py`

#### 1. PII Pattern Tests
```python
class TestPIIPatterns:
    - test_ssn_patterns
    - test_email_patterns
    - test_phone_patterns
    - test_address_patterns
    - test_name_patterns
```

#### 2. Medical Data Tests
```python
class TestMedicalPatterns:
    - test_patient_id_patterns
    - test_medical_record_numbers
    - test_insurance_numbers
    - test_prescription_patterns
```

#### 3. Financial Data Tests
```python
class TestFinancialPatterns:
    - test_account_numbers
    - test_tax_ids
    - test_invoice_numbers
    - test_transaction_ids
```

## Implementation Strategy

### Phase 1: Base Pattern System (45 minutes)
1. Test pattern creation and compilation
2. Verify matching algorithms
3. Test replacement strategies
4. Check priority handling

### Phase 2: Specialized Patterns (30 minutes)
1. Test payment patterns thoroughly
2. Verify sensitive data detection
3. Test international formats
4. Check edge cases

### Phase 3: Registry System (15 minutes)
1. Test registration mechanisms
2. Verify lookup functionality
3. Test conflict resolution
4. Check persistence

## Test Data

### Payment Test Data
```python
PAYMENT_TEST_DATA = {
    "credit_cards": {
        "valid": [
            "4111111111111111",  # Visa
            "5500000000000004",  # Mastercard
            "340000000000009",   # Amex
        ],
        "invalid": [
            "1234567890123456",
            "0000000000000000",
        ]
    },
    "ibans": {
        "valid": [
            "GB82 WEST 1234 5698 7654 32",
            "DE89 3704 0044 0532 0130 00",
        ],
        "invalid": [
            "GB00 XXXX 0000 0000 0000 00",
        ]
    }
}
```

### Pattern Test Cases
```python
PATTERN_TEST_CASES = [
    {
        "pattern": r"\b\d{3}-\d{2}-\d{4}\b",
        "test_string": "SSN: 123-45-6789",
        "expected_matches": ["123-45-6789"],
        "replacement": "XXX-XX-XXXX"
    },
    {
        "pattern": r"\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Z|a-z]{2,}\b",
        "test_string": "Email: user@example.com",
        "expected_matches": ["user@example.com"],
        "replacement": "[EMAIL]"
    }
]
```

## Performance Benchmarks

```python
def test_pattern_performance():
    patterns = [create_pattern(f"pattern_{i}") for i in range(1000)]
    test_text = "Large text with many potential matches" * 1000
    
    start = time.time()
    for pattern in patterns:
        pattern.find_all(test_text)
    duration = time.time() - start
    
    assert duration < 1.0  # Should process in under 1 second
```

## Success Criteria
- 100% coverage for all pattern modules
- All pattern types tested
- Registry fully functional
- Performance benchmarks met
- No pattern conflicts
- International formats supported

## Next Steps
1. Create test files for each module
2. Implement base pattern tests first
3. Add specialized pattern tests
4. Test registry functionality
5. Verify performance
6. Update status.json
7. Move to Step 08

## Automation Notes
- Use parameterized tests for pattern variations
- Create fixtures for common patterns
- Test with real-world data samples
- Mock external validators where needed
- Ensure thread safety for registry