# Feature Specification: Data Validation System

**GitHub Issue**: #27 (hypothetical)  
**Branch**: `feat/issue-27-data-validation`  
**Protocol(s)**: `DataProcessor` from `specs/interfaces/example.py`

---

## 📋 Overview

**What**: A comprehensive data validation system that validates input data against configurable rules and schemas.

**Why**: Library users need to validate data before processing to prevent errors and ensure data quality. Current library lacks input validation capabilities.

**Who**: Python developers using this library for data processing applications.

---

## 🎯 Acceptance Criteria

### Core Functionality
- [ ] **Given** a data validator with rules, **When** valid data is submitted, **Then** validation passes and returns success result
- [ ] **Given** a data validator with rules, **When** invalid data is submitted, **Then** validation fails and returns detailed error information
- [ ] **Given** multiple validation rules, **When** data is validated, **Then** all rules are checked and comprehensive results returned

### Edge Cases & Error Handling
- [ ] **Given** no validation rules configured, **When** any data is validated, **Then** validation passes (permissive by default)
- [ ] **Given** malformed validation rules, **When** validator is created, **Then** raises ValidationConfigError with clear message
- [ ] **Given** None or empty data, **When** validation occurs, **Then** handles gracefully based on allow_empty setting

### Performance & Quality
- [ ] **Given** large datasets (10k+ records), **When** batch validation occurs, **Then** completes within 5 seconds
- [ ] Test coverage: 90%+ for all new code
- [ ] All public functions have docstrings with examples
- [ ] Type hints for all function parameters and returns

---

## 🔧 Technical Specification

### Python Protocols
- **Implements**: `DataProcessor` from `specs/interfaces/example.py`
- **Extends**: None
- **Dependencies**: `jsonschema` (for JSON schema validation), `pydantic` (for model validation)

### API Design

```python
from typing import Any, Dict, List, Optional, Union
from enum import Enum

class ValidationSeverity(Enum):
    """Severity levels for validation results."""
    INFO = "info"
    WARNING = "warning"  
    ERROR = "error"

class ValidationResult:
    """Result of a data validation operation."""

    def __init__(self, success: bool, errors: List[Dict[str, Any]], data: Any = None):
        self.success = success
        self.errors = errors
        self.data = data
        self.metadata = {"validated_at": datetime.utcnow()}

class DataValidator:
    """Validates data against configurable rules and schemas."""

    def __init__(
        self,
        rules: Optional[Dict[str, Any]] = None,
        *,
        allow_empty: bool = False,
        strict_mode: bool = True
    ) -> None:
        """Initialize validator with rules.

        Args:
            rules: Validation rules dictionary or JSON schema
            allow_empty: Whether to allow None/empty values
            strict_mode: Whether to fail on warnings
        """
        pass

    def validate(self, data: Any) -> ValidationResult:
        """Validate single data item.

        Args:
            data: Data to validate

        Returns:
            ValidationResult with success/failure and details

        Raises:
            ValidationConfigError: If validator rules are invalid
        """
        pass

    def validate_batch(self, data_list: List[Any]) -> List[ValidationResult]:
        """Validate multiple data items efficiently.

        Args:
            data_list: List of data items to validate

        Returns:
            List of ValidationResult objects
        """
        pass

    def add_rule(self, name: str, rule: Dict[str, Any]) -> None:
        """Add or update a validation rule.

        Args:
            name: Rule identifier
            rule: Rule configuration
        """
        pass

# Convenience functions
def validate_json_schema(data: Any, schema: Dict[str, Any]) -> ValidationResult:
    """Validate data against JSON schema."""
    pass

def validate_required_fields(data: Dict[str, Any], fields: List[str]) -> ValidationResult:
    """Validate that required fields are present."""
    pass
```

### Configuration
- **Environment Variables**:
  - `AUGINT_VALIDATION_MODE`: Validation strictness level (default: `strict`)
- **Configuration Options**:
  - `validation_timeout`: Max time for validation in seconds (default: 30)

---

## 🧪 Test Scenarios

### Happy Path Tests
```python
def test_validator_valid_data_passes():
    """Test validation passes with valid data."""
    # Arrange
    rules = {"type": "object", "required": ["name", "email"]}
    validator = DataValidator(rules)
    data = {"name": "Alice", "email": "alice@example.com"}

    # Act
    result = validator.validate(data)

    # Assert
    assert result.success is True
    assert len(result.errors) == 0
    assert result.data == data

def test_validator_batch_validation():
    """Test batch validation processes multiple items."""
    # Arrange
    validator = DataValidator({"type": "string", "minLength": 2})
    data_list = ["hello", "world", "test"]

    # Act
    results = validator.validate_batch(data_list)

    # Assert
    assert len(results) == 3
    assert all(r.success for r in results)
```

### Edge Case Tests
```python
def test_validator_invalid_data_fails():
    """Test validation fails with invalid data and provides details."""
    # Arrange
    rules = {"type": "string", "minLength": 5}
    validator = DataValidator(rules)

    # Act
    result = validator.validate("hi")  # Too short

    # Assert
    assert result.success is False
    assert len(result.errors) == 1
    assert "minLength" in result.errors[0]["message"]

def test_validator_empty_data_handling():
    """Test empty data handling based on configuration."""
    # Arrange
    validator_allow = DataValidator(allow_empty=True)
    validator_deny = DataValidator(allow_empty=False)

    # Act & Assert
    assert validator_allow.validate(None).success is True
    assert validator_deny.validate(None).success is False

def test_validator_malformed_rules():
    """Test proper error handling for invalid rules."""
    # Act & Assert
    with pytest.raises(ValidationConfigError) as exc_info:
        DataValidator({"invalid": "rule format"})
    assert "rule format" in str(exc_info.value)
```

### Integration Tests
```python
@pytest.mark.slow
def test_validator_large_dataset_performance():
    """Test validation performance with large datasets."""
    # Arrange
    validator = DataValidator({"type": "object", "required": ["id"]})
    large_dataset = [{"id": i} for i in range(10000)]

    # Act
    start_time = time.time()
    results = validator.validate_batch(large_dataset)
    elapsed = time.time() - start_time

    # Assert
    assert elapsed < 5.0  # Must complete within 5 seconds
    assert len(results) == 10000
    assert all(r.success for r in results)
```

---

## 📚 Usage Examples

### Basic Usage
```python
from augint_library.validation import DataValidator, validate_required_fields

# Simple field validation
result = validate_required_fields(
    data={"name": "Alice", "email": "alice@example.com"},
    fields=["name", "email"]
)

if result.success:
    print("Data is valid!")
else:
    for error in result.errors:
        print(f"Error: {error['message']}")
```

### Advanced Usage  
```python
# JSON schema validation
schema = {
    "type": "object",
    "properties": {
        "name": {"type": "string", "minLength": 2},
        "age": {"type": "integer", "minimum": 0},
        "email": {"type": "string", "format": "email"}
    },
    "required": ["name", "email"]
}

validator = DataValidator(schema, strict_mode=True)

# Validate single item
result = validator.validate({
    "name": "Alice",
    "age": 30,
    "email": "alice@example.com"
})

# Validate batch
users = [
    {"name": "Alice", "email": "alice@example.com"},
    {"name": "Bob", "email": "bob@example.com"},
]
results = validator.validate_batch(users)

# Process results
for i, result in enumerate(results):
    if not result.success:
        print(f"User {i} validation failed:")
        for error in result.errors:
            print(f"  - {error['message']}")
```

### CLI Usage (if applicable)
```bash
# Validate JSON file against schema
ai-test-script validate data.json --schema schema.json

# Batch validate directory of files
ai-test-script validate ./data/ --schema schema.json --recursive
```

---

## 🚀 Implementation Plan

### Phase 1: Core Implementation
1. [ ] Create failing tests based on acceptance criteria
2. [ ] Implement `ValidationResult` class
3. [ ] Implement basic `DataValidator` class
4. [ ] Add JSON schema validation support
5. [ ] Add comprehensive docstrings

### Phase 2: Integration & Polish
1. [ ] Add batch validation with performance optimization
2. [ ] Add convenience validation functions
3. [ ] Update CLI interface with validation commands
4. [ ] Add configuration options and environment variables

### Phase 3: Validation
1. [ ] Run full test suite (`make test-all`)
2. [ ] Run security scans (`make security`)  
3. [ ] Generate documentation (`make docs`)
4. [ ] Performance testing with large datasets
5. [ ] Manual testing of all examples

---

## 🔗 Related Work

**Dependencies**:
- `jsonschema>=4.0.0` (JSON Schema validation)
- `email-validator>=1.3.0` (Email format validation)

**Documentation Updates**:
- [ ] Update README.md with validation examples
- [ ] Add validation guide to library documentation
- [ ] Update CLI help text with validation commands

**Follow-up Issues**:
- Custom validation rule plugins (#28)
- Performance optimization for streaming validation (#29)
- Integration with popular data frameworks (pandas, etc.) (#30)

---

## 🎛️ AI Implementation Notes

### Implementation Order
1. **Read the DataProcessor Protocol** in `specs/interfaces/example.py` first
2. **Write failing tests** for ValidationResult and DataValidator classes
3. **Implement minimal classes** to pass basic tests
4. **Add JSON schema support** and comprehensive validation
5. **Optimize for batch processing** while maintaining test coverage

### Key Patterns to Follow
- Use existing library patterns from `src/augint_library/`
- Follow Google docstring format with examples
- Add type hints for all parameters and returns
- Use `pytest.mark.slow` for performance tests
- Raise `ValidationConfigError` for configuration issues
- Use `ValidationSeverity` enum for error classification

### Code Quality Checks
Before completing implementation:
```bash
make lint      # Fix any linting issues  
make test      # All tests must pass (including new validation tests)
make security  # No security issues (especially with schema validation)
make docs      # Documentation builds with new validation examples
```

---

**Implementation Status**: 🔄 Ready for Implementation
