# Model Validation Improvements

## Background and Issue

The ModelChecker framework currently lacks a complete validation mechanism to ensure that models found by the Z3 solver truly satisfy the intended behavior of the semantic theory. Specifically, there's a disconnect between:

1. The constraint generation phase, where `ModelConstraints` applies the premise and conclusion behaviors defined in `define_invalidity()` 
2. The verification that models found actually satisfy these behaviors at the evaluation point

This causes cases like the `BM_CM_5` example where the solver returns a model in which both premises and conclusions are true, despite the semantic theory's explicit attempt to ensure conclusions are false at the evaluation point.

## Root Cause Analysis

The issue stems from a fundamental architectural gap:

1. `BimodalSemantics.define_invalidity()` correctly defines lambda functions for premise and conclusion behaviors
2. `ModelConstraints.__init__()` correctly applies these functions to create Z3 constraints
3. `ModelDefaults.solve()` attempts to find a model satisfying these constraints
4. The model is returned without verifying that it actually satisfies the intended behavior (premises true, conclusions false) at the evaluation point

The `BimodalSemantics.verify_model()` method exists but is never called in the execution flow.

## Implementation Plan

The solution requires a systematic approach that ensures model validation across all semantic theories, not just for BimodalSemantics. We'll introduce a formal validation step in the model building process that respects our design philosophy:

### 1. Standardize Model Verification Interface

Add a `verify_model` method to the `SemanticDefaults` base class:

```python
def verify_model(self, z3_model, premises, conclusions):
    """Verify that premises are true and conclusions are false in the found model.
    
    This method checks whether the model generated by Z3 correctly satisfies the 
    constraints for invalidating an argument - i.e., that all premises are true and
    all conclusions are false at the main evaluation point.
    
    Args:
        z3_model: The Z3 model to verify
        premises: List of premise formulas
        conclusions: List of conclusion formulas
        
    Returns:
        dict: Verification results with these keys:
            "premises_verified": Boolean indicating if all premises are true
            "conclusions_verified": Boolean indicating if all conclusions are false
            "errors": List of specific error messages for failed verifications
            
    Raises:
        NotImplementedError: If a derived semantic theory doesn't implement this method
    """
    raise NotImplementedError(
        "Semantic theories must implement verify_model to validate Z3 model results."
    )
```

### 2. Integrate Verification in ModelDefaults

Modify `ModelDefaults.__init__` to verify models after solving:

```python
def __init__(self, model_constraints, settings):
    # ... existing initialization code ...
    
    # Solve Z3 constraints and store results
    solver_results = self.solve(self.model_constraints, self.max_time)
    self._process_solver_results(solver_results)
    
    # Exit early if no valid model was found
    if self.timeout or self.z3_model is None:
        return
        
    # Verify that the model actually satisfies our logical requirements
    self.verification_results = self._verify_model()
    
    # If verification fails, invalidate the model
    if not self._is_model_valid():
        print("WARNING: Model found by Z3 does not satisfy required logical properties.")
        print("Model verification failed with these issues:")
        for error in self.verification_results["errors"]:
            print(f"- {error}")
        
        # Set z3_model to None to indicate no valid model was found
        # This is a clean approach that treats invalid models like no models
        self.z3_model = None
        self.z3_model_status = False
```

Add the verification methods to ModelDefaults:

```python
def _verify_model(self):
    """Verify that the found model satisfies the intended logical properties.
    
    Uses the semantics implementation's verify_model method to check that
    premises are true and conclusions are false at the evaluation point.
    
    Returns:
        dict: Verification results with information about validation status
    """
    if not self.z3_model:
        return {
            "premises_verified": False,
            "conclusions_verified": False,
            "errors": ["No model available to verify"]
        }
        
    try:
        # Call the semantics-specific verification method
        return self.semantics.verify_model(
            self.z3_model, 
            self.premises, 
            self.conclusions
        )
    except NotImplementedError:
        # If theory doesn't implement verification, record this as an error
        return {
            "premises_verified": False,
            "conclusions_verified": False,
            "errors": ["Semantic theory does not implement model verification"]
        }
    except Exception as e:
        # Capture any other errors that might occur during verification
        return {
            "premises_verified": False,
            "conclusions_verified": False,
            "errors": [f"Verification error: {str(e)}"]
        }
        
def _is_model_valid(self):
    """Check if the model passed verification.
    
    A valid model requires:
    - All premises are true at the evaluation point
    - All conclusions are false at the evaluation point
    
    Returns:
        bool: True if model satisfies verification requirements
    """
    if not hasattr(self, 'verification_results'):
        return False
        
    # For invalidity checking, we want premises true and conclusions false
    return (
        self.verification_results.get("premises_verified", False) and
        self.verification_results.get("conclusions_verified", False)
    )
```

### 3. Update Print Methods to Show Verification Status

Update `ModelDefaults.print_info` to include verification results:

```python
def print_info(self, model_status, settings, example_name, theory_name, output=sys.__stdout__):
    # ... existing code ...
    
    # Add verification status when printing model info
    if hasattr(self, 'verification_results') and self.verification_results:
        print("\nModel Verification:", file=output)
        print(f"  Premises verified: {self.verification_results['premises_verified']}", file=output)
        print(f"  Conclusions verified: {self.verification_results['conclusions_verified']}", file=output)
        
        if self.verification_results["errors"]:
            print("\nVerification errors:", file=output)
            for error in self.verification_results["errors"]:
                print(f"  - {error}", file=output)
    
    # ... rest of existing code ...
```

### 4. Implement in Each Semantic Theory

For each semantic theory, ensure proper implementation of `verify_model`:

1. Use existing implementation in `BimodalSemantics` 
2. Add implementations to other semantic theories (`DefaultSemantics`, `ExclusionSemantics`, etc.)

### 5. Add Tests for Model Verification

Create test cases that specifically verify the validation logic:

1. Test cases where models should be rejected (premise false or conclusion true)
2. Test cases with valid models (premises true, conclusions false)
3. Test error handling for invalid model states

## Expected Outcomes

This systematic approach will:

1. Close the architectural gap between constraint generation and model verification
2. Enforce the logical requirements (premises true, conclusions false) across all theories
3. Provide clear feedback when models don't satisfy the intended behavior
4. Maintain a clean API by treating invalid models as "no models found"

## Design Philosophy Alignment

This solution aligns with the project's design philosophy:

1. **Fail Fast**: Models that don't meet logical requirements fail immediately with clear errors
2. **No Silent Failures**: Invalid models are explicitly rejected, not silently accepted
3. **Structural Solutions**: Addresses the root cause in the architecture, not just a specific example
4. **Clear Data Flow**: Maintains a clear flow from definition to constraint to verification
5. **Error as Feedback**: Uses verification failures as signals to improve the model finding process
6. **Simplification**: Provides a cleaner mental model where "found model" always means "valid model"

## Next Steps

After implementation, consider these enhancements:

1. Add metrics about how often verification rejects models to identify optimization opportunities
2. Create more focused documentation about the model verification process
3. Add theory-specific verification steps for specialized logical properties