# Differential Testing Framework for croniter-rs

This directory contains a comprehensive differential testing framework that compares the Rust implementation (`croniter_rs`) against the original Python implementation (`croniter`).

## Overview

The framework provides:
- **Test case generation** covering all croniter features
- **Differential comparison** between Python and Rust implementations
- **Multiple test modes** (smoke, comprehensive, fuzzing)
- **Parallel execution** for CPU-saturating fuzzing
- **Code coverage analysis** using cargo-tarpaulin

## Files

- `test_generator.py` - Generates comprehensive test cases
- `differential_tester.py` - Core differential testing logic
- `run_tests.py` - Main test runner with multiple modes
- `parallel_fuzzer.py` - Parallel fuzzing for long-duration testing
- `run_coverage.sh` - Code coverage analysis script

## Quick Start

### 1. Smoke Tests (Quick validation)

```bash
cd /home/ubuntu-exp/cronproject/croniter-rs
source /home/ubuntu-exp/cronproject/.venv/bin/activate
python tests/differential/run_tests.py smoke
```

Runs ~10 basic test cases to verify the framework is working.

### 2. Comprehensive Tests (Full validation)

```bash
python tests/differential/run_tests.py comprehensive
```

Runs hundreds of test cases covering:
- All cron operators (*, -, /, ,, L, #, ?, H, R)
- All field formats (5, 6, 7 fields)
- Boundary values and edge cases
- Equivalence class partitioning
- Hash and random expressions

### 3. Quick Fuzzing (Random testing)

```bash
python tests/differential/run_tests.py quick-fuzz --count 1000
```

Generates and tests random cron expressions.

### 4. Long-Duration Fuzzing (24-hour CPU saturation)

```bash
# Single-threaded
python tests/differential/run_tests.py fuzz --duration 24

# Parallel (recommended for fuzzing)
python tests/differential/parallel_fuzzer.py --duration 24 --workers 8
```

Runs continuous fuzzing for the specified duration using multiple CPU cores.

## Test Coverage

### Operators Tested

- `*` - Wildcard (all values)
- `-` - Range (e.g., 1-5)
- `/` - Step (e.g., */5, 1-10/2)
- `,` - List (e.g., 1,3,5)
- `L` - Last day of month (e.g., L, L-1)
- `#` - Nth weekday (e.g., MON#1, FRI#2)
- `L` - Last weekday of month (e.g., MONL, FRIL)
- `H` - Hash (Jenkins-style consistent hashing)
- `R` - Random (random value within range)

### Field Formats

- **5-field**: `minute hour day month weekday`
- **6-field**: `second minute hour day month weekday`
- **7-field**: `minute hour day month weekday year`

### Special Cases

- Leap years (Feb 29)
- Month boundaries (28, 29, 30, 31 days)
- Year boundaries
- DST transitions (if timezone-aware)
- Weekday calculations
- Named months (JAN-DEC)
- Named weekdays (MON-SUN)

### API Methods Tested

- `get_next()` - Get next occurrence
- `get_prev()` - Get previous occurrence
- `match()` - Check if datetime matches expression
- `expand()` - Expand expression to component values
- `get_current()` - Get current time
- `set_current()` - Set current time

## Test Case Generation

The `TestCaseGenerator` class provides multiple strategies:

### 1. Basic Expressions
Pre-defined expressions covering common patterns.

### 2. Equivalence Class Partitioning
Groups inputs into equivalence classes and tests representatives from each class.

### 3. Boundary Value Analysis
Tests boundary values for each field (min, min+1, mid, max-1, max).

### 4. Random Fuzzing
Generates random valid cron expressions with random start dates.

## Output

### Log Files

- `/tmp/differential_tests.log` - Detailed test execution log
- `/tmp/differential_fuzzing.log` - Fuzzing-specific log
- `tests/differential/fuzzing_results/worker_*.log` - Per-worker logs

### Result Files

- `smoke_results.json` - Smoke test results
- `comprehensive_results.json` - Comprehensive test results
- `quick_fuzz_results.json` - Quick fuzzing results
- `fuzzing_final.json` - Long-duration fuzzing results
- `fuzzing_results/aggregate_results.json` - Parallel fuzzing aggregate

### Coverage Reports

- `coverage/index.html` - HTML coverage report
- `coverage/tarpaulin-report.json` - JSON coverage data

## Code Coverage

Run coverage analysis:

```bash
./tests/differential/run_coverage.sh
```

This generates:
- HTML report at `tests/differential/coverage/index.html`
- JSON data at `tests/differential/coverage/tarpaulin-report.json`

View the HTML report in a browser to see line-by-line coverage.

## Parallel Fuzzing

For maximum CPU utilization during long fuzzing runs:

```bash
# Use all available cores minus 1
python tests/differential/parallel_fuzzer.py --duration 24

# Specify number of workers
python tests/differential/parallel_fuzzer.py --duration 24 --workers 8

# Adjust batch size for memory/performance tuning
python tests/differential/parallel_fuzzer.py --duration 24 --batch-size 100
```

Each worker:
- Runs independently with its own random seed
- Saves checkpoints every 10 batches
- Logs to a separate file
- Reports progress periodically

## Interpreting Results

### Success Criteria

A test passes if:
- Both implementations produce the same results
- Both implementations raise errors for invalid inputs
- Timestamps match within 1ms precision
- Datetime objects match exactly

### Common Failure Patterns

1. **Timestamp precision** - Floating point rounding differences
2. **Timezone handling** - Different timezone interpretations
3. **Edge cases** - Boundary conditions (Feb 29, month ends)
4. **Unimplemented features** - Features not yet in Rust version

### Analyzing Failures

Check the detailed log file for:
- Exact expression that failed
- Start datetime
- Expected vs actual results
- Full stack traces for errors

Example:
```
Test: Basic: */5 * * * *
Expression: */5 * * * *
Start: 2020-01-01T00:00:00
get_next: FAIL
  Differences: ['Iteration 3: py=2020-01-01T00:15:00, rs=2020-01-01T00:20:00']
```

## Performance Benchmarking

While this framework focuses on correctness, you can measure performance:

```python
import time
from differential_tester import DifferentialTester

tester = DifferentialTester()
start = time.time()
# Run tests
elapsed = time.time() - start
print(f"Tests per second: {num_tests / elapsed:.2f}")
```

## Extending the Framework

### Add New Test Cases

Edit `test_generator.py`:

```python
def generate_custom_tests(self):
    return [
        {
            'expr': 'your expression',
            'start': datetime(...),
            'iterations': 10,
            'description': 'Your test'
        }
    ]
```

### Add New Comparison Methods

Edit `differential_tester.py`:

```python
def compare_new_method(self, expr, ...):
    # Compare new API method
    pass
```

### Add New Test Modes

Edit `run_tests.py`:

```python
def run_custom_mode():
    # Your custom test mode
    pass
```

## Troubleshooting

### Import Errors

Ensure the Python virtual environment is activated:
```bash
source /home/ubuntu-exp/cronproject/.venv/bin/activate
```

### croniter_rs Not Found

Build and install the Rust module:
```bash
cd /home/ubuntu-exp/cronproject/croniter-rs
maturin develop --release
```

### Coverage Tool Not Found

Install cargo-tarpaulin:
```bash
cargo install cargo-tarpaulin
```

### Out of Memory

Reduce batch size for fuzzing:
```bash
python tests/differential/parallel_fuzzer.py --batch-size 25
```

## Best Practices

1. **Start with smoke tests** - Verify basic functionality first
2. **Run comprehensive tests** - Before committing changes
3. **Use parallel fuzzing** - For long-duration testing
4. **Monitor logs** - Check for patterns in failures
5. **Track coverage** - Ensure all code paths are tested
6. **Save results** - Keep historical data for regression analysis

## CI/CD Integration

Add to your CI pipeline:

```yaml
- name: Run differential tests
  run: |
    source .venv/bin/activate
    python tests/differential/run_tests.py comprehensive
```

## Future Enhancements

- [ ] Property-based testing with Hypothesis
- [ ] Mutation testing
- [ ] Performance regression detection
- [ ] Automatic bug report generation
- [ ] Integration with fuzzing tools (AFL, libFuzzer)
- [ ] Continuous fuzzing infrastructure

## References

- Original croniter: https://github.com/kiorky/croniter
- Cron expression format: https://en.wikipedia.org/wiki/Cron
- Differential testing: https://en.wikipedia.org/wiki/Differential_testing
