# CaDaR Agent Guidelines

This document provides essential information for agents working with the CaDaR codebase.

## Project Overview

CaDaR (Canonicalization and Darija Representation) is a high-performance bidirectional transliteration library for Darija (Moroccan Arabic). It provides conversion between Arabic script and Latin (Romanized/Bizi) script with standardization capabilities.

- **Language**: Rust core with Python bindings via PyO3
- **Architecture**: 6-stage pipeline with Intermediate Canonical Representation (ICR)
- **Target**: Darija text processing (currently Moroccan dialect, extensible for others)

## Essential Commands

### Building and Development

```bash
# Development build (fast compilation)
maturin develop

# Release build (optimized)
maturin build --release

# Using the build script (recommended)
./build.sh dev      # Development
./build.sh release  # Release

# Clean build artifacts
cargo clean
rm -rf target/
```

### Testing

```bash
# Run all Rust tests
cargo test

# Run specific module tests
cargo test script_detection
cargo test normalization
cargo test icr

# Run tests with verbose output
cargo test -- --nocapture

# Run specific test
cargo test test_ara2bizi -- --nocapture

# Test Python bindings after installation
python -c "import cadar; print(cadar.ara2bizi('سلام', darija='Ma'))"
```

### Running Examples

```bash
# After building with maturin develop
python examples/basic_usage.py
```

### Documentation

```bash
# Build documentation locally
pip install mkdocs mkdocs-material mkdocstrings[python] markdown-include pymdown-extensions
mkdocs build --strict

# Serve documentation locally
mkdocs serve

# Generate Python API docs
python docs/generate_api_docs.py
```

### Publishing and Release

```bash
# Build and push release (automation)
./push_and_release.sh

# Manual tag creation
git tag -a v0.1.10 -m "Release version 0.1.10"
git push origin v0.1.10

# Build wheels for all platforms
maturin build --release --strip --out dist

# Build source distribution
maturin build --release --sdist --out dist
```

### Benchmarking

```bash
# Run benchmarks
cargo bench

# Generate benchmark report
python benchmark_cadar.py

# Analyze improvements
python analyze_improvements.py
```

## Project Structure

```
src/
├── lib.rs                    # Main library entry point, CaDaR struct
├── python_bindings.rs       # PyO3 bindings for Python
├── types/                   # Type definitions
│   ├── dialect.rs           # Dialect enum (Moroccan, etc.)
│   ├── script.rs            # Script detection (Arabic, Latin, Mixed)
│   └── mod.rs
├── linguistic/              # Linguistic processing utilities
│   ├── emphatic.rs          # Emphatic consonant handling
│   ├── patterns.rs          # Linguistic patterns
│   ├── vowels.rs            # Vowel processing
│   └── mod.rs
└── stages/                  # Processing pipeline stages
    ├── mod.rs               # Stage error types and common definitions
    ├── script_detection.rs  # Stage 1: Detect input script
    ├── normalization.rs     # Stage 2: Clean and normalize
    ├── tokenization.rs      # Stage 3: Darija-aware tokenization
    ├── icr.rs               # Stage 4: Build Intermediate Canonical Representation
    ├── script_generation.rs # Stage 5: Generate target script
    └── validation.rs        # Stage 6: Validate and fix output

python/
└── cadar/
    └── __init__.py           # Python package initialization

examples/
└── basic_usage.py           # Usage examples

docs/
├── dev/                     # Developer documentation
│   ├── building.md          # Build instructions
│   ├── testing.md           # Testing guide
│   └── contributing.md      # Contribution guidelines
└── guide/                   # User documentation
    ├── architecture.md      # Architecture overview
    ├── examples.md           # Usage examples
    └── overview.md
```

## Architecture Overview

CaDaR uses a 6-stage pipeline:

1. **Script Detection**: Identify if input is Arabic, Latin, or Mixed
2. **Normalization**: Clean text, remove diacritics, normalize characters
3. **Tokenization**: Split text into Darija-aware tokens
4. **ICR Building**: Convert to Intermediate Canonical Representation
5. **Script Generation**: Generate target script from ICR
6. **Validation**: Validate and fix the final output

### Key Innovation: ICR

The **Intermediate Canonical Representation (ICR)** is a script-independent, phonologically-grounded representation that:
- Abstracts away script-specific quirks
- Preserves Darija phonological distinctions
- Enables lossless round-trip conversions
- Allows consistent normalization across scripts

## Code Patterns and Conventions

### Error Handling

- Use `StageResult<T>` for pipeline operations
- Define errors in `stages/mod.rs` `StageError` enum
- Wrap external errors with context: `StageError::Normalization(format!("Failed: {}", e))`

### Dialect Handling

```rust
// Dialect is passed through the pipeline
pub struct CaDaR {
    dialect: Dialect,
    // ... other fields
}

impl CaDaR {
    pub fn new(dialect: Dialect) -> Self {
        // Create processors with dialect awareness
        tokenizer: Tokenizer::new(dialect),
        icr_builder: ICRBuilder::new(dialect),
        validator: Validator::new(dialect),
    }
}
```

### Pipeline Architecture

All processing follows the 6-stage pipeline pattern:

```rust
fn transliterate(&self, text: &str, target_script: Script) -> StageResult<String> {
    // Stage 1: Script Detection
    let detection_result = self.detector.detect(text)?;
    let source_script = detection_result.script;

    // Stage 2: Normalization
    let normalized = self.normalizer.normalize(text, source_script)?;

    // Stage 3: Tokenization
    let tokens = self.tokenizer.tokenize(&normalized, source_script)?;

    // Stage 4: ICR Building
    let icr = self.icr_builder.build_from_tokens(&tokens, source_script)?;

    // Stage 5: Script Generation
    let generated = self.generator.generate(&icr, target_script)?;

    // Stage 6: Validation
    let validated = self.validator.validate_and_fix(&generated, target_script)?;

    Ok(validated)
}
```

### Debugging Tools

The project includes comprehensive debugging utilities:

```bash
# Debug specific stages
./debug_normalization_simple  # Normalization stage
./debug_icr.py               # ICR building stage
./debug_step_by_step.py      # Full pipeline step-by-step

# Debug linguistic features
./debug_laa.py               # لام التعليل (alif with laam)
./debug_long_vowels.py       # Long vowel handling
./debug_words.py             # Word-level processing
```

### Testing Patterns

```rust
#[test]
fn test_ara2bizi() {
    let cadar = CaDaR::new(Dialect::Moroccan);
    let result = cadar.ara2bizi("سلام").unwrap();
    assert!(!result.is_empty());
    assert_eq!(Script::detect(&result), Script::Latin);
}

#[test]
fn test_round_trip() {
    let cadar = CaDaR::new(Dialect::Moroccan);
    let original = "salam";
    let arabic = cadar.bizi2ara(original).unwrap();
    let back_to_latin = cadar.ara2bizi(&arabic).unwrap();
    // Check round-trip consistency
}
```

### Python API Patterns

All Rust functions must have Python wrappers with consistent error handling:

```rust
#[pyfunction]
fn ara2bizi(text: &str, darija: Option<String>) -> PyResult<String> {
    let dialect = parse_dialect(darija)?;
    let cadar = CaDaR::new(dialect);
    cadar.ara2bizi(text)
        .map_err(|e| PyValueError::new_err(e.to_string()))
}
```

## Important Gotchas

### Unicode and Text Processing

- Always use Unicode normalization: `text.nfc().collect::<String>()`
- Handle zero-width characters and invisible marks
- Arabic text requires special handling for diacritics (harakat)
- Latin text must handle Darija-specific characters (3, 7, 9 for ع, ح, ق)

### Version Management

- Version numbers are stored in both `Cargo.toml` and `pyproject.toml`
- GitHub Actions automatically update versions when creating releases
- Use the release script for consistency: `./push_and_release.sh`

### ICR Complexity

- ICR segments preserve linguistic context
- Word-level analysis affects character-level processing
- Emphatic consonants require special handling
- Function words (من، في، ديال) need special treatment
- Long vowels (آ, إ, ا) have complex handling rules

### Script Detection Nuances

- Mixed script (Arabic + Latin) should be handled gracefully
- Numbers and punctuation are script-neutral
- Arabic includes various Unicode ranges beyond basic Arabic
- Darija Latin uses digit substitutions: 3=ع, 7=ح, 9=ق

### Development Workflow

1. **Always test after changes**: `cargo test && python -c "import cadar; print('OK')"`
2. **Use maturin develop for iterative development** (faster than full release builds)
3. **Check Python bindings work** after Rust changes
4. **Run examples** to verify end-to-end functionality
5. **Test edge cases**: empty strings, mixed scripts, very long text
6. **Use debug scripts** when investigating issues
7. **Benchmark performance** changes with `cargo bench`

### Darija-Specific Considerations

- Moroccan Darija is the only supported dialect currently
- Darija has unique phonological patterns not found in Modern Standard Arabic
- Clitic splitting is important: prefixes like و، ف، ب، ل
- Common words: بزاف، غير، واخا need special handling
- لام التعليل (laam al-ta'leel) requires special vowel handling

## Testing Approach

### Test Coverage

The project has comprehensive test coverage:
- **Unit tests**: In each module alongside source code
- **Integration tests**: In `src/lib.rs`
- **Expected**: Currently 66 tests passing, 14 failing (under active development)
- **Total**: 80 tests in the test suite

### Test Data

Common test phrases for validation:
```rust
// Greetings
"سلام" → "slam"
"السلام عليكم" → "salam 3likom"

// Questions  
"كيفاش داير؟" → "kifash dayer?"

// Common words
"واخا" → "wakha"
"بزاف" → "bzaf"
"شوية" → "shwiya"
```

### Python Testing

After building, test Python functionality:
```python
import cadar

# Test basic conversion
assert cadar.ara2bizi('سلام', darija='Ma') == 'slam'
assert cadar.bizi2ara('slam', darija='Ma') == 'سلام'

# Test standardization
assert cadar.ara2ara('أنَا', darija='Ma') == 'انا'
assert cadar.bizi2bizi('salaaaam', darija='Ma') == 'salam'

# Test convenience functions
assert cadar.transliterate('سلام', target='latin', darija='Ma') == 'slam'
assert cadar.standardize('salaaaam', script='auto', darija='Ma') == 'salam'
```

### Debug Tests

When investigating issues, use the specialized test scripts:
```bash
# Test normalization in isolation
python test_normalization_simple.py

# Test direct conversion bypassing some stages
python test_direct_conversion.py

# Test long vowel handling
python test_long_vowels.py

# Test full pipeline
python test_pipeline.py
```

## Dependencies and Tools

### Rust Dependencies
- **PyO3**: Python bindings (v0.24.1)
- **Serde**: Serialization
- **Unicode normalization**: Text processing
- **Unicode segmentation**: Word boundary detection
- **Regex**: Pattern matching
- **thiserror/anyhow**: Error handling
- **criterion**: Benchmarking (dev-dependency)

### Python Dependencies
- **maturin**: Rust-Python packaging
- **mkdocs**: Documentation generation
- **mkdocs-material**: Documentation theme
- **mkdocstrings**: API documentation
- **pymdown-extensions**: Markdown extensions

### Build Tools
- **Maturin**: Rust-Python packaging
- **Cargo**: Rust package manager
- **Python 3.8+**: Runtime environment
- **mkdocs**: Documentation generation

### Development Setup
1. Install Rust: `curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh`
2. Install Python 3.8+
3. Install Maturin: `pip install maturin`
4. Build: `maturin develop`
5. Test: `cargo test && python examples/basic_usage.py`

## Performance Considerations

- CaDaR is designed for high performance (~1-2ms per sentence)
- Memory efficiency is important (minimal allocations)
- Use Rust's zero-cost abstractions
- Benchmark with criterion if needed: `cargo bench`
- Profile performance with debug scripts when investigating slowdowns

### Release Process

The project uses automated CI/CD for releases:

1. **Tag Creation**: Create a version tag (e.g., `v0.1.10`)
2. **GitHub Actions**: Automatically triggers:
   - Multi-platform wheel building (Linux, macOS, Windows)
   - Python version compatibility testing (3.8-3.12)
   - PyPI publishing
   - Documentation building and deployment
   - GitHub release creation

3. **Version Management**: 
   - Tags update both `Cargo.toml` and `pyproject.toml`
   - Consistent versioning across Rust and Python packages

### CI/CD Pipelines

- **`.github/workflows/publish-pypi.yml`**: Builds and publishes to PyPI
- **`.github/workflows/docs.yml`**: Builds and deploys documentation
- **`push_and_release.sh`**: Helper script for creating releases

## Future Extensions

The architecture supports:
- Additional dialects (Algerian, Tunisian, etc.)
- Advanced morphological analysis
- Machine learning enhancements
- Web API and CLI tools
- Browser-based demo

When contributing, maintain the extensible design patterns.

## Getting Help

- **Documentation**: `docs/` directory
- **Examples**: `examples/basic_usage.py`
- **Tests**: Look at existing tests for patterns
- **Architecture**: `docs/guide/architecture.md`
- **Building**: `docs/dev/building.md`
- **Testing**: `docs/dev/testing.md`
- **Debug Scripts**: `debug_*.py` files for troubleshooting
- **Issues**: Check `ISSUES.md` for known problems and workarounds

### Common Issues

- **Test Failures**: Some tests are currently failing due to ongoing ICR improvements. Check `ISSUES.md`.
- **Build Errors**: Ensure Rust and Python versions are compatible. Use `./build.sh` for setup checks.
- **Import Errors**: Run `maturin develop` to build Python bindings after Rust changes.

Remember: This is a linguistic processing library - accuracy and cultural appropriateness are as important as code quality.