---
sidebar_position: 1
---

# The Transliteration Pipeline

CaDaR uses a sophisticated 6-stage pipeline to ensure accurate bidirectional transliteration between Arabic and Latin scripts for Moroccan Darija. This architecture preserves meaning, handles context, and maintains linguistic accuracy throughout the conversion process.

import PipelineVisualization from '@site/src/components/PipelineVisualization';

<PipelineVisualization />

## Stage 1: Script Detection

The first stage analyzes the input text to determine its script composition:

### What It Does
- **Character Classification**: Identifies each character as Arabic, Latin, number, or punctuation
- **Script Ratio Calculation**: Determines the predominant script in mixed text
- **Pattern Recognition**: Detects common Darija patterns (e.g., numbers within Latin text)

### Implementation Details
```rust
pub struct ScriptDetector {
    // Uses Unicode ranges for identification
    // Analyzes character distribution
    // Returns Script::Arabic, Script::Latin, or Script::Mixed
}
```

### Edge Cases Handled
- Mixed script content (e.g., "3la lkol")
- Numbers representing Arabic letters (3, 7, 8, 9)
- Punctuation and emoji in mixed contexts

## Stage 2: Normalization

Normalization cleans and standardizes the input text for consistent processing:

### Arabic Normalization
- Removes diacritics and vocalization marks
- Standardizes different forms of Alef (أ, إ, ا → ا)
- Normalizes Teh Marbuta (ة) and other variants

### Latin Normalization
- Removes duplicate characters (salaaaam → salam)
- Normalizes spacing
- Handles common Latin representations of Arabic sounds

### Code Example
```rust
impl Normalizer {
    pub fn normalize(&self, text: &str, script: Script) -> StageResult<String> {
        match script {
            Script::Arabic => self.normalize_arabic(text),
            Script::Latin => self.normalize_latin(text),
            _ => text.to_string(),
        }
    }
}
```

## Stage 3: Tokenization

This stage breaks text into meaningful tokens while preserving Darija-specific constructs:

### Word Segmentation
- Standard whitespace-based tokenization
- Special handling for clitic prefixes
- Recognition of function words

### Clitic Detection
CaDaR recognizes and handles Darija clitics:
- **Prepositions**: `b-` (ب), `l-` (ل), `f-` (ف), `k-` (ك)
- **Conjunctions**: `w-` (و)
- **Pronouns**: `-i` (ي), `-k` (ك), `-ha` (ها), etc.

### Example Tokenization
```python
Input: "flkol 3la 7mar"

Tokens: [
    "f",  # preposition "في"
    "l",  # definite article "ال"
    "kol",  # word "كل"
    "3la",  # preposition "على"
    "7mar"  # word "حمر"
]
```

## Stage 4: ICR Building

The **Intermediate Canonical Representation (ICR)** is CaDaR's core innovation:

### What is ICR?
ICR is a script-independent, phonologically-grounded representation that:
- Abstracts away script-specific quirks
- Preserves Darija phonological distinctions
- Enables lossless round-trip conversions

### ICR Structure
```rust
pub struct ICRSegment {
    text: String,
    segment_type: SegmentType,
    // Additional metadata for context
    word_context: Option<WordContext>,
}

pub enum SegmentType {
    Consonant,
    Vowel,
    Number,
    Punctuation,
    Whitespace,
}
```

### Context Awareness
The ICR builder maintains:
- **Word Context**: Linguistic properties of the current word
- **Position Information**: Character position within word
- **Emphatic Detection**: Whether consonants should be emphatic

## Stage 5: Script Generation

Converts the ICR back to the target script with context-aware rules:

### Character Mapping Rules
- **Direct Mappings**: a → ا, b → ب, etc.
- **Context-Dependent**: t → ت or ط based on context
- **Special Sequences**: "ch" → ش, "Dr" → ض

### Emphatic Consonant Handling
```rust
fn map_t_consonant(word_context: &WordContext) -> ArabicChar {
    if word_context.is_emphatic() {
        ARABIC_TAH  // ط
    } else if word_context.has_aspect_marker() {
        ARABIC_TA   // ت
    } else {
        ARABIC_TA   // ت (default)
    }
}
```

### Vowel Intelligence
CaDaR implements smart vowel handling:
- **Darija Mode** (default): More readable output
- **Standard Mode**: Strict Arabic orthography
- **Position Awareness**: Different rules for start/middle/end of word

## Stage 6: Validation

Final validation ensures output quality:

### Validation Checks
- **Character Validation**: Ensures all characters are valid in target script
- **Pattern Validation**: Checks for invalid sequences
- **Quality Checks**: Detects and fixes common errors

### Error Recovery
- Fixes duplicate characters
- Corrects malformed words
- Preserves meaning while fixing structure

## Performance Optimizations

### Hybrid Architecture
CaDaR combines:
- **Character-level ICR** for precision
- **Word-level context** for accuracy
- **Efficient caching** of linguistic properties

### Benchmark Results
```
Input: "kayn chi wahed f dar"
Processing time: ~1.2ms
Memory usage: Linear with input size
Throughput: ~50,000 chars/second
```

## Testing the Pipeline

Each stage is thoroughly tested:
```rust
#[test]
fn test_script_detection() {
    let detector = ScriptDetector::new();
    
    // Test Arabic
    let result = detector.detect("سلام").unwrap();
    assert_eq!(result.script, Script::Arabic);
    
    // Test Latin
    let result = detector.detect("salam").unwrap();
    assert_eq!(result.script, Script::Latin);
    
    // Test Mixed
    let result = detector.detect("3la kol").unwrap();
    assert_eq!(result.script, Script::Mixed);
}
```

## Future Enhancements

Planned improvements to the pipeline:

1. **Neural Context**: Machine learning for better context detection
2. **Multi-dialect Support**: Extend to other Darija dialects
3. **Semantic Awareness**: Preserve meaning at sentence level
4. **Performance**: Further optimization for real-time applications

## Understanding the Pipeline

To better understand how the pipeline works:

1. Try the [interactive playground](/playground) to see real-time processing
2. Examine the [character mappings](../user-guide/character-mappings)
3. Review the [linguistic modules](./linguistic-modules)
4. Check the [Python API documentation](../user-guide/python-api)

The pipeline is designed to be modular, allowing easy extension and customization for specific use cases while maintaining the core transliteration accuracy that makes CaDaR unique.