# Character Mappings in CaDaR

This document describes all character mappings and special handling in CaDaR for accurate bidirectional transliteration between Arabic and Latin scripts.

## Basic Character Mappings

### Consonants

| Latin | Arabic | Notes |
|--------|---------|-------|
| b | ب |  |
| t | ت, ط | Context-dependent (see emphatic handling) |
| th | ث |  |
| j | ج |  |
| h | ح, ه | Context-dependent (see emphatic handling) |
| kh | خ |  |
| d | د |  |
| dh | ذ |  |
| r | ر |  |
| z | ز |  |
| s | س, ص | Context-dependent (see emphatic handling) |
| sh | ش |  |
| 9 | ق |  |
| k | ك |  |
| l | ل |  |
| m | م |  |
| n | ن |  |
| w | و |  |
| y | ي |  |
| 3 | ع |  |
| 7 | ح |  |
| Dr | ض | Darija verbs (see special sequences) |

### Vowels

| Latin | Arabic | Notes |
|--------|---------|-------|
| a | ا |  |
| aa | ا | Long vowel |
| e | ى | Short vowel, often omitted in middle position |
| ee | ي | Long vowel |
| i | ي | Short vowel, often omitted in middle position |
| ii | ي | Long vowel |
| o | و | Short vowel |
| oo | و | Long vowel |
| u | و | Short vowel |
| uu | و | Long vowel |

### Special Characters

| Latin | Arabic | Notes |
|--------|---------|-------|
| ch | ش | Common Darija spelling |
| 8 | غ, ه | Context-dependent (default: غ) |
| gh | غ |  |
| f | ف |  |
| q | ق | Alternative to 9 |

### Punctuation Mapping

Bidirectional punctuation mapping preserves meaning across scripts:

| Latin | Arabic | Notes |
|--------|---------|-------|
| ? | ؟ | Question mark |
| , | ، | Comma |
| ; | ؛ | Semicolon |

## Special Cases and Context-Dependent Rules

### 1. Emphatic Consonants

CaDaR uses context-aware rules to determine when to use emphatic consonants:

- **T** becomes **ط** for words in emphatic lexicon
- **T** becomes **ت** for words with aspect markers (ta-, tay-, ka-)
- **S** becomes **ص** for words with emphatic roots
- **S** becomes **س** for function words and loanwords

Examples:
- `tbi3` → `تبيع` (regular T)
- `tlemm` → `طلم` (emphatic T)
- `sbah` → `صباح` (emphatic S)
- `sba7` → `صباح` (emphatic S)

### 2. Number 8 Context Handling

The digit "8" has context-dependent mapping:

- Default: `8` → `غ` (ghayn)
- In word contexts: `8` → `ه` (ha)

Examples:
- `8er` → `غير` (default)
- `b8` → `به` (word context)

### 3. Definite Article "l"

The definite article "l" is context-aware:

- When followed by a word: `l` → `ال` (definite article)
- As a preposition: `l` → `ل` (preposition)

Examples:
- `lkol` → `لكل` (definite article)
- `l 3arbi` → `ل العربي` (definite article with space)
- `3la` → `على` (preposition)

### 4. Clitic Handling

Clitic prefixes are properly attached without spaces:

- `fl` → `فال` (not `ف ل`)
- `bl` → `بال` (not `ب ل`)
- `ll` → `لل` (not `ل ل`)

### 5. Smart Vowel Handling

CaDaR implements intelligent vowel processing based on position:

**Darija Mode (default):**
- Short 'a', 'o', 'u' are written in middle positions for clarity
- Short 'e', 'i' are often skipped in middle position (absorbed)
- Long vowels are always preserved

Examples:
- `salam` → `سالام` (writes middle 'a')
- `imken` → `يمكن` (skips middle 'e')
- `daba` → `دابا` (writes middle 'a')
- `salaam` → `سالام` (long aa preserved)

## Linguistic Patterns

### Common Darija Words

| Latin | Arabic | Usage |
|--------|---------|-------|
| wakha | واخا | "okay" |
| bzaf | بزاف | "a lot" |
| shwiya | شوية | "a little" |
| smi7 | سميح | "excuse me" |
| 3likom | عليكم | "to you" |
| kayn | كاين | "there is" |

### Verb Patterns

Darija uses simplified verb forms:

| Latin | Arabic | Notes |
|--------|---------|-------|
| kan3mel | كنعمل | Present continuous |
| kanmchi | كامشي | Present continuous |
| n3ml | نعمل | Present |
| mchi | مشي | Past |

### Function Words

Function words follow specific patterns:

| Latin | Arabic | Type |
|--------|---------|-------|
| mn | من | Preposition |
| 3la | على | Preposition |
| bl | بال | Preposition with article |
| l | ال, ل | Article or preposition |
| dyal | ديال | Possessive |

## Implementation Details

### Character Classification

Characters are classified into types for processing:

1. **Consonants**: Regular and emphatic variants
2. **Vowels**: Short and long variants
3. **Numbers**: Arabic numerals with special mappings
4. **Punctuation**: Script-specific and universal marks
5. **Whitespace**: Word boundaries

### Processing Order

1. **Script Detection**: Identify input script type
2. **Normalization**: Clean diacritics and standardize forms
3. **Tokenization**: Split into words and clitics
4. **Character Mapping**: Apply context-aware rules
5. **Validation**: Ensure output validity
6. **Post-processing**: Handle spacing and punctuation

### Performance Optimizations

- **Hybrid Architecture**: Word-level context in character-level ICR
- **Caching**: Linguistic properties extracted once per word
- **Context Awareness**: Efficient lookup tables for common patterns

## Testing Coverage

The character mappings are comprehensively tested with:

- **Basic mappings**: 45+ direct character conversions
- **Context tests**: 20+ context-dependent cases
- **Pattern tests**: 30+ linguistic pattern conversions
- **Edge cases**: Empty input, mixed scripts, special characters

## Future Enhancements

Planned improvements to character handling:

1. **More dialects**: Support for other Darija dialects
2. **Context learning**: Machine learning for better context detection
3. **Emoji handling**: Preserve emoji in transliteration
4. **Specialized domains**: Domain-specific terminology support

## Contributing

To contribute new character mappings or rules:

1. Identify the linguistic pattern
2. Add test cases in the appropriate test module
3. Implement the mapping in `src/stages/script_generation.rs`
4. Update this documentation
5. Ensure all tests pass

For more details on implementation, see the [Architecture Guide](../guide/architecture.md).