# Translation Workflow

## Pipeline Overview

The AiNiee-Next translation pipeline follows a sophisticated multi-stage process to handle various text formats while preserving formatting, placeholders, and special content.

### Core Pipeline Stages

```
1. Text Input
   ↓
2. TextProcessor.normalize_text (Plugin Event)
   ↓
3. TextProcessor.replace_all (Pre-translation)
   ↓
4. Plugin Event: preproces_text
   ↓
5. PromptBuilder.build_prompt
   ↓
6. LLMRequester.request_translation
   ↓
7. ResponseExtractor.extract_translation
   ↓
8. ResponseChecker.validate_response
   ↓
9. TextProcessor.restore_all (Post-translation)
   ↓
10. Plugin Event: postprocess_text
   ↓
11. Output Translation
```

## Detailed Stage Analysis

### Stage 1: Text Input
**Component**: `FileReader` implementations
**Location**: `ModuleFolders/Domain/FileReader/`

- Reads source files (TXT, JSON, SRT, ASS, PO, etc.)
- Extracts translatable text while preserving structure
- Creates `CacheItem` objects with `source_text`

### Stage 2: Text Normalization (Plugin Event)
**Event**: `normalize_text`
**Purpose**: Preprocessing before pipeline

```python
# Plugin can modify source_text_dict
plugin_manager.broadcast_event("normalize_text", config, source_text_dict)
```

**Use Cases**:
- Unicode normalization
- Encoding fixes
- Custom text transformations

### Stage 3: Pre-translation Processing
**Component**: `TextProcessor.replace_all()`
**Location**: `ModuleFolders/Domain/TextProcessor/TextProcessor.py`

**Operations**:

1. **Line Ending Normalization**
   - Unifies `\r\n`, `\r`, `\n`, `<br>` to `\n`
   - Stores original line ending types for restoration

2. **Pre-translation Rules**
   - Applies regex replacements from `config.pre_translation_data`
   - Example: Replace special quotes, normalize punctuation

3. **Code/Placeholder Extraction**
   - Identifies and extracts code patterns, placeholders
   - Replaces with safe placeholders: `{p1}`, `{p2}`, etc.
   - Stores mapping in `placeholder_order`

4. **Whitespace Handling**
   - Extracts leading/trailing whitespace
   - Stores in `affix_whitespace_storage`
   - Prevents whitespace loss during translation

5. **Prefix/Suffix Extraction**
   - Identifies non-translatable prefixes/suffixes
   - Example: `[CharacterName]` in dialogue
   - Stores in `prefix_codes`, `suffix_codes`

**Output**:
```python
{
    "processed_text_dict": {...},  # Clean text ready for translation
    "prefix_codes": {...},         # Extracted prefixes
    "suffix_codes": {...},         # Extracted suffixes
    "placeholder_order": {...},    # Placeholder mappings
    "affix_whitespace_storage": {...}  # Whitespace data
}
```

### Stage 4: Plugin Preprocessing Event
**Event**: `preproces_text`
**Purpose**: Plugin can modify processed text before translation

```python
plugin_manager.broadcast_event(
    "preproces_text",
    config,
    source_text_dict,
    prompt_text_dict
)
```

### Stage 5: Prompt Building
**Component**: `PromptBuilder` variants
**Location**: `ModuleFolders/Domain/PromptBuilder/`

**Prompt Builder Types**:

1. **PromptBuilder** (Standard)
   - Template: `Resource/Prompt/Translate/common_system_*.txt`
   - Features: Basic translation with formatting preservation

2. **PromptBuilderLocal**
   - Template: `Resource/Prompt/Local/local_system_*.txt`
   - Features: Optimized for local/offline models

3. **PromptBuilderSakura**
   - Template: `Resource/Prompt/Sakura/sakura_system_*.txt`
   - Features: Specialized Japanese→Chinese model

4. **PromptBuilderPolishing**
   - Template: `Resource/Prompt/Polishing/*.txt`
   - Features: Text refinement and polishing

**Prompt Structure**:
```
[System Prompt]
- Translation instructions
- Format preservation rules
- Placeholder handling guidelines
- Target language requirements

[User Message]
- Source text with numbering: 【1】Text line 1
- Context from previous translations
- Glossary terms if available
```

**Placeholders**:
- `{source_language}`: Source language name
- `{target_language}`: Target language name
- `{glossary}`: Formatted glossary entries

### Stage 6: LLM Request
**Component**: `LLMRequester`
**Location**: `ModuleFolders/Infrastructure/LLMRequester/LLMRequester.py`

**Request Process**:
1. Build request payload from prompt
2. Apply rate limiting via `RequestLimiter`
3. Send to configured API endpoint
4. Handle retries and errors
5. Return raw response

**Supported APIs**:
- OpenAI-compatible APIs
- Anthropic Claude
- Local LLM servers
- Custom endpoints

**Error Handling**:
- Automatic retry with exponential backoff
- API failover support
- Error threshold detection
- Consecutive error tracking

### Stage 7: Response Extraction
**Component**: `ResponseExtractor`
**Location**: `ModuleFolders/Domain/ResponseExtractor/ResponseExtractor.py`

**Extraction Steps**:

1. **Format Detection**
   - Detects output format: `<textarea>`, numbered list, plain text
   - Handles different LLM output styles

2. **Line Extraction**
   - Parses numbered format: `1.Translation text`
   - Parses bracket format: `【1】Translation text`
   - Extracts from textarea tags

3. **Line Count Validation**
   - Ensures output line count matches input
   - Detects merged or split lines

**Output**: Dictionary mapping line numbers to translations

### Stage 8: Response Validation
**Component**: `ResponseChecker`
**Location**: `ModuleFolders/Domain/ResponseChecker/ResponseChecker.py`

**Validation Checks**:

1. **Base Checks** (`BaseChecks`)
   - Length validation (min/max ratio)
   - Line count matching
   - Empty translation detection

2. **Advanced Checks** (`AdvancedChecks`)
   - Language detection (is output in target language?)
   - Placeholder preservation (all `{pN}` present?)
   - Format marker preservation

3. **Custom Checks**
   - Domain-specific validation
   - Plugin-injected checks

**Failure Handling**:
- Mark translation as failed in cache
- Trigger retry if configured
- Log error details
- Continue with next batch

### Stage 9: Post-translation Restoration
**Component**: `TextProcessor.restore_all()`
**Location**: `ModuleFolders/Domain/TextProcessor/TextProcessor.py`

**Restoration Order**:

1. **Restore Placeholders**
   - Replace `{p1}`, `{p2}` back to original content
   - Use `placeholder_order` mapping

2. **Restore Whitespace**
   - Re-add leading/trailing whitespace
   - Use `affix_whitespace_storage`

3. **Restore Prefix/Suffix**
   - Re-attach extracted prefixes/suffixes
   - Use `prefix_codes`, `suffix_codes`

4. **Post-translation Rules**
   - Apply regex replacements from `config.post_translation_data`
   - Example: Fix quote styles, adjust punctuation

5. **Restore Line Endings**
   - Convert `\n` back to original types
   - Restore `<br>` tags if needed

### Stage 10: Plugin Postprocessing Event
**Event**: `postprocess_text`
**Purpose**: Final modifications after translation

```python
plugin_manager.broadcast_event(
    "postprocess_text",
    config,
    translation_dict
)
```

**Use Cases**:
- Bilingual output generation
- Quality scoring
- Translation memory updates

### Stage 11: Output Writing
**Component**: `FileWriter` implementations
**Location**: `ModuleFolders/Domain/FileOutputer/`

- Writes translations back to original format
- Preserves file structure and metadata
- Updates cache with final translations

## Batch Processing

### Batch Assembly
**Component**: `TaskExecutor._assemble_translation_batch()`

```python
batch = {
    "items": [CacheItem, ...],           # Translation units
    "previous_items": [CacheItem, ...],  # Context lines
    "source_context_items": [...]        # RAG context
}
```

### Batch Size Management
- **Default**: 10-20 lines per batch
- **Factors**: Token limits, content complexity, model capacity
- **Adjustment**: Via `batch_size` config parameter

### Concurrency Control
- **Thread Pool**: `concurrent.futures.ThreadPoolExecutor`
- **Max Concurrent**: Configured via `max_concurrent_requests`
- **Rate Limiting**: Per-API rate limit enforcement

## Cache Integration

### Cache Structure
**Location**: `Resource/cache/`

```
cache/
  project_name/
    metadata.json           # Project metadata
    file1.json.gz          # Cached translations
    file2.json.gz
    statistics.json        # Translation statistics
```

### Cache States
- **UNTRANSLATED**: New content, not yet translated
- **TRANSLATED**: Successfully translated
- **FAILED**: Translation failed validation
- **SKIPPED**: Manually skipped

### Cache Benefits
- Resume interrupted translations
- Avoid re-translating unchanged content
- Track translation history
- Support incremental updates

## Plugin Integration Points

### Available Events

1. **normalize_text**: Pre-normalization
   - Input: `config`, `source_text_dict`
   - Output: Modified `source_text_dict`

2. **preproces_text**: After preprocessing
   - Input: `config`, `source_text_dict`, `prompt_text_dict`
   - Output: Modified prompt text

3. **postprocess_text**: After translation
   - Input: `config`, `translation_dict`
   - Output: Modified translation

4. **build_rag_context**: Context enhancement
   - Input: `config`, `rag_context_data`
   - Output: RAG context string

5. **text_filter**: Content filtering
   - Input: `config`, `text_dict`
   - Output: Filtered text

### Plugin Examples

**Glossary Plugin**:
```python
def on_preproces_text(self, config, source_dict, prompt_dict):
    # Inject glossary terms into prompt
    glossary = self.load_glossary()
    # Highlight terms in source text
    for key, text in prompt_dict.items():
        for term in glossary:
            if term in text:
                # Mark for translator attention
                prompt_dict[key] = text.replace(term, f"【{term}】")
```

**Bilingual Plugin**:
```python
def on_postprocess_text(self, config, translation_dict):
    # Create bilingual output
    for key, trans in translation_dict.items():
        source = self.source_cache[key]
        translation_dict[key] = f"{source}\n{trans}"
```

## Performance Optimization

### Token Management
- **Token Counting**: `ModuleFolders/Infrastructure/Tokener/Tokener.py`
- **Optimal Batch Size**: Balance between context and token limits
- **Context Window**: Limit previous lines to avoid overflow

### Concurrency Tuning
- **API Limits**: Respect rate limits (requests per minute)
- **Thread Count**: Match to available API connections
- **Queue Depth**: Balance memory vs throughput

### Caching Strategy
- **Cache Hits**: Reuse existing translations
- **Incremental**: Only translate new/changed content
- **Validation**: Check cache validity before use

## Error Handling and Recovery

### Error Types

1. **API Errors**
   - Rate limit exceeded
   - Invalid API key
   - Network timeout
   - Model overloaded

2. **Validation Errors**
   - Line count mismatch
   - Missing placeholders
   - Wrong target language
   - Empty translations

3. **Processing Errors**
   - File read/write failures
   - Encoding issues
   - Format parsing errors

### Recovery Strategies

1. **Retry Logic**
   - Exponential backoff
   - Maximum retry attempts
   - Failover to backup API

2. **Graceful Degradation**
   - Skip failed batches
   - Continue with partial results
   - Manual intervention hooks

3. **User Notification**
   - Progress updates via events
   - Error logging
   - Status dashboard (WebServer)

## Monitoring and Logging

### Progress Tracking
**Events**:
- `TASK_START`: Translation begins
- `TASK_PROGRESS`: Batch completed
- `TASK_STOP`: Translation finished/stopped
- `SYSTEM_STATUS_UPDATE`: Status changes

### Logging Levels
- **INFO**: Normal operations, batch completions
- **WARNING**: Recoverable errors, retries
- **ERROR**: Failed translations, API errors
- **DEBUG**: Detailed pipeline steps

### Statistics
**Location**: `CacheProjectStatistics`
- Total lines translated
- Success/failure rates
- API usage metrics
- Time per batch

## Summary

The translation workflow in AiNiee-Next is a robust, extensible pipeline designed for:

- **Flexibility**: Multiple formats, languages, and models
- **Quality**: Comprehensive validation and checking
- **Reliability**: Error handling, retries, and failover
- **Extensibility**: Plugin hooks throughout the pipeline
- **Performance**: Concurrency, caching, and optimization

Understanding this workflow enables effective troubleshooting, optimization, and customization of translation tasks.
