# Architecture

> **Relevant source files**
> * [.github/copilot-instructions.md](../.github/copilot-instructions.md)
> * [.github/workflows/python-publish.yml](../.github/workflows/python-publish.yml)
> * [ArWikiCats/config.py](../ArWikiCats/config.py)
> * [ArWikiCats/jsons/population/pop_All_2018.json](../ArWikiCats/jsons/population/pop_All_2018.json)
> * [ArWikiCats/main_processers/main_resolve.py](../ArWikiCats/main_processers/main_resolve.py)
> * [CLAUDE.md](../CLAUDE.md)
> * [README.md](../README.md)
> * [changelog.md](../changelog.md)
> * [tests_require_fixes/test_papua_new_guinean.py](../tests_require_fixes/test_papua_new_guinean.py)
> * [tests_require_fixes/test_skip_data_all.py](../tests_require_fixes/test_skip_data_all.py)
> * [tests_require_fixes/text_to_fix.py](../tests_require_fixes/text_to_fix.py)

This page describes the overall system architecture of ArWikiCats. The system translates English Wikipedia category labels to Arabic through a five-layer architecture: a public API layer, a main resolution engine, a prioritized resolver chain, a template formatting engine, and a translation data layer.

For detailed information about specific subsystems, see the child pages: page 3.1 (Resolution Pipeline), page 3.2 (Data Architecture), and page 3.3 (Resolver Chain Priority System)

## System Architecture Overview

ArWikiCats implements a five-layer architecture that processes category labels through specialized resolvers and formatting engines:

**Diagram: Complete System Architecture**

```mermaid
flowchart TD

PublicAPI["resolve_arabic_category_label()<br/>batch_resolve_labels()<br/>resolve_label_ar()"]
MainResolver["resolve_label()<br/>main_resolve.py:33-94"]
EventProcessor["EventProcessor<br/>event_processing.py:30-180"]
Time["Time Resolvers<br/>labs_years.py"]
Pattern["Pattern Resolvers<br/>patterns_resolvers/"]
Jobs["Jobs Resolvers<br/>jobs_resolvers/"]
Sports["Sports Resolvers<br/>sports_resolvers/"]
Nats["Nationalities Resolvers<br/>nationalities_resolvers/"]
Countries["Countries Resolvers<br/>countries_names_resolvers/"]
Films["Films Resolvers<br/>resolve_films_bots/"]
Legacy["Legacy Resolvers<br/>legacy_bots/"]
FormatBase["FormatDataBase<br/>model_data_base.py"]
Multi["MultiDataFormatter<br/>model_multi_data.py"]
Year["YearFormatData<br/>model_data_time.py"]
Translations["translations/<br/>Python modules"]
JSONData["jsons/<br/>Raw JSON files"]
Geo["geo/<br/>Cities, Regions, Countries"]
JobsData["jobs/<br/>96,552 entries"]
SportsData["sports/<br/>431 sport keys"]
NatsData["nats/<br/>843 nationalities"]
FilmsData["tv/<br/>13,146 entries"]

PublicAPI --> MainResolver
MainResolver --> Time
Jobs --> FormatBase
Sports --> Multi
Nats --> FormatBase
Countries --> Year
FormatBase --> Translations
Multi --> Translations
Year --> Translations

subgraph Data ["Translation Data Layer"]
    Translations
    JSONData
    Geo
    JobsData
    SportsData
    NatsData
    FilmsData
    Translations --> JSONData
    JSONData --> Geo
    JSONData --> JobsData
    JSONData --> SportsData
    JSONData --> NatsData
    JSONData --> FilmsData
end

subgraph Formatting ["Template Formatting Engine"]
    FormatBase
    Multi
    Year
end

subgraph Chain ["Resolver Chain (Priority Order)"]
    Time
    Pattern
    Jobs
    Sports
    Nats
    Countries
    Films
    Legacy
    Time --> Pattern
    Pattern --> Jobs
    Jobs --> Sports
    Sports --> Nats
    Nats --> Countries
    Countries --> Films
    Films --> Legacy
end

subgraph Engine ["Resolution Engine"]
    MainResolver
    EventProcessor
    MainResolver --> EventProcessor
end

subgraph API ["External Interface"]
    PublicAPI
end
```

**Sources:** [ArWikiCats/main_processers/main_resolve.py L1-L106](../ArWikiCats/main_processers/main_resolve.py#L1-L106), [ArWikiCats/__init__.py L1-L42](../ArWikiCats/__init__.py#L1-L42), [ArWikiCats/event_processing.py L1-L180](../ArWikiCats/event_processing.py#L1-L180), [ArWikiCats/new_resolvers/__init__.py L1-L57](../ArWikiCats/new_resolvers/__init__.py#L1-L57)

## Layer 1: External Interface (Public API)

The public API provides entry points for category translation. All functions are exported from `ArWikiCats/__init__.py`.

**Diagram: Public API Functions**

```mermaid
flowchart TD

User["User Code"]
API1["resolve_arabic_category_label(category)<br/>Returns: str with 'تصنيف:' prefix"]
API2["resolve_label_ar(category, fix_label=True)<br/>Returns: str without prefix"]
API3["batch_resolve_labels(categories)<br/>Returns: BatchResult dataclass"]
API4["EventProcessor.process_single(category)<br/>Returns: CategoryResult dataclass"]
Core["resolve_label()<br/>main_resolve.py:33"]

User --> API1
User --> API2
User --> API3
User --> API4
API1 --> Core
API2 --> Core
API3 --> Core
API4 --> Core
```

| Function | Location | Purpose | Return Type |
| --- | --- | --- | --- |
| `resolve_label_ar()` | `main_resolve.py:96-99` | Translate single category without prefix | `str` |
| `resolve_arabic_category_label()` | `__init__.py:19-24` | Translate with "تصنيف:" prefix | `str` |
| `batch_resolve_labels()` | `__init__.py:27-32` | Process multiple categories | `BatchResult` |
| `EventProcessor.process_single()` | `event_processing.py:67-111` | Detailed processing with metadata | `CategoryResult` |

**Sources:** [ArWikiCats/__init__.py L1-L42](../ArWikiCats/__init__.py#L1-L42), [ArWikiCats/main_processers/main_resolve.py L96-L99](../ArWikiCats/main_processers/main_resolve.py#L96-L99), [ArWikiCats/event_processing.py L67-L111](../ArWikiCats/event_processing.py#L67-L111)

## Layer 2: Resolution Engine

The resolution engine coordinates category processing through `resolve_label()` in `main_resolve.py`. This function implements the waterfall resolver pattern with early exit on first match.

**Diagram: Resolution Engine Flow**

```mermaid
flowchart TD

Input["Input Category"]
Preprocess["Preprocessing<br/>change_cat()<br/>format_bots/change_cat.py"]
Filter["Filter Check<br/>filter_en.is_category_allowed()<br/>fix/filter_en.py"]
Empty["Return empty string"]
Patterns["all_patterns_resolvers()<br/>patterns_resolvers/init.py"]
FixLabel["fixlabel()<br/>fix/fixtitle.py"]
NewResolvers["all_new_resolvers()<br/>new_resolvers/init.py"]
University["university_resolver.resolve_university_category()<br/>sub_new_resolvers/university_resolver.py"]
LegacyResolvers["legacy_resolvers()<br/>legacy_bots/init.py"]
Cleanse["cleanse_category_label()<br/>fix/fixlists.py"]
Result["CategoryResult(en, ar, from_match)"]

Input --> Preprocess
Preprocess --> Filter
Filter --> Empty
Filter --> Patterns
Patterns --> FixLabel
Patterns --> NewResolvers
NewResolvers --> FixLabel
NewResolvers --> University
University --> FixLabel
University --> LegacyResolvers
LegacyResolvers --> FixLabel
FixLabel --> Cleanse
Cleanse --> Result
```

The `resolve_label()` function is cached with `@functools.lru_cache(maxsize=50000)` for performance. Each category is processed once and cached for subsequent requests.

**Sources:** [ArWikiCats/main_processers/main_resolve.py L33-L94](../ArWikiCats/main_processers/main_resolve.py#L33-L94), [ArWikiCats/format_bots/change_cat.py L1-L25](../ArWikiCats/format_bots/change_cat.py#L1-L25), [ArWikiCats/fix/filter_en.py L1-L50](../ArWikiCats/fix/filter_en.py#L1-L50), [ArWikiCats/fix/fixtitle.py L1-L150](../ArWikiCats/fix/fixtitle.py#L1-L150)

## Layer 3: Resolver Chain (Priority Order)

The resolver chain processes categories through specialized resolvers in priority order. This ordering prevents conflicts where multiple resolvers could match the same pattern.

**Diagram: Resolver Chain Priority**

```mermaid
flowchart TD

Start["all_patterns_resolvers()"]
P1["Time Patterns<br/>labs_years.py<br/>Priority: 1"]
P2["Pattern Resolvers<br/>country_time_pattern.py<br/>nat_males_pattern.py<br/>Priority: 2"]
Start2["all_new_resolvers()"]
N1["Jobs Resolvers<br/>jobs_resolvers/init.py<br/>main_jobs_resolvers()<br/>Priority: 3 (HIGHEST)"]
N2["Time+Jobs Resolvers<br/>time_and_jobs_resolvers/<br/>Priority: 4"]
N3["Sports Resolvers<br/>sports_resolvers/init.py<br/>main_sports_resolvers()<br/>Priority: 5"]
N4["Nationalities Resolvers<br/>nationalities_resolvers/init.py<br/>resolve_by_nats()<br/>Priority: 6"]
N5["Countries Resolvers<br/>countries_names_resolvers/init.py<br/>resolve_by_countries_names()<br/>Priority: 7"]
N6["Films Resolvers<br/>resolve_films_bots/init.py<br/>Priority: 8"]
N7["Relations Resolvers<br/>new_relations_resolvers.py<br/>Priority: 9"]
N8["Languages Resolvers<br/>languages_bot/resolver.py<br/>Priority: 10"]
Start3["legacy_resolvers()"]
L1["LegacyBotsResolver<br/>legacy_bots/init.py<br/>Priority: 11 (Fallback)"]

Start --> P1
P1 --> P2
Start2 --> N1
N1 --> N2
N2 --> N3
N3 --> N4
N4 --> N5
N5 --> N6
N6 --> N7
N7 --> N8
Start3 --> L1
```

The priority ordering is critical to prevent mismatches. For example:

* **Jobs before Sports**: "football manager" could match sports ("football") or jobs ("manager"). Jobs wins to correctly identify it as a management position.
* **Nationalities before Countries**: "Italy political leader" should resolve as nationality-based ("قادة سياسيون إيطاليون") not country-based ("قادة إيطاليا السياسيون")

**Sources:** [ArWikiCats/new_resolvers/__init__.py L1-L57](../ArWikiCats/new_resolvers/__init__.py#L1-L57), [ArWikiCats/patterns_resolvers/__init__.py L1-L30](../ArWikiCats/patterns_resolvers/__init__.py#L1-L30), [ArWikiCats/legacy_bots/__init__.py L29-L57](../ArWikiCats/legacy_bots/__init__.py#L29-L57)

## Layer 4: Template Formatting Engine

The template formatting engine implements pattern matching and placeholder replacement using a class hierarchy rooted in `FormatDataBase`.

**Diagram: Template Formatting Class Hierarchy**

```mermaid
flowchart TD

Base["FormatDataBase<br/>model_data_base.py<br/>Abstract base class<br/>Pattern compilation<br/>Placeholder replacement"]
Single1["FormatData<br/>model_data.py<br/>String → String<br/>'{sport}' → 'كرة القدم'"]
Single2["FormatDataV2<br/>model_data_v2.py<br/>Dict → Template<br/>'{sport}' → {'team': '...', 'jobs': '...'}"]
Single3["FormatDataFrom<br/>model_data_time.py<br/>Callback-based<br/>For temporal patterns"]
Multi1["MultiDataFormatterBase<br/>model_multi_data.py<br/>Combines two FormatData objects<br/>Example: Nationality + Sport"]
Multi2["MultiDataFormatterBaseV2<br/>model_multi_data.py<br/>Combines two FormatDataV2 objects<br/>Dict-based dual elements"]
Multi3["MultiDataFormatterBaseYear<br/>model_multi_data.py<br/>FormatData + YearFormatData<br/>Nationality + Year patterns"]
Multi4["MultiDataFormatterDataDouble<br/>model_multi_data_double.py<br/>Nationality + Genre pairs"]
Factory1["format_multi_data()<br/>multi_data.py"]
Factory2["format_multi_data_v2()<br/>multi_data.py"]
Factory3["format_year_country_data()<br/>data_with_time.py"]
Factory4["format_films_country_data()<br/>data_new_model.py"]

Base --> Single1
Base --> Single2
Base --> Single3
Single1 --> Multi1
Single2 --> Multi2
Single3 --> Multi3
Single1 --> Multi4
Multi1 --> Factory1
Multi2 --> Factory2
Multi3 --> Factory3
Multi4 --> Factory4
```

Each formatter provides:

* Pattern compilation: Regex patterns with case-insensitive matching
* Placeholder substitution: `{sport}` → `{sport_ar}` with Arabic translations
* Result caching: `@lru_cache` on search methods
* Label reordering: Ensures grammatically correct Arabic

**Example Flow:**

```yaml
Input: "british football players"
1. Match pattern: "{nat} {sport} players"
2. Extract: nat="british", sport="football"
3. Lookup: nat_ar="بريطانيون", sport_ar="كرة القدم"
4. Template: "لاعبو {sport} {nat}"
5. Output: "لاعبو كرة القدم بريطانيون"
```

**Sources:** [ArWikiCats/translations_formats/DataModel/model_data_base.py L1-L150](../ArWikiCats/translations_formats/DataModel/model_data_base.py#L1-L150), [ArWikiCats/translations_formats/DataModel/model_multi_data.py L1-L400](../ArWikiCats/translations_formats/DataModel/model_multi_data.py#L1-L400), [ArWikiCats/translations_formats/multi_data.py L1-L100](../ArWikiCats/translations_formats/multi_data.py#L1-L100)

## Layer 5: Translation Data Layer

The translation data layer provides domain-specific translation mappings. Raw JSON files are processed by Python aggregator modules that build lookup tables.

**Diagram: Translation Data Aggregation**

```mermaid
flowchart TD

J1["jobs.json<br/>Jobs_22.json"]
J2["Sports_Keys_New.json<br/>431 sports"]
J3["nationalities_data.json<br/>843 nationalities"]
J4["popopo.json<br/>P17_2_final_ll.json<br/>Geographic data"]
J5["Films_key_For_nat.json<br/>13,146 film entries"]
A1["jobs/Jobs.py<br/>_finalise_jobs_dataset()<br/>→ 96,552 jobs"]
A2["sports/Sport_key.py<br/>_build_tables()<br/>→ Sport records"]
A3["nats/Nationality.py<br/>build_lookup_tables()<br/>→ 18 lookup tables"]
A4["geo/labels_country.py<br/>_build_country_label_index()<br/>→ 68,981 entries"]
A5["tv/films_mslslat.py<br/>_build_gender_key_maps()<br/>→ Gender-specific maps"]
E1["translations/init.py<br/>Central export point"]
E2["build_data/init.py<br/>pf_keys2: 33,657 entries<br/>NEW_P17_FINAL: 68,981"]
S1["jobs_mens_data: 96,552<br/>jobs_womens_data<br/>Jobs_new: 1,304"]
S2["SPORT_KEY_RECORDS: 431<br/>SPORTS_KEYS_FOR_LABEL<br/>SPORT_JOB_VARIANTS: 571"]
S3["All_Nat: 843<br/>Nat_men, Nat_womens<br/>countries_from_nat: 287"]
S4["CITY_TRANSLATIONS_LOWER<br/>COUNTRY_LABEL_OVERRIDES<br/>US_STATES"]

J1 --> A1
J2 --> A2
J3 --> A3
J4 --> A4
J5 --> A5
A1 --> S1
A2 --> S2
A3 --> S3
A4 --> S4
S1 --> E1
S2 --> E1
S3 --> E1
S4 --> E1
S1 --> E2
S2 --> E2
S3 --> E2
S4 --> E2

subgraph Structures ["Data Structures"]
    S1
    S2
    S3
    S4
end

subgraph Exports ["Unified Exports"]
    E1
    E2
end

subgraph Aggregators ["Python Aggregators"]
    A1
    A2
    A3
    A4
    A5
end

subgraph Raw ["Raw JSON Sources"]
    J1
    J2
    J3
    J4
    J5
end
```

| Domain | Raw JSON Location | Python Module | Exported Data Structures | Size |
| --- | --- | --- | --- | --- |
| Jobs | `jsons/jobs/` | `translations/jobs/Jobs.py` | `jobs_mens_data`, `jobs_womens_data` | 96,552 + ~40,000 |
| Sports | `jsons/sports/` | `translations/sports/Sport_key.py` | `SPORT_KEY_RECORDS`, `SPORT_JOB_VARIANTS` | 431 + 571 |
| Nationalities | `jsons/nationalities/` | `translations/nats/Nationality.py` | `All_Nat`, 18 lookup tables | 843 entries |
| Geography | `jsons/geography/` | `translations/geo/labels_country.py` | `NEW_P17_FINAL`, `CITY_TRANSLATIONS` | 68,981 + 10,526 |
| Films/TV | `jsons/media/` | `translations/tv/films_mslslat.py` | `Films_key_For_nat`, gender-specific maps | 13,146 |
| Politics | `jsons/keys/` | `translations/politics/ministers_keys.py` | `ministers_keys` | ~94 |

The system maintains two export layers:

1. **Direct exports** via `translations/__init__.py` for immediate use by resolvers
2. **Aggregated exports** via `build_data/__init__.py` for combined datasets (e.g., `pf_keys2` with 33,657 entries)

**Sources:** [ArWikiCats/translations/__init__.py L1-L50](../ArWikiCats/translations/__init__.py#L1-L50), [ArWikiCats/translations/jobs/Jobs.py L1-L200](../ArWikiCats/translations/jobs/Jobs.py#L1-L200), [ArWikiCats/translations/sports/Sport_key.py L1-L150](../ArWikiCats/translations/sports/Sport_key.py#L1-L150), [ArWikiCats/translations/nats/Nationality.py L1-L300](../ArWikiCats/translations/nats/Nationality.py#L1-L300)

## Core Design Patterns

### Waterfall Resolver Pattern with Early Exit

The resolution engine implements a waterfall pattern where resolvers are tried in sequence until one succeeds:

```markdown
# From main_resolve.py:72-82
category_lab = all_patterns_resolvers(changed_cat)
from_match = bool(category_lab)

if not category_lab:
    category_lab = (
        ""
        or all_new_resolvers(changed_cat)
        or university_resolver.resolve_university_category(changed_cat)
        or legacy_resolvers(changed_cat)
        or ""
    )
```

This pattern ensures:

1. Unambiguous patterns resolve first (years, decades, centuries)
2. High-frequency patterns prioritized (jobs before sports)
3. Expensive lookups deferred (legacy resolvers last)
4. Deterministic resolution order prevents conflicts

**Sources:** [ArWikiCats/main_processers/main_resolve.py L72-L82](../ArWikiCats/main_processers/main_resolve.py#L72-L82)

### Caching Strategy

The system implements multi-level caching to optimize performance:

| Cache Level | Implementation | Location | Benefit |
| --- | --- | --- | --- |
| Function-level | `@functools.lru_cache(maxsize=50000)` | `main_resolve.py:32` | 50,000 category results cached |
| Resolver-level | `@functools.lru_cache(maxsize=None)` | Individual resolvers | Unlimited resolver-specific cache |
| Pattern-level | Compiled regex cache | `FormatDataBase` classes | Regex compilation avoided |
| Data-level | Module-level dictionaries | `translations/` modules | Instant lookup vs. file I/O |

**Example:**

```python
# From main_resolve.py:32-33
@functools.lru_cache(maxsize=50000)
def resolve_label(category: str, fix_label: bool = True) -> CategoryResult:
```

First call processes through entire resolver chain (~10-50ms). Subsequent calls retrieve from cache (<1ms)

**Sources:** [ArWikiCats/main_processers/main_resolve.py L32-L33](../ArWikiCats/main_processers/main_resolve.py#L32-L33), [changelog.md L277](../changelog.md#L277)

### Domain-Driven Data Organization

Translation data is organized by semantic domain rather than technical structure:

```markdown
translations/
├── jobs/                   # Occupations and professions
│   ├── Jobs.py            # jobs_mens_data, jobs_womens_data
│   ├── activists_jobs.py  # Activism-specific roles
│   └── religious_jobs.py  # Religious positions
├── sports/                 # Sports and teams
│   ├── Sport_key.py       # SPORT_KEY_RECORDS
│   └── sub_teams.py       # Team-specific mappings
├── nats/                   # Nationalities
│   └── Nationality.py     # All_Nat, 18 lookup tables
├── geo/                    # Geographic entities
│   ├── labels_country.py  # Country names
│   └── labels_city.py     # City names
└── tv/                     # Films and television
    └── films_mslslat.py   # Genre and nationality patterns
```

Each domain maintains:

* Python dictionaries for frequently-accessed data (loaded at import time)
* JSON files for large datasets in `jsons/` directory
* Aggregator functions that process raw JSON into structured lookups

**Sources:** [ArWikiCats/translations/__init__.py L1-L50](../ArWikiCats/translations/__init__.py#L1-L50), [README.md L90-L113](../README.md#L90-L113)

## Component Interaction Example

The following sequence diagram shows how a category flows through the system:

**Diagram: Resolution Sequence for "2010 British football players"**

```mermaid
sequenceDiagram
  participant User
  participant resolve_label
  participant all_patterns_resolvers
  participant all_new_resolvers
  participant main_sports_resolvers
  participant FormatData
  participant fixlabel

  User->>resolve_label: "2010 British football players"
  resolve_label->>resolve_label: change_cat() → lowercase, normalize
  resolve_label->>all_patterns_resolvers: "2010 british football players"
  all_patterns_resolvers->>all_patterns_resolvers: Check time patterns
  all_patterns_resolvers->>all_patterns_resolvers: Extract year: "2010"
  all_patterns_resolvers-->>resolve_label: No complete match (year found, but compound pattern)
  resolve_label->>all_new_resolvers: "2010 british football players"
  all_new_resolvers->>main_sports_resolvers: Try sports resolver
  main_sports_resolvers->>FormatData: search("{nat} {sport} players")
  FormatData->>FormatData: Match: nat="british", sport="football"
  FormatData->>FormatData: Lookup: nat_ar="بريطانيون", sport_ar="كرة القدم"
  FormatData->>FormatData: Template: "لاعبو {sport} {nat} عام {year}"
  FormatData-->>main_sports_resolvers: "لاعبو كرة القدم بريطانيون عام 2010"
  main_sports_resolvers-->>all_new_resolvers: Result found
  all_new_resolvers-->>resolve_label: category_lab
  resolve_label->>fixlabel: "لاعبو كرة القدم بريطانيون عام 2010"
  fixlabel->>fixlabel: Apply Arabic grammar rules
  fixlabel-->>resolve_label: "لاعبو كرة القدم بريطانيون عام 2010"
  resolve_label->>resolve_label: cleanse_category_label()
  resolve_label-->>User: CategoryResult(ar="لاعبو كرة القدم بريطانيون عام 2010")
```

**Sources:** [ArWikiCats/main_processers/main_resolve.py L33-L94](../ArWikiCats/main_processers/main_resolve.py#L33-L94), [ArWikiCats/new_resolvers/__init__.py L1-L57](../ArWikiCats/new_resolvers/__init__.py#L1-L57), [ArWikiCats/translations_formats/DataModel/model_data_base.py L1-L150](../ArWikiCats/translations_formats/DataModel/model_data_base.py#L1-L150)

## Data Flow Pipeline

The translation pipeline processes categories through six stages:

**Diagram: Complete Data Flow Pipeline**

```mermaid
flowchart TD

S1["Stage 1:<br/>Raw Input<br/>Category: String"]
S2["Stage 2:<br/>Normalization<br/>change_cat()<br/>format_bots/change_cat.py"]
S3["Stage 3:<br/>Filtering<br/>filter_en.is_category_allowed()<br/>fix/filter_en.py"]
S4["Stage 4:<br/>Pattern Resolution<br/>all_patterns_resolvers()<br/>all_new_resolvers()<br/>legacy_resolvers()"]
S5["Stage 5:<br/>Arabic Grammar<br/>fixlabel()<br/>fix/fixtitle.py"]
S6["Stage 6:<br/>Cleansing<br/>cleanse_category_label()<br/>fix/fixlists.py"]
Output["Output:<br/>CategoryResult<br/>en, ar, from_match"]

S1 --> S2
S2 --> S3
S3 --> S4
S4 --> S5
S5 --> S6
S6 --> Output
```

### Stage 1: Raw Input

Input categories may include:

* Wikipedia category syntax: "Category:British footballers"
* Plain text: "British footballers"
* With year prefixes: "2010 British footballers"
* Complex patterns: "21st-century British football managers"

### Stage 2: Normalization

The `change_cat()` function in [ArWikiCats/format_bots/change_cat.py L1-L25](../ArWikiCats/format_bots/change_cat.py#L1-L25)

 applies:

* Lowercase conversion
* Underscore to space: "British_footballers" → "british footballers"
* "Category:" prefix removal
* Whitespace normalization
* Key mappings: "labor" → "labour", "womens" → "female"

### Stage 3: Filtering

The `filter_en.is_category_allowed()` function in [ArWikiCats/fix/filter_en.py L1-L50](../ArWikiCats/fix/filter_en.py#L1-L50)

 rejects categories containing blocked terms or patterns. Filtered categories return an empty Arabic label.

### Stage 4: Pattern Resolution

This stage attempts resolvers in priority order:

1. `all_patterns_resolvers()` - Time patterns and complex patterns
2. `all_new_resolvers()` - Domain-specific resolvers (jobs, sports, nats, countries, films)
3. `university_resolver.resolve_university_category()` - University-specific patterns
4. `legacy_resolvers()` - Legacy bot fallback

Each resolver returns on first match (early exit pattern)

### Stage 5: Arabic Grammar

The `fixlabel()` function in [ArWikiCats/fix/fixtitle.py L1-L150](../ArWikiCats/fix/fixtitle.py#L1-L150)

 applies:

* Article agreement: Proper handling of "ال" prefix
* Preposition insertion: Add "في" or "من" based on English separators
* Duplicate removal: Prevent "في في" patterns
* Gender-specific adjustments
* Final formatting cleanup

### Stage 6: Cleansing

The `cleanse_category_label()` function in [ArWikiCats/fix/fixlists.py L1-L100](../ArWikiCats/fix/fixlists.py#L1-L100)

 performs final cleanup:

* Remove trailing colons
* Normalize whitespace
* Remove empty translations
* Ensure consistent output format

**Sources:** [ArWikiCats/main_processers/main_resolve.py L33-L94](../ArWikiCats/main_processers/main_resolve.py#L33-L94), [ArWikiCats/format_bots/change_cat.py L1-L25](../ArWikiCats/format_bots/change_cat.py#L1-L25), [ArWikiCats/fix/filter_en.py L1-L50](../ArWikiCats/fix/filter_en.py#L1-L50), [ArWikiCats/fix/fixtitle.py L1-L150](../ArWikiCats/fix/fixtitle.py#L1-L150), [ArWikiCats/fix/fixlists.py L1-L100](../ArWikiCats/fix/fixlists.py#L1-L100)

## Module Organization

The codebase exhibits a **modular architecture** with clear separation of concerns:

```mermaid
flowchart TD

INIT["init.py<br/>Exports"]
MAIN["main_processers/<br/>main_resolve.py<br/>event_lab_bot.py"]
NEW["new_resolvers/<br/>Modern Architecture"]
LEGACY["legacy_bots/<br/>Original Bots"]
TIME["time_resolvers/<br/>Temporal Patterns"]
PATTERNS["patterns_resolvers/<br/>Complex Patterns"]
FIX["fix/<br/>Arabic Grammar"]
FORMAT_BOTS["format_bots/<br/>Text Processing"]
TRANS["translations/<br/>Domain Dicts"]
FORMATS["translations_formats/<br/>Template System"]
JSONS["jsons/<br/>JSON Data"]
CONFIG["config.py"]
HELPS["helps/<br/>Logger, Memory"]
UTILS["utils/"]

INIT --> MAIN
MAIN --> NEW
MAIN --> LEGACY
MAIN --> TIME
MAIN --> PATTERNS
MAIN --> FIX
NEW --> TRANS
NEW --> FORMATS
LEGACY --> TRANS
MAIN --> CONFIG
MAIN --> FORMAT_BOTS

subgraph Infrastructure ["Infrastructure"]
    CONFIG
    HELPS
    UTILS
end

subgraph subGraph4 ["Data & Formatting"]
    TRANS
    FORMATS
    JSONS
    FORMATS --> JSONS
    TRANS --> JSONS
end

subgraph subGraph3 ["Support Systems"]
    TIME
    PATTERNS
    FIX
    FORMAT_BOTS
end

subgraph subGraph2 ["Resolver Modules"]
    NEW
    LEGACY
end

subgraph subGraph1 ["Core Processing"]
    MAIN
end

subgraph subGraph0 ["Public Interface"]
    INIT
end
```

**Sources:** [README.md L349-L445](../README.md#L349-L445)

### Directory Structure and Responsibilities

| Directory | Responsibility | Key Files | Lines of Code Est. |
| --- | --- | --- | --- |
| `main_processers/` | Orchestration and main entry point | `main_resolve.py`, `event_lab_bot.py` | ~700 |
| `new_resolvers/` | Modern resolver implementations | `reslove_all.py`, domain-specific modules | ~3,000 |
| `legacy_bots/` | Original bot implementations | `ma_bots/`, `make_bots/` | ~5,000 |
| `translations/` | Translation data (Python dicts) | `geo/`, `jobs/`, `nats/`, `sports/`, `tv/` | ~2,000 |
| `translations_formats/` | Template formatting framework | `DataModel/`, `multi_data.py` | ~1,500 |
| `time_resolvers/` | Temporal pattern handling | `labs_years.py`, `time_to_arabic.py` | ~800 |
| `patterns_resolvers/` | Complex pattern matchers | `country_time_pattern.py`, `nat_men_pattern.py` | ~600 |
| `fix/` | Arabic text corrections | `fixtitle.py`, `fixlists.py` | ~400 |
| `jsons/` | JSON data files | 8 domain subdirectories | N/A (data) |
| `helps/` | Infrastructure utilities | `log.py`, `memory.py` | ~300 |

**Sources:** [README.md L349-L445](../README.md#L349-L445), [changelog.md L1-L750](../changelog.md#L1-L750)

## Architectural Evolution: Legacy vs. Modern Resolvers

The codebase contains two resolver architectures that coexist:

**Legacy Architecture** (`legacy_bots/`):

* Located in [ArWikiCats/legacy_bots/](../ArWikiCats/legacy_bots/)
* Organized around bot scripts with mixed concerns
* Direct dictionary lookups without abstraction
* Monolithic functions handling multiple patterns
* Called as last fallback via `legacy_resolvers()` function
* Recently refactored from `RESOLVER_PIPELINE` list to `LegacyBotsResolver` class

**Modern Architecture** (`new_resolvers/`):

* Located in [ArWikiCats/new_resolvers/](../ArWikiCats/new_resolvers/)
* Domain-specific modules (jobs, sports, nationalities, countries, films)
* Template-based formatting via `FormatData` classes
* Composition over inheritance
* Clear separation: resolver logic, data access, formatting
* Called via `all_new_resolvers()` function with priority ordering

**Diagram: Dual Architecture**

```mermaid
flowchart TD

Input["Category Input"]
Modern["all_new_resolvers()<br/>new_resolvers/init.py"]
Output["Arabic Label"]
Legacy["legacy_resolvers()<br/>legacy_bots/init.py"]
Jobs["jobs_resolvers/<br/>Template-based<br/>FormatData"]
Sports["sports_resolvers/<br/>Multi-element<br/>MultiDataFormatter"]
Nats["nationalities_resolvers/<br/>FormatDataV2<br/>Gender-aware"]
LegacyBots["LegacyBotsResolver<br/>Class-based refactor"]
Univ["_resolve_university()"]
Country["_resolve_country_and_event()"]
Years["_resolve_years()"]
General["_resolve_general()"]

Input --> Modern
Modern --> Output
Modern --> Legacy
Legacy --> Output

subgraph LegacyArch ["Legacy Architecture"]
    Legacy
    LegacyBots
    Univ
    Country
    Years
    General
    Legacy --> LegacyBots
    LegacyBots --> Univ
    LegacyBots --> Country
    LegacyBots --> Years
    LegacyBots --> General
end

subgraph ModernArch ["Modern Architecture"]
    Modern
    Jobs
    Sports
    Nats
    Modern --> Jobs
    Modern --> Sports
    Modern --> Nats
end
```

The legacy architecture was recently refactored (changelog.md:170-200) from a list-based `RESOLVER_PIPELINE` to a class-based `LegacyBotsResolver` for better maintainability, but both architectures remain active to preserve existing translation coverage.

**Sources:** [ArWikiCats/new_resolvers/__init__.py L1-L57](../ArWikiCats/new_resolvers/__init__.py#L1-L57), [ArWikiCats/legacy_bots/__init__.py L1-L200](../ArWikiCats/legacy_bots/__init__.py#L1-L200), [changelog.md L170-L200](../changelog.md#L170-L200)

## Configuration System

The configuration system uses frozen dataclasses that read from environment variables and command-line arguments.

**Diagram: Configuration Architecture**

```mermaid
flowchart TD

Env["Environment Variables<br/>SAVE_DATA_PATH"]
Args["sys.argv<br/>Command-line flags"]
OneReq["one_req(name)<br/>config.py:14-16"]
AppConfig["AppConfig<br/>@dataclass frozen=True<br/>save_data_path: str"]
Config["Config<br/>@dataclass frozen=True<br/>app: AppConfig"]
Settings["settings: Config<br/>Global instance"]
AppSettings["app_settings: AppConfig<br/>Convenience export"]

Env --> OneReq
Args --> OneReq
OneReq --> AppConfig
AppConfig --> Config
Config --> Settings
Config --> AppSettings
```

**Configuration Options:**

| Setting | Type | Default | Source | Purpose |
| --- | --- | --- | --- | --- |
| `SAVE_DATA_PATH` | str | `""` | Environment variable | Path to save temporary data files |

**Usage:**

```python
from ArWikiCats.config import app_settings

# Access configuration
if app_settings.save_data_path:
    save_to_path(app_settings.save_data_path)
```

The configuration is immutable (`frozen=True`) to prevent accidental modification during runtime.

**Sources:** [ArWikiCats/config.py L1-L52](../ArWikiCats/config.py#L1-L52)

## Performance Characteristics

### Caching Strategy

The system achieves sub-second processing through multi-level caching:

```python
# From main_resolve.py:32-33
@functools.lru_cache(maxsize=50000)
def resolve_label(category: str, fix_label: bool = True) -> CategoryResult:
    """Cached for entire program lifetime."""
```

**Performance Metrics:**

* First call: 10-50ms (full resolver chain)
* Cached call: <1ms (dictionary lookup)
* Batch processing: 5,000 categories in ~5-10 seconds with warm cache

The 50,000 entry cache limit accommodates large-scale batch processing workflows.

### Memory Footprint

Memory consumption was optimized from 2GB to <100MB through:

* Module-level data loading (load once at import time)
* Lazy JSON loading for large datasets
* Bounded LRU caches (`maxsize=50000` on main resolver)
* Frozen dataclasses instead of mutable dictionaries

**Sources:** [ArWikiCats/main_processers/main_resolve.py L32-L33](../ArWikiCats/main_processers/main_resolve.py#L32-L33), [README.md L499-L502](../README.md#L499-L502)

## Architectural Trade-offs

### Resolver Chain Depth vs. Accuracy

The system prioritizes **accuracy over speed** by attempting up to 7 different resolvers before falling back to general translation. This ensures high-quality results but increases latency for rare patterns.

**Mitigation:** Aggressive caching means expensive resolution happens only once per unique category.

### Python Dicts vs. Database

Translation data lives in **Python dictionaries and JSON files** rather than a database. This choice provides:

**Advantages:**

* Zero database setup/dependencies
* Instant module-level loading
* Version control friendly (git diff works)
* Fast in-memory lookups

**Disadvantages:**

* Higher memory baseline (~100MB)
* No ACID guarantees for data updates
* Manual metadata tracking via `data_len.json`

**Sources:** [README.md L407-L441](../README.md#L407-L441)

### Legacy Bot Preservation

The decision to keep `legacy_bots/` while building `new_resolvers/` creates **technical debt** but ensures:

* No regression in coverage for existing categories
* Gradual migration path
* Continued functionality during refactoring

**Sources:** [changelog.md L273-L289](../changelog.md#L273-L289), [changelog.md L505-L514](../changelog.md#L505-L514)

---

This architecture enables ArWikiCats to process over **28,500 test cases** covering diverse category types while maintaining extensibility for new translation domains. The three-tier design, waterfall resolver pattern, and aggressive caching combine to deliver both accuracy and performance for Wikipedia category translation workflows.
