# Overview

> **Relevant source files**
> * [.github/copilot-instructions.md](../.github/copilot-instructions.md)
> * [.github/workflows/python-publish.yml](../.github/workflows/python-publish.yml)
> * [ArWikiCats/config.py](../ArWikiCats/config.py)
> * [ArWikiCats/jsons/population/pop_All_2018.json](../ArWikiCats/jsons/population/pop_All_2018.json)
> * [ArWikiCats/main_processers/main_resolve.py](../ArWikiCats/main_processers/main_resolve.py)
> * [CLAUDE.md](../CLAUDE.md)
> * [README.md](../README.md)
> * [changelog.md](../changelog.md)
> * [tests_require_fixes/test_papua_new_guinean.py](../tests_require_fixes/test_papua_new_guinean.py)
> * [tests_require_fixes/test_skip_data_all.py](../tests_require_fixes/test_skip_data_all.py)
> * [tests_require_fixes/text_to_fix.py](../tests_require_fixes/text_to_fix.py)

## Purpose and Scope

ArWikiCats is a Python library that automatically translates English Wikipedia category names into standardized Arabic equivalents. The system processes category labels through a multi-stage resolution pipeline, leveraging extensive translation dictionaries covering temporal patterns, geographic entities, occupations, sports, nationalities, and media-related terms.

This page provides a high-level understanding of the system's architecture, capabilities, and core components. For detailed information about specific subsystems:

* Installation and basic usage: see [Getting Started](1.Getting-Started.md)
* Detailed architecture: see [Architecture](2.Architecture.md)
* Translation data organization: see [Translation Data](6.Translation-Data.md)
* Resolver implementations: see [Resolver System](14.Resolver-System.md)
* Template formatting engine: see [Formatting System](22.Formatting-System.md)

**Sources:** [README.md L1-L560](../README.md#L1-L560), [CLAUDE.md L1-L227](../CLAUDE.md#L1-L227), [.github/copilot-instructions.md L1-L121](../.github/copilot-instructions.md#L1-L121)

---

## System Capabilities

The ArWikiCats translation engine maintains comprehensive translation datasets across multiple domains:

| Domain | Dataset Size | Key Examples |
| --- | --- | --- |
| **Jobs and Occupations** | 96,552 male entries, extensive female mappings | footballers, painters, scientists, religious occupations |
| **Sports** | 431 sport key records, 571 job variants | football, basketball, teams, players, competitions |
| **Nationalities** | 843 entries, 18 lookup tables | British, American, Egyptian (male/female/plural/definite forms) |
| **Geographic Data** | 68,981 entries | cities, regions, countries, counties, US states |
| **Films and Television** | 13,146 entries | film genres, TV series, directors, actors |
| **Test Coverage** | 28,500+ tests | unit, integration, end-to-end tests achieving 91% coverage |

The system handles complex multi-element categories such as:

* Temporal + Nationality + Occupation: "2010 British football players"
* Country + Year + Event: "1990s establishments in France"
* Nationality + Sport + Team: "Argentine football club managers"

**Sources:** [README.md L42-L113](../README.md#L42-L113), [changelog.md L1-L80](../changelog.md#L1-L80)

---

## High-Level Architecture

The system follows a layered architecture with specialized resolver chains that process categories in priority order:

```mermaid
flowchart TD

API1["resolve_arabic_category_label()"]
API2["batch_resolve_labels()"]
API3["resolve_label_ar()"]
API4["EventProcessor"]
MainResolve["main_processers/main_resolve.py<br/>resolve_label()"]
Cache["@lru_cache(maxsize=50000)"]
ChangeCAT["format_bots/change_cat()"]
FilterEN["fix/filter_en.py<br/>is_category_allowed()"]
PatternResolvers["patterns_resolvers/<br/>all_patterns_resolvers()"]
NewResolvers["new_resolvers/<br/>all_new_resolvers()"]
UnivResolver["sub_new_resolvers/<br/>university_resolver"]
LegacyResolver["legacy_bots/<br/>legacy_resolvers()"]
FixLabel["fix/fixtitle.py<br/>fixlabel()"]
Cleanse["fix/<br/>cleanse_category_label()"]
TranslationsGeo["translations/geo/<br/>CITY_TRANSLATIONS"]
TranslationsJobs["translations/jobs/<br/>jobs_mens_data"]
TranslationsSports["translations/sports/<br/>SPORT_KEY_RECORDS"]
TranslationsNats["translations/nats/<br/>All_Nat"]

API1 --> MainResolve
API2 --> MainResolve
API3 --> MainResolve
API4 --> MainResolve
Cache --> ChangeCAT
FilterEN --> PatternResolvers
PatternResolvers --> FixLabel
NewResolvers --> FixLabel
UnivResolver --> FixLabel
LegacyResolver --> FixLabel
NewResolvers --> TranslationsGeo
NewResolvers --> TranslationsJobs
NewResolvers --> TranslationsSports
NewResolvers --> TranslationsNats

subgraph subGraph5 ["Translation Data Sources"]
    TranslationsGeo
    TranslationsJobs
    TranslationsSports
    TranslationsNats
end

subgraph subGraph4 ["Output Processing"]
    FixLabel
    Cleanse
    FixLabel --> Cleanse
end

subgraph subGraph3 ["Resolution Pipeline (Priority Order)"]
    PatternResolvers
    NewResolvers
    UnivResolver
    LegacyResolver
    PatternResolvers --> NewResolvers
    NewResolvers --> UnivResolver
    UnivResolver --> LegacyResolver
end

subgraph subGraph2 ["Input Processing"]
    ChangeCAT
    FilterEN
    ChangeCAT --> FilterEN
end

subgraph subGraph1 ["Main Resolution Coordinator"]
    MainResolve
    Cache
    MainResolve --> Cache
end

subgraph subGraph0 ["Public API Layer"]
    API1
    API2
    API3
    API4
end
```

**Architecture Overview**

The system is organized into five primary layers:

1. **Public API Layer** - User-facing functions in [ArWikiCats/__init__.py](../ArWikiCats/__init__.py)  that provide simple interfaces for single-category and batch translation
2. **Main Resolution Coordinator** - [main_processers/main_resolve.py L33-L93](../ArWikiCats/main_processers/main_resolve.py#L33-L93)  orchestrates the resolution pipeline with LRU caching
3. **Input Processing** - Normalizes categories and filters invalid inputs before resolution
4. **Resolution Pipeline** - Prioritized chain of specialized resolvers that attempt pattern matching
5. **Output Processing** - Applies Arabic formatting rules and cleansing to finalized labels

**Sources:** [ArWikiCats/main_processers/main_resolve.py L1-L106](../ArWikiCats/main_processers/main_resolve.py#L1-L106), [CLAUDE.md L69-L103](../CLAUDE.md#L69-L103)

---

## Resolution Pipeline Flow

The category resolution process follows a strict priority order to prevent conflicts and ensure accurate translations:

```mermaid
flowchart TD

Input["Input: English Category<br/>'Category:2010 British footballers'"]
ChangeCat["change_cat()<br/>Remove 'Category:' prefix<br/>Lowercase, clean spaces"]
Filter["filter_en.is_category_allowed()<br/>Check if category is valid"]
AllPatterns["all_patterns_resolvers()<br/>patterns_resolvers/init.py"]
TimePatterns["Time patterns<br/>years, decades, centuries"]
CountryTime["Country + Year patterns<br/>nat_males_pattern"]
AllNew["all_new_resolvers()<br/>new_resolvers/init.py"]
Jobs["Jobs Resolvers (Priority 3)<br/>jobs_resolvers/init.py<br/>main_jobs_resolvers()"]
Sports["Sports Resolvers (Priority 5)<br/>sports_resolvers/init.py<br/>main_sports_resolvers()"]
Nats["Nationality Resolvers (Priority 6)<br/>nationalities_resolvers/init.py"]
Countries["Countries Resolvers (Priority 7)<br/>countries_names_resolvers/init.py"]
Univ["university_resolver<br/>sub_new_resolvers/"]
Legacy["legacy_resolvers()<br/>legacy_bots/init.py<br/>LegacyBotsResolver"]
Fix["fixlabel()<br/>Apply Arabic grammar rules"]
Clean["cleanse_category_label()<br/>Final formatting"]
Result["Output: Arabic Category<br/>'تصنيف:لاعبو كرة قدم بريطانيون عام 2010'"]
Empty["Return empty string"]

Input --> ChangeCat
Filter --> AllPatterns
Filter --> Empty
AllPatterns --> AllNew
Countries --> Univ
AllPatterns --> Fix
AllNew --> Fix
Univ --> Fix
Legacy --> Fix
Legacy --> Empty

subgraph OutputStage ["Output Processing"]
    Fix
    Clean
    Result
    Fix --> Clean
    Clean --> Result
end

subgraph LegacyStage ["Legacy Resolvers (Priority 4)"]
    Univ
    Legacy
    Univ --> Legacy
end

subgraph NewStage ["New Resolvers (Priority 2)"]
    AllNew
    Jobs
    Sports
    Nats
    Countries
    AllNew --> Jobs
    Jobs --> Sports
    Sports --> Nats
    Nats --> Countries
end

subgraph PatternStage ["Pattern Resolution (Priority 1)"]
    AllPatterns
    TimePatterns
    CountryTime
    AllPatterns --> TimePatterns
    AllPatterns --> CountryTime
end

subgraph Normalization ["Normalization Stage"]
    ChangeCat
    Filter
    ChangeCat --> Filter
end
```

**Resolution Priority Rationale**

The resolver ordering is critical to prevent conflicts:

1. **Pattern resolvers first** - Fast regex-based matching for common patterns (years, time + country)
2. **Jobs before Sports** - "football manager" must resolve as a job, not a sports category
3. **Nationalities before Countries** - "Italy political leader" uses nationality form, not country name
4. **Sports after Jobs** - Prevents job titles from being misclassified as sports terms
5. **Legacy resolvers last** - Backward compatibility for patterns not yet migrated

The system uses `@functools.lru_cache(maxsize=50000)` on the main resolver to cache results and achieve high throughput.

**Sources:** [ArWikiCats/main_processers/main_resolve.py L32-L94](../ArWikiCats/main_processers/main_resolve.py#L32-L94), [CLAUDE.md L79-L92](../CLAUDE.md#L79-L92)

---

## Core Components and File Organization

The codebase is organized into specialized modules, each handling a specific aspect of translation:

```mermaid
flowchart TD

Legacy["legacy_bots/<br/>Backward-compatible resolvers"]
LegacyResolver["LegacyBotsResolver<br/>Refactored pipeline"]
CircularDep["resolvers/<br/>Circular dependency resolution"]
NewResolvers["new_resolvers/<br/>Modern resolver implementations"]
JobsResolvers["new_resolvers/jobs_resolvers/<br/>mens.py, womens.py, religious.py"]
SportsResolvers["new_resolvers/sports_resolvers/<br/>raw_sports, teams, nationality+sport"]
NatsResolvers["new_resolvers/nationalities_resolvers/<br/>nationality-based categories"]
CountriesResolvers["new_resolvers/countries_names_resolvers/<br/>country-based categories"]
Patterns["patterns_resolvers/<br/>Regex-based pattern resolvers"]
CountryTime["country_time_pattern.py<br/>'1990s in France'"]
NatMales["nat_males_pattern.py<br/>'British male actors'"]
Formats["translations_formats/<br/>Template formatting system"]
FormatBase["DataModel/model_data_base.py<br/>FormatDataBase"]
MultiFormatter["DataModel/model_multi_data.py<br/>MultiDataFormatterBase"]
Factories["data_with_time.py, multi_data.py<br/>Factory functions"]
TransData["translations/<br/>Python modules with data"]
GeoData["geo/, cities/<br/>Geographic translations"]
JobsData["jobs/<br/>96,552 job entries"]
SportsData["sports/<br/>431 sport records"]
NatsData["nats/<br/>843 nationality entries"]
JSONs["jsons/<br/>Raw JSON data files"]
JSONGeo["geography/, cities/"]
JSONJobs["jobs/"]
JSONSports["sports/"]
JSONNats["nationalities/"]
Fix["fix/<br/>Normalization & cleaning"]
Utils["utils/<br/>Helper functions"]
Config["config.py<br/>Environment settings"]

JobsResolvers --> Formats
SportsResolvers --> Formats
NatsResolvers --> Formats
CountriesResolvers --> Formats
JobsResolvers --> TransData
SportsResolvers --> TransData
NatsResolvers --> TransData
CountriesResolvers --> TransData
TransData --> JSONs
NewResolvers --> Fix
Patterns --> Fix

subgraph Utilities ["Utilities"]
    Fix
    Utils
    Config
end

subgraph subGraph4 ["JSON Source Files"]
    JSONs
    JSONGeo
    JSONJobs
    JSONSports
    JSONNats
    JSONs --> JSONGeo
    JSONs --> JSONJobs
    JSONs --> JSONSports
    JSONs --> JSONNats
end

subgraph subGraph3 ["Translation Data"]
    TransData
    GeoData
    JobsData
    SportsData
    NatsData
    TransData --> GeoData
    TransData --> JobsData
    TransData --> SportsData
    TransData --> NatsData
end

subgraph subGraph2 ["Template Engine"]
    Formats
    FormatBase
    MultiFormatter
    Factories
    Formats --> FormatBase
    Formats --> MultiFormatter
    Formats --> Factories
end

subgraph subGraph1 ["Pattern Matching"]
    Patterns
    CountryTime
    NatMales
end

subgraph subGraph0 ["Resolution Modules"]
    NewResolvers
    JobsResolvers
    SportsResolvers
    NatsResolvers
    CountriesResolvers
    NewResolvers --> JobsResolvers
    NewResolvers --> SportsResolvers
    NewResolvers --> NatsResolvers
    NewResolvers --> CountriesResolvers
end

subgraph subGraph5 ["Legacy System"]
    Legacy
    LegacyResolver
    CircularDep
end
```

**Module Responsibilities**

| Module Path | Responsibility | Key Files |
| --- | --- | --- |
| `main_processers/` | Orchestrates resolution pipeline | `main_resolve.py` |
| `new_resolvers/` | Modern domain-specific resolvers | `jobs_resolvers/`, `sports_resolvers/`, `nationalities_resolvers/` |
| `patterns_resolvers/` | Regex-based pattern matching | `country_time_pattern.py`, `nat_males_pattern.py` |
| `translations_formats/` | Template formatting engine | `DataModel/`, factory functions |
| `translations/` | Translation data as Python modules | `geo/`, `jobs/`, `sports/`, `nats/`, `tv/` |
| `jsons/` | Raw JSON translation data | `geography/`, `jobs/`, `sports/`, `nationalities/` |
| `legacy_bots/` | Backward-compatible resolvers | `LegacyBotsResolver`, `resolvers/` |
| `fix/` | Text normalization and cleaning | `fixtitle.py`, `fixlists.py` |

**Sources:** [README.md L333-L430](../README.md#L333-L430), [CLAUDE.md L199-L220](../CLAUDE.md#L199-L220)

---

## Public API

The system exposes four main entry points for category translation:

```mermaid
flowchart TD

API1["resolve_arabic_category_label(category: str)<br/>→ str<br/>Returns: 'تصنيف:...'"]
API2["resolve_label_ar(category: str)<br/>→ str<br/>Returns Arabic without 'تصنيف:' prefix"]
API3["batch_resolve_labels(categories: List[str])<br/>→ CategoryBatchResult<br/>Returns dict of translations"]
API4["EventProcessor.process_single(category: str)<br/>→ CategoryProcessingResult<br/>Returns detailed result with metadata"]
Result1["str - Arabic category with prefix"]
Result2["str - Arabic label only"]
Result3["CategoryBatchResult<br/>labels: dict<br/>no_labels: list<br/>category_patterns: dict"]
Result4["CategoryProcessingResult<br/>original, normalized, raw_label<br/>final_label, has_label"]

API1 --> Result1
API2 --> Result2
API3 --> Result3
API4 --> Result4

subgraph subGraph1 ["Return Types"]
    Result1
    Result2
    Result3
    Result4
end

subgraph subGraph0 ["Public API Functions"]
    API1
    API2
    API3
    API4
end
```

**Usage Examples**

```python
from ArWikiCats import (
    resolve_arabic_category_label,
    resolve_label_ar,
    batch_resolve_labels,
    EventProcessor
)

# Single category with prefix
label = resolve_arabic_category_label("Category:2015 British footballers")
# Returns: "تصنيف:لاعبو كرة قدم بريطانيون عام 2015"

# Single category without prefix
label = resolve_label_ar("British footballers")
# Returns: "لاعبو كرة قدم بريطانيون"

# Batch processing
categories = [
    "Category:American basketball players",
    "Category:1990s establishments in France"
]
result = batch_resolve_labels(categories)
# result.labels: dict of successful translations
# result.no_labels: list of categories without translations

# Detailed processing with metadata
processor = EventProcessor()
result = processor.process_single("Category:British footballers")
# result.original: "Category:British footballers"
# result.normalized: "british footballers"
# result.final_label: "تصنيف:لاعبو كرة قدم بريطانيون"
# result.has_label: True
```

All public API functions are exported from [ArWikiCats/ L1-L40](../ArWikiCats/__init__.py#L1-L40)

 and documented in the package's top-level module.

**Sources:** [README.md L170-L230](../README.md#L170-L230), [ArWikiCats/main_processers/main_resolve.py L33-L100](../ArWikiCats/main_processers/main_resolve.py#L33-L100)

---

## Translation Data Architecture

Translation data flows from raw JSON files through Python aggregator modules into specialized resolver implementations:

```mermaid
flowchart TD

JSON1["jobs/jobs.json<br/>jobs/Jobs_22.json"]
JSON2["sports/Sports_Keys_New.json<br/>431 sports"]
JSON3["nationalities/nationalities_data.json<br/>843 nationalities"]
JSON4["geography/P17_2_final_ll.json<br/>cities/popopo.json"]
JSON5["media/Films_key_For_nat.json<br/>13,146 film entries"]
Agg1["jobs/Jobs.py<br/>_finalise_jobs_dataset()<br/>→ 96,552 entries"]
Agg2["sports/Sport_key.py<br/>_build_tables()<br/>→ SPORT_KEY_RECORDS"]
Agg3["nats/Nationality.py<br/>build_lookup_tables()<br/>→ 18 lookup tables"]
Agg4["geo/labels_country.py<br/>_build_country_label_index()<br/>→ 68,981 entries"]
Agg5["tv/films_mslslat.py<br/>_build_gender_key_maps()"]
Export1["jobs_mens_data: 96,552<br/>jobs_womens_data<br/>Jobs_new: 1,304"]
Export2["SPORT_KEY_RECORDS: 431<br/>SPORTS_KEYS_FOR_LABEL<br/>SPORT_JOB_VARIANTS: 571"]
Export3["All_Nat: 843<br/>Nat_men, Nat_womens<br/>countries_from_nat: 287"]
Export4["CITY_TRANSLATIONS_LOWER<br/>COUNTRY_LABEL_OVERRIDES<br/>US_STATES"]
Resolvers["new_resolvers/<br/>Domain-specific resolvers"]
JobsRes["jobs_resolvers/"]
SportsRes["sports_resolvers/"]
NatsRes["nationalities_resolvers/"]
CountriesRes["countries_names_resolvers/"]

JSON1 --> Agg1
JSON2 --> Agg2
JSON3 --> Agg3
JSON4 --> Agg4
JSON5 --> Agg5
Agg1 --> Export1
Agg2 --> Export2
Agg3 --> Export3
Agg4 --> Export4
Export1 --> JobsRes
Export2 --> SportsRes
Export3 --> NatsRes
Export4 --> CountriesRes

subgraph subGraph3 ["Resolver Consumption"]
    Resolvers
    JobsRes
    SportsRes
    NatsRes
    CountriesRes
    JobsRes --> Resolvers
    SportsRes --> Resolvers
    NatsRes --> Resolvers
    CountriesRes --> Resolvers
end

subgraph subGraph2 ["Exported Data Structures"]
    Export1
    Export2
    Export3
    Export4
end

subgraph subGraph1 ["Aggregation Layer (translations/)"]
    Agg1
    Agg2
    Agg3
    Agg4
    Agg5
end

subgraph subGraph0 ["Raw Data Sources (jsons/)"]
    JSON1
    JSON2
    JSON3
    JSON4
    JSON5
end
```

**Data Processing Pipeline**

The translation data undergoes three stages of transformation:

1. **Raw JSON Storage** - Source data files in `jsons/` directory maintain original mappings
2. **Python Aggregation** - Modules in `translations/` process, merge, and transform raw data
3. **Exported Structures** - Final data structures (dicts, sets, lookup tables) used by resolvers

Key aggregation operations include:

* **Jobs**: Combines multiple JSON sources, handles gender variants, generates 96,552 male job entries
* **Sports**: Builds sport key records, job variants, team-related mappings from 431 base sports
* **Nationalities**: Creates 18 specialized lookup tables for different grammatical forms (male, female, plural, definite)
* **Geography**: Indexes 68,981 entries with city translations, country overrides, US state mappings

**Sources:** [README.md L89-L113](../README.md#L89-L113), [changelog.md L247-L294](../changelog.md#L247-L294)

---

## Performance Characteristics

The system is optimized for high-throughput batch processing with the following characteristics:

| Metric | Value | Implementation |
| --- | --- | --- |
| **Memory Usage** | <100 MB | Optimized from 2GB in legacy system |
| **Cache Size** | 50,000 entries | `@functools.lru_cache` on `resolve_label()` |
| **Test Suite Speed** | ~23 seconds | 28,500+ tests with pytest |
| **Batch Throughput** | >5,000 categories/second | Demonstrated in `examples/5k.py` |
| **Test Coverage** | 91% | Recent improvements to legacy_bots and translations_formats |

**Caching Strategy**

The system employs multiple levels of caching:

```python
# Main resolution function with LRU cache
@functools.lru_cache(maxsize=50000)
def resolve_label(category: str, fix_label: bool = True) -> CategoryResult:
    # ... resolution logic
```

Additional caching is applied to:

* Static data loading functions in resolver modules
* Pattern compilation in `patterns_resolvers/`
* Lookup table generation in `translations/` modules

**Optimization Techniques**

1. **Early filtering** - Invalid categories rejected before entering resolver chain
2. **Priority ordering** - Most common patterns checked first
3. **Lazy loading** - Translation data loaded on-demand in some modules
4. **Pre-compiled regexes** - Pattern matchers compiled at module load time

**Sources:** [README.md L498-L508](../README.md#L498-L508), [ArWikiCats/main_processers/main_resolve.py L32](../ArWikiCats/main_processers/main_resolve.py#L32), [changelog.md L269-L293](../changelog.md#L269-L293)

---

## System Design Principles

ArWikiCats follows several key design principles that inform its architecture:

**1. Resolver Chain Priority Pattern**

Resolvers are ordered to prevent semantic conflicts. For example, "football manager" must be resolved by the jobs resolver (as an occupation) before the sports resolver can interpret it as a sports management role. The priority order is explicitly documented in [new_resolvers/ L29-L57](../new_resolvers/__init__.py#L29-L57)

**2. Template-Based Formatting**

The `FormatDataBase` class and its variants provide a template engine for pattern matching and placeholder replacement. This allows resolvers to define translation patterns like `"{nat} {sport} players"` → `"لاعبو {sport} {nat}"` with automatic substitution of Arabic equivalents.

**3. Separation of Data and Logic**

Translation data resides in `jsons/` and `translations/` directories, completely separate from resolver logic. This allows non-developers to contribute translations without modifying code.

**4. Backward Compatibility**

The `legacy_bots/` module maintains compatibility with pre-existing translation patterns through the `LegacyBotsResolver` class, which was refactored from a list-based pipeline while preserving 100% of original behavior.

**5. Comprehensive Testing**

With 28,500+ tests organized into unit, integration, and end-to-end categories, the system achieves 91% code coverage and validates translation accuracy across thousands of real-world categories.

**Sources:** [changelog.md L170-L200](../changelog.md#L170-L200), [CLAUDE.md L139-L143](../CLAUDE.md#L139-L143), [README.md L434-L508](../README.md#L434-L508)
