# Translation Data

> **Relevant source files**
> * [ArWikiCats/translations/__init__.py](../ArWikiCats/translations/__init__.py)
> * [ArWikiCats/translations/build_data/__init__.py](../ArWikiCats/translations/build_data/__init__.py)
> * [ArWikiCats/translations/funcs.py](../ArWikiCats/translations/funcs.py)
> * [ArWikiCats/translations/geo/__init__.py](../ArWikiCats/translations/geo/__init__.py)
> * [ArWikiCats/translations/geo/labels_country.py](../ArWikiCats/translations/geo/labels_country.py)
> * [ArWikiCats/translations/jobs/Jobs.py](../ArWikiCats/translations/jobs/Jobs.py)
> * [ArWikiCats/translations/jobs/Jobs2.py](../ArWikiCats/translations/jobs/Jobs2.py)
> * [ArWikiCats/translations/jobs/jobs_data_basic.py](../ArWikiCats/translations/jobs/jobs_data_basic.py)
> * [ArWikiCats/translations/jobs/jobs_players_list.py](../ArWikiCats/translations/jobs/jobs_players_list.py)
> * [ArWikiCats/translations/jobs/jobs_singers.py](../ArWikiCats/translations/jobs/jobs_singers.py)
> * [ArWikiCats/translations/jobs/jobs_womens.py](../ArWikiCats/translations/jobs/jobs_womens.py)
> * [ArWikiCats/translations/mixed/all_keys2.py](../ArWikiCats/translations/mixed/all_keys2.py)
> * [ArWikiCats/translations/mixed/female_keys.py](../ArWikiCats/translations/mixed/female_keys.py)
> * [ArWikiCats/translations/mixed/keys2.py](../ArWikiCats/translations/mixed/keys2.py)
> * [ArWikiCats/translations/others/__init__.py](../ArWikiCats/translations/others/__init__.py)
> * [ArWikiCats/translations/others/tax_table.py](../ArWikiCats/translations/others/tax_table.py)
> * [ArWikiCats/translations/sports/Sport_key.py](../ArWikiCats/translations/sports/Sport_key.py)
> * [ArWikiCats/translations/tv/films_mslslat.py](../ArWikiCats/translations/tv/films_mslslat.py)
> * [_work_files/data_len.json](../_work_files/data_len.json)

## Purpose and Scope

The Translation Data layer provides the foundational English-to-Arabic mapping tables used throughout the ArWikiCats system. This page documents the organization, structure, and scale of these translation resources across all supported domains.

For information about how this data is processed and aggregated, see [Data Aggregation Pipeline](7.Data-Aggregation-Pipeline.md). For details about specific resolvers that consume this data, see [Resolver System](14.Resolver-System.md)

## Overview

The translation data consists of structured dictionaries mapping English Wikipedia category terms to their Arabic equivalents. The system manages approximately **250,000+ translation entries** across multiple domains, organized into domain-specific modules under [ArWikiCats/translations/](../ArWikiCats/translations/)

The data originates from raw JSON files in [jsons/](../jsons/)

 which are processed by Python aggregator modules into typed dictionaries exported via [ArWikiCats/translations/__init__.py L1-L152](../ArWikiCats/translations/__init__.py#L1-L152)

## Data Architecture

```mermaid
flowchart TD

JSON_GEO["jsons/geography/<br/>P17_2_final_ll.json<br/>popopo.json<br/>yy2.json"]
JSON_JOBS["jsons/jobs/<br/>jobs.json<br/>Jobs_22.json<br/>jobs_3.json"]
JSON_SPORTS["jsons/sports/<br/>Sports_Keys_New.json<br/>clubs_teams_leagues.json"]
JSON_NATS["jsons/nationalities/<br/>nationalities_data.json"]
JSON_MEDIA["jsons/media/<br/>Films_key_For_nat.json<br/>films_mslslat_tab_found.json"]
GEO_PY["translations/geo/<br/>labels_country.py<br/>Cities.py<br/>regions.py"]
JOBS_PY["translations/jobs/<br/>Jobs.py<br/>jobs_players_list.py<br/>jobs_singers.py"]
SPORTS_PY["translations/sports/<br/>Sport_key.py"]
NATS_PY["translations/nats/<br/>Nationality.py"]
MEDIA_PY["translations/tv/<br/>films_mslslat.py"]
MIXED_PY["translations/mixed/<br/>all_keys2.py<br/>keys2.py"]
CITY["CITY_TRANSLATIONS_LOWER<br/>10,526 entries"]
COUNTRY["COUNTRY_LABEL_OVERRIDES<br/>1,459 entries"]
JOBS_M["jobs_mens_data<br/>97,797 entries"]
JOBS_W["jobs_womens_data<br/>75,244 entries"]
NATS["All_Nat<br/>843 entries<br/>18 lookup tables"]
SPORT_KEYS["SPORT_KEY_RECORDS<br/>431 entries"]
FILMS["Films_key_CAO<br/>13,146 entries"]
PF_KEYS["pf_keys2<br/>33,657 entries"]
NEW_P17["NEW_P17_FINAL<br/>68,981 entries<br/>Unified geo index"]
BUILD_DATA["build_data/init.py<br/>Final aggregation"]
RESOLVERS["Resolver Chain"]

JSON_GEO --> GEO_PY
JSON_JOBS --> JOBS_PY
JSON_SPORTS --> SPORTS_PY
JSON_NATS --> NATS_PY
JSON_MEDIA --> MEDIA_PY
GEO_PY --> CITY
GEO_PY --> COUNTRY
JOBS_PY --> JOBS_M
JOBS_PY --> JOBS_W
SPORTS_PY --> SPORT_KEYS
NATS_PY --> NATS
MEDIA_PY --> FILMS
MIXED_PY --> PF_KEYS
CITY --> NEW_P17
COUNTRY --> NEW_P17
NATS --> NEW_P17
PF_KEYS --> BUILD_DATA
JOBS_M --> RESOLVERS
SPORT_KEYS --> RESOLVERS
FILMS --> RESOLVERS
BUILD_DATA --> RESOLVERS

subgraph subGraph3 ["Aggregated Datasets"]
    NEW_P17
    BUILD_DATA
    NEW_P17 --> BUILD_DATA
end

subgraph subGraph2 ["Exported Dictionaries"]
    CITY
    COUNTRY
    JOBS_M
    JOBS_W
    NATS
    SPORT_KEYS
    FILMS
    PF_KEYS
end

subgraph subGraph1 ["Python Aggregators"]
    GEO_PY
    JOBS_PY
    SPORTS_PY
    NATS_PY
    MEDIA_PY
    MIXED_PY
end

subgraph subGraph0 ["Raw Data Sources"]
    JSON_GEO
    JSON_JOBS
    JSON_SPORTS
    JSON_NATS
    JSON_MEDIA
end
```

**Sources:** [ArWikiCats/translations/__init__.py L1-L152](../ArWikiCats/translations/__init__.py#L1-L152), [ArWikiCats/translations/build_data/__init__.py L1-L83](../ArWikiCats/translations/build_data/__init__.py#L1-L83), [_work_files/data_len.json L1-L135](../_work_files/data_len.json#L1-L135)

## Data Organization by Domain

The translation data is partitioned into seven major domains, each with specialized structure and processing requirements:

| Domain | Primary Modules | Key Datasets | Scale |
| --- | --- | --- | --- |
| **Geography** | `translations/geo/` | `CITY_TRANSLATIONS_LOWER`, `COUNTRY_LABEL_OVERRIDES`, `US_STATES` | 10,526 cities, 1,459 countries |
| **Jobs** | `translations/jobs/` | `jobs_mens_data`, `jobs_womens_data`, `Jobs_new` | 97,797 male jobs, 75,244 female jobs |
| **Nationalities** | `translations/nats/` | `All_Nat`, `Nat_men`, `Nat_women` | 843 nationalities, 18 forms |
| **Sports** | `translations/sports/` | `SPORT_KEY_RECORDS`, `SPORTS_KEYS_FOR_LABEL` | 431 sports |
| **Films/TV** | `translations/tv/` | `Films_key_CAO`, `Films_key_For_nat` | 13,146 film entries |
| **Ministers** | `translations/others/` | `ministers_keys` | 99 political roles |
| **Mixed** | `translations/mixed/` | `pf_keys2`, `keys_of_without_in` | 33,657 generic keys |

**Sources:** [ArWikiCats/translations/__init__.py L9-L85](../ArWikiCats/translations/__init__.py#L9-L85), [_work_files/data_len.json L1-L135](../_work_files/data_len.json#L1-L135)

## Export Structure

Translation data is exported through two layers:

### Direct Exports (translations/__init__.py)

Domain-specific dictionaries exported directly from their aggregator modules for immediate use by resolvers:

```mermaid
flowchart TD

INIT["translations/init.py"]
CITY_T["CITY_TRANSLATIONS_LOWER"]
COUNTRY_O["COUNTRY_LABEL_OVERRIDES"]
US_S["US_STATES"]
JOBS_M["jobs_mens_data"]
JOBS_W["jobs_womens_data"]
JOBS_N["Jobs_new"]
SPORT_JOB["SPORT_JOB_VARIANTS"]
ALL_NAT["All_Nat"]
NAT_MEN["Nat_men"]
NAT_WOMEN["Nat_women"]
SPORT_REC["SPORT_KEY_RECORDS"]
SPORT_LAB["SPORTS_KEYS_FOR_LABEL"]
SPORT_TEAM["SPORTS_KEYS_FOR_TEAM"]
FILMS_CAO["Films_key_CAO"]
FILMS_NAT["Films_key_For_nat"]
TV_KEYS["TELEVISION_KEYS"]

INIT --> CITY_T
INIT --> COUNTRY_O
INIT --> US_S
INIT --> JOBS_M
INIT --> JOBS_W
INIT --> JOBS_N
INIT --> SPORT_JOB
INIT --> ALL_NAT
INIT --> NAT_MEN
INIT --> NAT_WOMEN
INIT --> SPORT_REC
INIT --> SPORT_LAB
INIT --> SPORT_TEAM
INIT --> FILMS_CAO
INIT --> FILMS_NAT
INIT --> TV_KEYS

subgraph subGraph4 ["Media Exports"]
    FILMS_CAO
    FILMS_NAT
    TV_KEYS
end

subgraph subGraph3 ["Sports Exports"]
    SPORT_REC
    SPORT_LAB
    SPORT_TEAM
end

subgraph subGraph2 ["Nationality Exports"]
    ALL_NAT
    NAT_MEN
    NAT_WOMEN
end

subgraph subGraph1 ["Jobs Exports"]
    JOBS_M
    JOBS_W
    JOBS_N
    SPORT_JOB
end

subgraph subGraph0 ["Geography Exports"]
    CITY_T
    COUNTRY_O
    US_S
end
```

**Sources:** [ArWikiCats/translations/__init__.py L87-L151](../ArWikiCats/translations/__init__.py#L87-L151)

### Aggregated Exports (build_data/__init__.py)

Comprehensive merged datasets combining multiple sources:

```markdown
# build_data/__init__.py exports two unified indexes:

pf_keys2 = generate_key_mappings(...)  # 33,657 entries
# Merges: keys2_py, pop_final_3, film keys, language keys, etc.

NEW_P17_FINAL = _build_country_label_index(...)  # 68,981 entries
# Merges: cities, countries, regions, states, taxonomies
```

**Sources:** [ArWikiCats/translations/build_data/__init__.py L42-L69](../ArWikiCats/translations/build_data/__init__.py#L42-L69)

## Data Access Patterns

The system provides multiple access methods for translation lookups:

```mermaid
flowchart TD

CLIENT["Resolver Code"]
DIRECT_IMPORT["Direct Import<br/>from translations import jobs_mens_data"]
DICT_LOOKUP["Dictionary Lookup<br/>jobs_mens_data.get('engineers')"]
GET_P17["get_from_new_p17_final(text)<br/>translations/funcs.py:37-56"]
GET_PF["get_from_pf_keys2(text)<br/>translations/funcs.py:101-113"]
GET_ALIAS["_get_from_alias(key)<br/>translations/funcs.py:116-149"]
PF_KEYS["pf_keys2"]
NEW_P17["NEW_P17_FINAL"]
JOBS["jobs_mens_data"]
FILMS["films_mslslat_tab"]
ALIASES["ALIASES_CHAIN"]

CLIENT --> DIRECT_IMPORT
CLIENT --> GET_P17
CLIENT --> GET_PF
CLIENT --> GET_ALIAS
GET_P17 --> ALIASES
GET_P17 --> PF_KEYS
GET_P17 --> NEW_P17
GET_PF --> PF_KEYS
GET_ALIAS --> PF_KEYS
GET_ALIAS --> JOBS
GET_ALIAS --> FILMS
GET_ALIAS --> NEW_P17

subgraph subGraph2 ["Data Sources"]
    PF_KEYS
    NEW_P17
    JOBS
    FILMS
    ALIASES
end

subgraph subGraph1 ["Helper Functions"]
    GET_P17
    GET_PF
    GET_ALIAS
end

subgraph subGraph0 ["Direct Access"]
    DIRECT_IMPORT
    DICT_LOOKUP
    DIRECT_IMPORT --> DICT_LOOKUP
end
```

**Sources:** [ArWikiCats/translations/funcs.py L1-L159](../ArWikiCats/translations/funcs.py#L1-L159)

## Data Types and Structures

### Simple Mappings

Basic string-to-string dictionaries for straightforward translations:

```python
# Example from COUNTRY_LABEL_OVERRIDES
{
    "united states": "الولايات المتحدة",
    "united kingdom": "المملكة المتحدة"
}
```

### Gendered Mappings

Job and role translations with masculine and feminine forms:

```python
# Example from jobs_mens_data and jobs_womens_data
{
    "engineers": {
        "males": "مهندسون",
        "females": "مهندسات"
    }
}
```

### Template Mappings

Patterns with placeholders for dynamic substitution:

```python
# Example from Films_key_For_nat
{
    "drama films": "أفلام درامية {}",  # {} = nationality placeholder
    "action films": "أفلام حركة {}"
}
```

### Multi-attribute Records

Complex structures with multiple translation contexts:

```python
# Example from SPORT_KEY_RECORDS (Sport_key.py)
{
    "football": {
        "label": "كرة القدم",      # General label
        "team": "كرة قدم",         # For teams
        "jobs": "كرة قدم"          # For jobs/occupations
    }
}
```

**Sources:** [ArWikiCats/translations/sports/Sport_key.py L1-L73](../ArWikiCats/translations/sports/Sport_key.py#L1-L73), [ArWikiCats/translations/jobs/Jobs.py L1-L211](../ArWikiCats/translations/jobs/Jobs.py#L1-L211), [ArWikiCats/translations/tv/films_mslslat.py L118-L143](../ArWikiCats/translations/tv/films_mslslat.py#L118-L143)

## Scale and Performance

The translation data layer manages significant volumes requiring optimization:

| Dataset | Entries | Memory | Load Time |
| --- | --- | --- | --- |
| `jobs_mens_data` | 97,797 | 3.7 MiB | Module import |
| `jobs_womens_data` | 75,244 | 1.8 MiB | Module import |
| `NEW_P17_FINAL` | 68,981 | N/A | Built at import |
| `pf_keys2` | 33,657 | N/A | Built at import |
| `Films_key_CAO` | 13,146 | N/A | Module import |
| `CITY_TRANSLATIONS_LOWER` | 10,526 | N/A | Module import |
| **Total Managed** | ~250,000+ | ~6 MiB | <1s |

All translation data is loaded at module import time and cached in memory for fast lookups. The resolver chain benefits from Python's built-in dictionary hash table performance (O(1) average case)

**Sources:** [ArWikiCats/translations/jobs/Jobs.py L185-L194](../ArWikiCats/translations/jobs/Jobs.py#L185-L194), [_work_files/data_len.json L1-L135](../_work_files/data_len.json#L1-L135)

## Data Validation

Each aggregator module uses `len_print.data_len()` to validate dataset sizes during development:

```python
# Example from Jobs.py
len_print.data_len(
    "jobs.py",
    {
        "jobs_mens_data": jobs_mens_data,      # Validates entry count
        "jobs_womens_data": jobs_womens_data,
        "Jobs_new": Jobs_new,
    },
)
```

This validation runs during module initialization and helps detect unexpected changes in dataset sizes when JSON sources are updated.

**Sources:** [ArWikiCats/translations/jobs/Jobs.py L196-L204](../ArWikiCats/translations/jobs/Jobs.py#L196-L204), [ArWikiCats/translations/helps/len_print.py](../ArWikiCats/translations/helps/len_print.py)

## Related Pages

* For detailed aggregation pipelines, see [Data Aggregation Pipeline](7.Data-Aggregation-Pipeline.md)
* For geographic data specifics, see [Geographic Data](8.Geographic-Data.md)
* For jobs data structures, see [Jobs and Occupations](9.Jobs-and-Occupations.md)
* For nationality handling, see [Nationalities](10.Nationalities.md)
* For sports translations, see [Sports Data](11.Sports-Data.md)
* For media categories, see [Films and Television](12.Films-and-Television.md)
* For how resolvers consume this data, see [Resolver System](14.Resolver-System.md)
