"""Converts raw data files into json files usable by the training script.

Currently it supports converting WikiNER datasets, available here:
  https://figshare.com/articles/dataset/Learning_multilingual_named_entity_recognition_from_Wikipedia/5462500
  - download the language of interest to {Language}-WikiNER
  - then run
    prepare_ner_dataset.py French-WikiNER

A gold re-edit of WikiNER for French is here:
  - https://huggingface.co/datasets/danrun/WikiNER-fr-gold/tree/main
  - https://arxiv.org/abs/2411.00030
    Danrun Cao, Nicolas Béchet, Pierre-François Marteau
  - download to $NERBASE/wikiner-fr-gold/wikiner-fr-gold.conll
    prepare_ner_dataset.py fr_wikinergold

French WikiNER and its gold re-edit can be mixed together with
    prepare_ner_dataset.py fr_wikinermixed
  - the data for both WikiNER and WikiNER-fr-gold needs to be in the right place first

Also, Finnish Turku dataset, available here:
  - https://turkunlp.org/fin-ner.html
  - https://github.com/TurkuNLP/turku-ner-corpus
    git clone the repo into $NERBASE/finnish
    you will now have a directory
    $NERBASE/finnish/turku-ner-corpus
  - prepare_ner_dataset.py fi_turku

FBK in Italy produced an Italian dataset.
  - KIND: an Italian Multi-Domain Dataset for Named Entity Recognition
    Paccosi T. and Palmero Aprosio A.
    LREC 2022
  - https://arxiv.org/abs/2112.15099
  The processing here is for a combined .tsv file they sent us.
  - prepare_ner_dataset.py it_fbk
  There is a newer version of the data available here:
    https://github.com/dhfbk/KIND
  TODO: update to the newer version of the data

IJCNLP 2008 produced a few Indian language NER datasets.
  description:
    http://ltrc.iiit.ac.in/ner-ssea-08/index.cgi?topic=3
  download:
    http://ltrc.iiit.ac.in/ner-ssea-08/index.cgi?topic=5
  The models produced from these datasets have extremely low recall, unfortunately.
  - prepare_ner_dataset.py hi_ijc

FIRE 2013 also produced NER datasets for Indian languages.
  http://au-kbc.org/nlp/NER-FIRE2013/index.html
  The datasets are password locked.
  For Stanford users, contact Chris Manning for license details.
  For external users, please contact the organizers for more information.
  - prepare_ner_dataset.py hi-fire2013

HiNER is another Hindi dataset option
  https://github.com/cfiltnlp/HiNER
  - HiNER: A Large Hindi Named Entity Recognition Dataset
    Murthy, Rudra and Bhattacharjee, Pallab and Sharnagat, Rahul and
    Khatri, Jyotsana and Kanojia, Diptesh and Bhattacharyya, Pushpak
  There are two versions:
    hi_hinercollapsed and hi_hiner
  The collapsed version has just PER, LOC, ORG
  - convert data as follows:
    cd $NERBASE
    mkdir hindi
    cd hindi
    git clone git@github.com:cfiltnlp/HiNER.git
    python3 -m stanza.utils.datasets.ner.prepare_ner_dataset hi_hiner
    python3 -m stanza.utils.datasets.ner.prepare_ner_dataset hi_hinercollapsed

IL-NER has four datasets: HI, OR, TE, UR
  https://github.com/ltrc/IL-NER
  - Fine-tuning Pre-trained Named Entity Recognition Models For Indian Languages
    Bahad, Sankalp and Mishra, Pruthwik and
    Krishnamurthy, Parameswari and Sharma, Dipti
  Convert the data as follows:
    cd $NERBASE
    mkdir indic
    cd indic
    git clone git@github.com:ltrc/IL-NER.git
    python3 -m stanza.utils.datasets.ner.prepare_ner_dataset or_ilner

suralk/multiNER contains three languages, EN, SI, and TA
  https://github.com/suralk/multiNER
  https://arxiv.org/abs/2412.02056
  - Ranathunga, Surangika, et al.
    A Multi-way Parallel Named Entity Annotated Corpus for English, Tamil and Sinhala
  The tags are in BIO format, with the same 4 tags as CoNLL
  Convert the data as follows:
    cd $NERBASE
    mkdir mixed
    cd mixed
    git clone git@github.com:suralk/multiNER.git
    python3 -m stanza.utils.datasets.ner.prepare_ner_dataset ta_suralk

Ukranian NER is provided by lang-uk, available here:
  https://github.com/lang-uk/ner-uk
  git clone the repo to $NERBASE/lang-uk
  There should be a subdirectory $NERBASE/lang-uk/ner-uk/data at that point
  Conversion script graciously provided by Andrii Garkavyi @gawy
  - prepare_ner_dataset.py uk_languk

There are two Hungarian datasets are available here:
  https://rgai.inf.u-szeged.hu/node/130
  http://www.lrec-conf.org/proceedings/lrec2006/pdf/365_pdf.pdf
  We combined them and give them the label hu_rgai
  You can also build individual pieces with hu_rgai_business or hu_rgai_criminal
  Create a subdirectory of $NERBASE, $NERBASE/hu_rgai, and download both of
    the pieces and unzip them in that directory.
  - prepare_ner_dataset.py hu_rgai

Another Hungarian dataset is here:
  - https://github.com/nytud/NYTK-NerKor
  - git clone the entire thing in your $NERBASE directory to operate on it
  - prepare_ner_dataset.py hu_nytk

The two Hungarian datasets can be combined with hu_combined
  TODO: verify that there is no overlap in text
  - prepare_ner_dataset.py hu_combined

BSNLP publishes NER datasets for Eastern European languages.
  - In 2019 they published BG, CS, PL, RU.
  - http://bsnlp.cs.helsinki.fi/bsnlp-2019/shared_task.html
  - In 2021 they added some more data, but the test sets
    were not publicly available as of April 2021.
    Therefore, currently the model is made from 2019.
    In 2021, the link to the 2021 task is here:
    http://bsnlp.cs.helsinki.fi/shared-task.html
  - The below method processes the 2019 version of the corpus.
    It has specific adjustments for the BG section, which has
    quite a few typos or mis-annotations in it.  Other languages
    probably need similar work in order to function optimally.
  - make a directory $NERBASE/bsnlp2019
  - download the "training data are available HERE" and
    "test data are available HERE" to this subdirectory
  - unzip those files in that directory
  - we use the code name "bg_bsnlp19".  Other languages from
    bsnlp 2019 can be supported by adding the appropriate
    functionality in convert_bsnlp.py.
  - prepare_ner_dataset.py bg_bsnlp19

NCHLT produced NER datasets for many African languages.
  Unfortunately, it is difficult to make use of many of these,
  as there is no corresponding UD data from which to build a
  tokenizer or other tools.
  - Afrikaans:  https://repo.sadilar.org/handle/20.500.12185/299
  - isiNdebele: https://repo.sadilar.org/handle/20.500.12185/306
  - isiXhosa:   https://repo.sadilar.org/handle/20.500.12185/312
  - isiZulu:    https://repo.sadilar.org/handle/20.500.12185/319
  - Sepedi:     https://repo.sadilar.org/handle/20.500.12185/328
  - Sesotho:    https://repo.sadilar.org/handle/20.500.12185/334
  - Setswana:   https://repo.sadilar.org/handle/20.500.12185/341
  - Siswati:    https://repo.sadilar.org/handle/20.500.12185/346
  - Tsivenda:   https://repo.sadilar.org/handle/20.500.12185/355
  - Xitsonga:   https://repo.sadilar.org/handle/20.500.12185/362
  Agree to the license, download the zip, and unzip it in
  $NERBASE/NCHLT

UCSY built a Myanmar dataset.  They have not made it publicly
  available, but they did make it available to Stanford for research
  purposes.  Contact Chris Manning or John Bauer for the data files if
  you are Stanford affiliated.
  - https://arxiv.org/abs/1903.04739
  - Syllable-based Neural Named Entity Recognition for Myanmar Language
    by Hsu Myat Mo and Khin Mar Soe

Hanieh Poostchi et al produced a Persian NER dataset:
  - git@github.com:HaniehP/PersianNER.git
  - https://github.com/HaniehP/PersianNER
  - Hanieh Poostchi, Ehsan Zare Borzeshi, Mohammad Abdous, and Massimo Piccardi,
    "PersoNER: Persian Named-Entity Recognition"
  - Hanieh Poostchi, Ehsan Zare Borzeshi, and Massimo Piccardi,
    "BiLSTM-CRF for Persian Named-Entity Recognition; ArmanPersoNERCorpus: the First Entity-Annotated Persian Dataset"
  - Conveniently, this dataset is already in BIO format.  It does not have a dev split, though.
    git clone the above repo, unzip ArmanPersoNERCorpus.zip, and this script will split the
    first train fold into a dev section.

SUC3 is a Swedish NER dataset provided by Språkbanken
  - https://spraakbanken.gu.se/en/resources/suc3
  - The splitting tool is generously provided by
    Emil Stenstrom
    https://github.com/EmilStenstrom/suc_to_iob
  - Download the .bz2 file at this URL and put it in $NERBASE/sv_suc3shuffle
    It is not necessary to unzip it.
  - Gustafson-Capková, Sophia and Britt Hartmann, 2006, 
    Manual of the Stockholm Umeå Corpus version 2.0.
    Stockholm University.
  - Östling, Robert, 2013, Stagger 
    an Open-Source Part of Speech Tagger for Swedish
    Northern European Journal of Language Technology 3: 1–18
    DOI 10.3384/nejlt.2000-1533.1331
  - The shuffled dataset can be converted with dataset code
    prepare_ner_dataset.py sv_suc3shuffle
  - If you fill out the license form and get the official data,
    you can get the official splits by putting the provided zip file
    in $NERBASE/sv_suc3licensed.  Again, not necessary to unzip it
    python3 -m stanza.utils.datasets.ner.prepare_ner_dataset sv_suc3licensed

DDT is a reformulation of the Danish Dependency Treebank as an NER dataset
  - https://danlp-alexandra.readthedocs.io/en/latest/docs/datasets.html#dane
  - direct download link as of late 2021: https://danlp.alexandra.dk/304bd159d5de/datasets/ddt.zip
  - https://aclanthology.org/2020.lrec-1.565.pdf
    DaNE: A Named Entity Resource for Danish
    Rasmus Hvingelby, Amalie Brogaard Pauli, Maria Barrett,
    Christina Rosted, Lasse Malm Lidegaard, Anders Søgaard
  - place ddt.zip in $NERBASE/da_ddt/ddt.zip
    python3 -m stanza.utils.datasets.ner.prepare_ner_dataset da_ddt

NorNE is the Norwegian Dependency Treebank with NER labels
  - LREC 2020
    NorNE: Annotating Named Entities for Norwegian
    Fredrik Jørgensen, Tobias Aasmoe, Anne-Stine Ruud Husevåg,
    Lilja Øvrelid, and Erik Velldal
  - both Bokmål and Nynorsk
  - This dataset is in a git repo:
    https://github.com/ltgoslo/norne
    Clone it into $NERBASE
    git clone git@github.com:ltgoslo/norne.git
    python3 -m stanza.utils.datasets.ner.prepare_ner_dataset nb_norne
    python3 -m stanza.utils.datasets.ner.prepare_ner_dataset nn_norne

tr_starlang is a set of constituency trees for Turkish
  The words in this dataset (usually) have NER labels as well

  A dataset in three parts from the Starlang group in Turkey:
  Neslihan Kara, Büşra Marşan, et al
    Creating A Syntactically Felicitous Constituency Treebank For Turkish
    https://ieeexplore.ieee.org/document/9259873
  git clone the following three repos
    https://github.com/olcaytaner/TurkishAnnotatedTreeBank-15
    https://github.com/olcaytaner/TurkishAnnotatedTreeBank2-15
    https://github.com/olcaytaner/TurkishAnnotatedTreeBank2-20
  Put them in
    $CONSTITUENCY_HOME/turkish    (yes, the constituency home)
  python3 -m stanza.utils.datasets.ner.prepare_ner_dataset tr_starlang

GermEval2014 is a German NER dataset
  https://sites.google.com/site/germeval2014ner/data
  https://drive.google.com/drive/folders/1kC0I2UGl2ltrluI9NqDjaQJGw5iliw_J
  Download the files in that directory
    NER-de-train.tsv NER-de-dev.tsv NER-de-test.tsv
  put them in
    $NERBASE/germeval2014
  then run
    python3 -m stanza.utils.datasets.ner.prepare_ner_dataset de_germeval2014

The UD Japanese GSD dataset has a conversion by Megagon Labs
  https://github.com/megagonlabs/UD_Japanese-GSD
  https://github.com/megagonlabs/UD_Japanese-GSD/tags
  - r2.9-NE has the NE tagged files inside a "spacy"
    folder in the download
  - expected directory for this data:
    unzip the .zip of the release into
      $NERBASE/ja_gsd
    so it should wind up in
      $NERBASE/ja_gsd/UD_Japanese-GSD-r2.9-NE
    python3 -m stanza.utils.datasets.ner.prepare_ner_dataset ja_gsd

L3Cube is a Marathi dataset
  - https://arxiv.org/abs/2204.06029
    https://arxiv.org/pdf/2204.06029.pdf
    https://github.com/l3cube-pune/MarathiNLP
  - L3Cube-MahaNER: A Marathi Named Entity Recognition Dataset and BERT models
    Parth Patil, Aparna Ranade, Maithili Sabane, Onkar Litake, Raviraj Joshi

  Clone the repo into $NERBASE/marathi
    git clone git@github.com:l3cube-pune/MarathiNLP.git
  Then run
    python3 -m stanza.utils.datasets.ner.prepare_ner_dataset mr_l3cube

Daffodil University produced a Bangla NER dataset
  - https://github.com/Rifat1493/Bengali-NER
  - https://ieeexplore.ieee.org/document/8944804
  - Bengali Named Entity Recognition:
    A survey with deep learning benchmark
    Md Jamiur Rahman Rifat, Sheikh Abujar, Sheak Rashed Haider Noori,
    Syed Akhter Hossain

  Clone the repo into a "bangla" subdirectory of $NERBASE
    cd $NERBASE/bangla
    git clone git@github.com:Rifat1493/Bengali-NER.git
  Then run
    python3 -m stanza.utils.datasets.ner.prepare_ner_dataset bn_daffodil

LST20 is a Thai NER dataset from 2020
  - https://arxiv.org/abs/2008.05055
    The Annotation Guideline of LST20 Corpus
    Prachya Boonkwan, Vorapon Luantangsrisuk, Sitthaa Phaholphinyo,
    Kanyanat Kriengket, Dhanon Leenoi, Charun Phrombut,
    Monthika Boriboon, Krit Kosawat, Thepchai Supnithi
  - This script processes a version which can be downloaded here after registration:
    https://aiforthai.in.th/index.php
  - There is another version downloadable from HuggingFace
    The script will likely need some modification to be compatible
    with the HuggingFace version
  - Download the data in $NERBASE/thai/LST20_Corpus
    There should be "train", "eval", "test" directories after downloading
  - Then run
    pytohn3 -m stanza.utils.datasets.ner.prepare_ner_dataset th_lst20

Thai-NNER is another Thai NER dataset, from 2022
  - https://github.com/vistec-AI/Thai-NNER
  - https://aclanthology.org/2022.findings-acl.116/
    Thai Nested Named Entity Recognition Corpus
    Weerayut Buaphet, Can Udomcharoenchaikit, Peerat Limkonchotiwat,
    Attapol Rutherford, and Sarana Nutanong
  - git clone the data to $NERBASE/thai
  - On the git repo, there should be a link to a more complete version
    of the dataset.  For example, in Sep. 2023 it is here:
    https://github.com/vistec-AI/Thai-NNER#dataset
    The Google drive it goes to has "postproc".
    Put the train.json, dev.json, and test.json in
    $NERBASE/thai/Thai-NNER/data/scb-nner-th-2022/postproc/
  - Then run
    pytohn3 -m stanza.utils.datasets.ner.prepare_ner_dataset th_nner22


NKJP is a Polish NER dataset
  - http://nkjp.pl/index.php?page=0&lang=1
    About the Project
  - http://zil.ipipan.waw.pl/DistrNKJP
    Wikipedia subcorpus used to train charlm model
  - http://clip.ipipan.waw.pl/NationalCorpusOfPolish?action=AttachFile&do=view&target=NKJP-PodkorpusMilionowy-1.2.tar.gz
    Annotated subcorpus to train NER model.
    Download and extract to $NERBASE/Polish-NKJP or leave the gzip in $NERBASE/polish/...

kk_kazNERD is a Kazakh dataset published in 2021
  - https://github.com/IS2AI/KazNERD
  - https://arxiv.org/abs/2111.13419
    KazNERD: Kazakh Named Entity Recognition Dataset
    Rustem Yeshpanov, Yerbolat Khassanov, Huseyin Atakan Varol
  - in $NERBASE, make a "kazakh" directory, then git clone the repo there
    mkdir -p $NERBASE/kazakh
    cd $NERBASE/kazakh
    git clone git@github.com:IS2AI/KazNERD.git
  - Then run
    pytohn3 -m stanza.utils.datasets.ner.prepare_ner_dataset kk_kazNERD

Masakhane NER is a set of NER datasets for African languages
  - MasakhaNER: Named Entity Recognition for African Languages
    Adelani, David Ifeoluwa; Abbott, Jade; Neubig, Graham;
    D’souza, Daniel; Kreutzer, Julia; Lignos, Constantine;
    Palen-Michel, Chester; Buzaaba, Happy; Rijhwani, Shruti;
    Ruder, Sebastian; Mayhew, Stephen; Azime, Israel Abebe;
    Muhammad, Shamsuddeen H.; Emezue, Chris Chinenye;
    Nakatumba-Nabende, Joyce; Ogayo, Perez; Anuoluwapo, Aremu;
    Gitau, Catherine; Mbaye, Derguene; Alabi, Jesujoba;
    Yimam, Seid Muhie; Gwadabe, Tajuddeen Rabiu; Ezeani, Ignatius;
    Niyongabo, Rubungo Andre; Mukiibi, Jonathan; Otiende, Verrah;
    Orife, Iroro; David, Davis; Ngom, Samba; Adewumi, Tosin;
    Rayson, Paul; Adeyemi, Mofetoluwa; Muriuki, Gerald;
    Anebi, Emmanuel; Chukwuneke, Chiamaka; Odu, Nkiruka;
    Wairagala, Eric Peter; Oyerinde, Samuel; Siro, Clemencia;
    Bateesa, Tobius Saul; Oloyede, Temilola; Wambui, Yvonne;
    Akinode, Victor; Nabagereka, Deborah; Katusiime, Maurice;
    Awokoya, Ayodele; MBOUP, Mouhamadane; Gebreyohannes, Dibora;
    Tilaye, Henok; Nwaike, Kelechi; Wolde, Degaga; Faye, Abdoulaye;
    Sibanda, Blessing; Ahia, Orevaoghene; Dossou, Bonaventure F. P.;
    Ogueji, Kelechi; DIOP, Thierno Ibrahima; Diallo, Abdoulaye;
    Akinfaderin, Adewale; Marengereke, Tendai; Osei, Salomey
  - https://github.com/masakhane-io/masakhane-ner
  - git clone the repo to $NERBASE
  - Then run
    python3 -m stanza.utils.datasets.ner.prepare_ner_dataset lcode_masakhane
  - You can use the full language name, the 3 letter language code,
    or in the case of languages with a 2 letter language code,
    the 2 letter code for lcode.  The tool will throw an error
    if the language is not supported in Masakhane.

SiNER is a Sindhi NER dataset
  - https://aclanthology.org/2020.lrec-1.361/
    SiNER: A Large Dataset for Sindhi Named Entity Recognition
    Wazir Ali, Junyu Lu, Zenglin Xu
  - It is available via git repository
    https://github.com/AliWazir/SiNER-dataset
    As of Nov. 2022, there were a few changes to the dataset
    to update a couple instances of broken tags & tokenization
  - Clone the repo to $NERBASE/sindhi
    mkdir $NERBASE/sindhi
    cd $NERBASE/sindhi
    git clone git@github.com:AliWazir/SiNER-dataset.git
  - Then, prepare the dataset with this script:
    python3 -m stanza.utils.datasets.ner.prepare_ner_dataset sd_siner

en_sample is the toy dataset included with stanza-train
  https://github.com/stanfordnlp/stanza-train
  this is not meant for any kind of actual NER use

ArmTDP-NER is an Armenian NER dataset
  - https://github.com/myavrum/ArmTDP-NER.git
    ArmTDP-NER: The corpus was developed by the ArmTDP team led by Marat M. Yavrumyan
    at the Yerevan State University by the collaboration of "Armenia National SDG Innovation Lab"
    and "UC Berkley's Armenian Linguists' network".
  - in $NERBASE, make a "armenian" directory, then git clone the repo there
    mkdir -p $NERBASE/armenian
    cd $NERBASE/armenian
    git clone https://github.com/myavrum/ArmTDP-NER.git
  - Then run
    python3 -m stanza.utils.datasets.ner.prepare_ner_dataset hy_armtdp

en_conll03 is the classic 2003 4 class CoNLL dataset
  - The version we use is posted on HuggingFace
  - https://huggingface.co/datasets/conll2003
  - The prepare script will download from HF
    using the datasets package, then convert to json
  - Introduction to the CoNLL-2003 Shared Task:
    Language-Independent Named Entity Recognition
    Tjong Kim Sang, Erik F. and De Meulder, Fien
  - python3 stanza/utils/datasets/ner/prepare_ner_dataset.py en_conll03

en_conll03ww is CoNLL 03 with Worldwide added to the training data.
  - python3 stanza/utils/datasets/ner/prepare_ner_dataset.py en_conll03ww

en_conllpp is a test set from 2020 newswire
  - https://arxiv.org/abs/2212.09747
  - https://github.com/ShuhengL/acl2023_conllpp
  - Do CoNLL-2003 Named Entity Taggers Still Work Well in 2023?
    Shuheng Liu, Alan Ritter
  - git clone the repo in $NERBASE
  - then run
    python3 stanza/utils/datasets/ner/prepare_ner_dataset.py en_conllpp

en_ontonotes is the OntoNotes 5 on HuggingFace
  - https://huggingface.co/datasets/conll2012_ontonotesv5
  - python3 stanza/utils/datasets/ner/prepare_ner_dataset.py en_ontonotes
  - this downloads the "v12" version of the data

en_worldwide-4class is an English non-US newswire dataset
  - annotated by MLTwist and Aya Data, with help from Datasaur,
    collected at Stanford
  - work to be published at EMNLP Findings
  - the 4 class version is converted to the 4 classes in conll,
    then split into train/dev/test
  - clone https://github.com/stanfordnlp/en-worldwide-newswire
    into $NERBASE/en_worldwide

en_worldwide-9class is an English non-US newswire dataset
  - annotated by MLTwist and Aya Data, with help from Datasaur,
    collected at Stanford
  - work to be published at EMNLP Findings
  - the 9 class version is not edited
  - clone https://github.com/stanfordnlp/en-worldwide-newswire
    into $NERBASE/en_worldwide

zh-hans_ontonotes is the ZH split of the OntoNotes dataset
  - https://catalog.ldc.upenn.edu/LDC2013T19
  - https://huggingface.co/datasets/conll2012_ontonotesv5
  - python3 stanza/utils/datasets/ner/prepare_ner_dataset.py zh-hans_ontonotes
  - this downloads the "v4" version of the data


AQMAR is a small dataset of Arabic Wikipedia articles
  - http://www.cs.cmu.edu/~ark/ArabicNER/
  - Recall-Oriented Learning of Named Entities in Arabic Wikipedia
    Behrang Mohit, Nathan Schneider, Rishav Bhowmick, Kemal Oflazer, and Noah A. Smith.
    In Proceedings of the 13th Conference of the European Chapter of
    the Association for Computational Linguistics, Avignon, France,
    April 2012.
  - download the .zip file there and put it in
    $NERBASE/arabic/AQMAR
  - there is a challenge for it here:
    https://www.topcoder.com/challenges/f3cf483e-a95c-4a7e-83e8-6bdd83174d38
  - alternatively, we just randomly split it ourselves
  - currently, running the following reproduces the random split:
    python3 stanza/utils/datasets/ner/prepare_ner_dataset.py ar_aqmar

IAHLT contains NER for Hebrew in the knesset treebank
  - as of UD 2.14, it is only in the git repo
  - download that git repo to $UDBASE_GIT:
    https://github.com/UniversalDependencies/UD_Hebrew-IAHLTknesset
  - change to the dev branch in that repo
    python3 stanza/utils/datasets/ner/prepare_ner_dataset.py he_iahlt

ang_ewt is an Old English dataset available here:
  https://github.com/dmetola/Old_English-OEDT/tree/main
  As more information, including a citation, will be added here
  - install in NERBASE:
    mkdir $NERBASE/ang
    cd $NERBASE/ang
    git clone git@github.com:dmetola/Old_English-OEDT.git
  - python3 stanza/utils/datasets/ner/prepare_ner_dataset.py ang_ewt
"""

import glob
import os
import json
import random
import re
import shutil
import sys
import tempfile

from stanza.models.common.constant import treebank_to_short_name, lcode2lang, lang_to_langcode, two_to_three_letters
from stanza.models.ner.utils import to_bio2, bio2_to_bioes
import stanza.utils.default_paths as default_paths

from stanza.utils.datasets.common import UnknownDatasetError
from stanza.utils.datasets.ner.preprocess_wikiner import preprocess_wikiner
from stanza.utils.datasets.ner.split_wikiner import split_wikiner, split_wikiner_data
import stanza.utils.datasets.ner.build_en_combined as build_en_combined
import stanza.utils.datasets.ner.conll_to_iob as conll_to_iob
import stanza.utils.datasets.ner.convert_ar_aqmar as convert_ar_aqmar
import stanza.utils.datasets.ner.convert_bn_daffodil as convert_bn_daffodil
import stanza.utils.datasets.ner.convert_bsf_to_beios as convert_bsf_to_beios
import stanza.utils.datasets.ner.convert_bsnlp as convert_bsnlp
import stanza.utils.datasets.ner.convert_en_conll03 as convert_en_conll03
import stanza.utils.datasets.ner.convert_fire_2013 as convert_fire_2013
import stanza.utils.datasets.ner.convert_he_iahlt as convert_he_iahlt
import stanza.utils.datasets.ner.convert_ijc as convert_ijc
import stanza.utils.datasets.ner.convert_kk_kazNERD as convert_kk_kazNERD
import stanza.utils.datasets.ner.convert_lst20 as convert_lst20
import stanza.utils.datasets.ner.convert_nner22 as convert_nner22
import stanza.utils.datasets.ner.convert_mr_l3cube as convert_mr_l3cube
import stanza.utils.datasets.ner.convert_my_ucsy as convert_my_ucsy
import stanza.utils.datasets.ner.convert_ontonotes as convert_ontonotes
import stanza.utils.datasets.ner.convert_rgai as convert_rgai
import stanza.utils.datasets.ner.convert_nytk as convert_nytk
import stanza.utils.datasets.ner.convert_starlang_ner as convert_starlang_ner
import stanza.utils.datasets.ner.convert_nkjp as convert_nkjp
import stanza.utils.datasets.ner.prepare_ner_file as prepare_ner_file
import stanza.utils.datasets.ner.convert_sindhi_siner as convert_sindhi_siner
import stanza.utils.datasets.ner.ontonotes_multitag as ontonotes_multitag
import stanza.utils.datasets.ner.simplify_en_worldwide as simplify_en_worldwide
import stanza.utils.datasets.ner.suc_to_iob as suc_to_iob
import stanza.utils.datasets.ner.suc_conll_to_iob as suc_conll_to_iob
import stanza.utils.datasets.ner.convert_hy_armtdp as convert_hy_armtdp
from stanza.utils.datasets.ner.utils import convert_bioes_to_bio, convert_bio_to_json, get_tags, read_tsv, write_sentences, write_dataset, random_shuffle_by_prefixes, read_prefix_file, combine_files

SHARDS = ('train', 'dev', 'test')

def process_turku(paths, short_name):
    assert short_name == 'fi_turku'
    base_input_path = os.path.join(paths["NERBASE"], "finnish", "turku-ner-corpus", "data", "conll")
    base_output_path = paths["NER_DATA_DIR"]
    for shard in SHARDS:
        input_filename = os.path.join(base_input_path, '%s.tsv' % shard)
        if not os.path.exists(input_filename):
            raise FileNotFoundError('Cannot find %s component of %s in %s' % (shard, short_name, input_filename))
        output_filename = os.path.join(base_output_path, '%s.%s.json' % (short_name, shard))
        prepare_ner_file.process_dataset(input_filename, output_filename)

def process_it_fbk(paths, short_name):
    assert short_name == "it_fbk"
    base_input_path = os.path.join(paths["NERBASE"], short_name)
    csv_file = os.path.join(base_input_path, "all-wiki-split.tsv")
    if not os.path.exists(csv_file):
        raise FileNotFoundError("Cannot find the FBK dataset in its expected location: {}".format(csv_file))
    base_output_path = paths["NER_DATA_DIR"]
    split_wikiner(base_output_path, csv_file, prefix=short_name, suffix="io", shuffle=False, train_fraction=0.8, dev_fraction=0.1)
    convert_bio_to_json(base_output_path, base_output_path, short_name, suffix="io")

def process_suralk_multiner(paths, short_name):
    lang_filenames = {
        "en": "Final_English.txt",
        "si": "Final_Sinhala.txt",
        "ta": "Final_Tamil.txt",
    }
    lang, ending = short_name.split("_")
    assert ending == "suralk"
    assert lang in lang_filenames, "suralk/multiNER only supports %s" % (", ".join(lang_filenames.keys()))
    suralk_path = os.path.join(paths["NERBASE"], "mixed", "multiNER", "nerannotateddatasets.zip")
    if not os.path.exists(suralk_path):
        raise FileNotFoundError("Expected to find the suralk/multiNER dataset in %s" % suralk_path)
    sentences = read_tsv(lang_filenames[lang], text_column=0, annotation_column=1, separator=None, zip_filename=suralk_path)
    print("Read %d sentences from %s::%s" % (len(sentences), suralk_path, lang_filenames[lang]))

    base_output_path = paths["NER_DATA_DIR"]
    split_wikiner_data(base_output_path, sentences, prefix=short_name, suffix="bio", shuffle=True)
    convert_bio_to_json(base_output_path, base_output_path, short_name, suffix="bio")

def process_il_ner(paths, short_name):
    joiner = chr(0x200c)
    def fix_tag(tag):
        if tag == '-':
            return 'O'
        if tag.endswith("'"):
            # not sure the correct fix, but we filed an issue, so hopefully they fix it
            return "O"
        if tag.endswith("NIMI") or tag.endswith("NET"):
            return tag[:2] + "NETI"
        tag = tag.replace(joiner, "").upper()
        if tag.startswith("-"):
            return 'B%s' % tag
        return tag

    def fix_line(line):
        if line == 'O':
            return '-\tO'
        return line

    lang_paths = {
        "hi": "Hindi",
        "or": "Odia",
        "te": "Telugu",
        "ur": "Urdu",
    }
    lang, ending = short_name.split("_")
    assert ending == "ilner"
    assert lang in lang_paths, "IL-NER only supports %s" % (", ".join(lang_paths.keys()))
    ilner_path = os.path.join(paths["NERBASE"], "indic", "IL-NER")
    if not os.path.exists(ilner_path):
        raise FileNotFounderror("Cannot find the IL-NER dataset in its expected location: {}".format(ilner_path))
    ilner_path = os.path.join(ilner_path, "Datasets", lang_paths[lang])
    if not os.path.exists(ilner_path):
        raise FileNotFoundError("IL-NER not in the layout expected: directory not found {}".format(ilner_path))
    filenames = os.listdir(ilner_path)

    base_output_path = paths["NER_DATA_DIR"]
    for shard in SHARDS:
        input_filenames = [x for x in filenames if shard in x]
        if len(input_filenames) == 0:
            raise FileNotFoundError("No %s file in %s" % (shard, ilner_path))
        if len(input_filenames) > 1:
            raise FileNotFoundError("Unexpected multiple files for %s in %s: %s" % (shard, ilner_path, input_filenames))
        input_filename = os.path.join(ilner_path, input_filenames[0])
        int_filename = os.path.join(base_output_path, '%s.%s.tsv' % (short_name, shard))
        output_filename = os.path.join(base_output_path, '%s.%s.json' % (short_name, shard))
        sentences = read_tsv(input_filename, text_column=0, annotation_column=1, remap_tag_fn=fix_tag, remap_line=fix_line)
        print("Loaded %d sentences from %s" % (len(sentences), input_filename))
        write_sentences(int_filename, sentences)

        prepare_ner_file.process_dataset(int_filename, output_filename)

def process_languk(paths, short_name):
    assert short_name == 'uk_languk'
    base_input_path = os.path.join(paths["NERBASE"], 'lang-uk', 'ner-uk', 'data')
    base_output_path = paths["NER_DATA_DIR"]
    train_test_split_fname = os.path.join(paths["NERBASE"], 'lang-uk', 'ner-uk', 'doc', 'dev-test-split.txt')
    convert_bsf_to_beios.convert_bsf_in_folder(base_input_path, base_output_path, train_test_split_file=train_test_split_fname)
    for shard in SHARDS:
        input_filename = os.path.join(base_output_path, convert_bsf_to_beios.CORPUS_NAME, "%s.bio" % shard)
        if not os.path.exists(input_filename):
            raise FileNotFoundError('Cannot find %s component of %s in %s' % (shard, short_name, input_filename))
        output_filename = os.path.join(base_output_path, '%s.%s.json' % (short_name, shard))
        prepare_ner_file.process_dataset(input_filename, output_filename)


def process_ijc(paths, short_name):
    """
    Splits the ijc Hindi dataset in train, dev, test

    The original data had train & test splits, so we randomly divide
    the files in train to make a dev set.

    The expected location of the IJC data is hi_ijc.  This method
    should be possible to use for other languages, but we have very
    little support for the other languages of IJC at the moment.
    """
    base_input_path = os.path.join(paths["NERBASE"], short_name)
    base_output_path = paths["NER_DATA_DIR"]

    test_files = [os.path.join(base_input_path, "test-data-hindi.txt")]
    test_csv_file = os.path.join(base_output_path, short_name + ".test.csv")
    print("Converting test input %s to space separated file in %s" % (test_files[0], test_csv_file))
    convert_ijc.convert_ijc(test_files, test_csv_file)

    train_input_path = os.path.join(base_input_path, "training-hindi", "*utf8")
    train_files = glob.glob(train_input_path)
    train_csv_file = os.path.join(base_output_path, short_name + ".train.csv")
    dev_csv_file = os.path.join(base_output_path, short_name + ".dev.csv")
    print("Converting training input from %s to space separated files in %s and %s" % (train_input_path, train_csv_file, dev_csv_file))
    convert_ijc.convert_split_ijc(train_files, train_csv_file, dev_csv_file)

    for csv_file, shard in zip((train_csv_file, dev_csv_file, test_csv_file), SHARDS):
        output_filename = os.path.join(base_output_path, '%s.%s.json' % (short_name, shard))
        prepare_ner_file.process_dataset(csv_file, output_filename)


def process_fire_2013(paths, dataset):
    """
    Splits the FIRE 2013 dataset into train, dev, test

    The provided datasets are all mixed together at this point, so it
    is not possible to recreate the original test conditions used in
    the bakeoff
    """
    short_name = treebank_to_short_name(dataset)
    langcode, _ = short_name.split("_")
    short_name = "%s_fire2013" % langcode
    if not langcode in ("hi", "en", "ta", "bn", "mal"):
        raise UnkonwnDatasetError(dataset, "Language %s not one of the FIRE 2013 languages" % langcode)
    language = lcode2lang[langcode].lower()
    
    # for example, FIRE2013/hindi_train
    base_input_path = os.path.join(paths["NERBASE"], "FIRE2013", "%s_train" % language)
    base_output_path = paths["NER_DATA_DIR"]

    train_csv_file = os.path.join(base_output_path, "%s.train.csv" % short_name)
    dev_csv_file   = os.path.join(base_output_path, "%s.dev.csv" % short_name)
    test_csv_file  = os.path.join(base_output_path, "%s.test.csv" % short_name)

    convert_fire_2013.convert_fire_2013(base_input_path, train_csv_file, dev_csv_file, test_csv_file)

    for csv_file, shard in zip((train_csv_file, dev_csv_file, test_csv_file), SHARDS):
        output_filename = os.path.join(base_output_path, '%s.%s.json' % (short_name, shard))
        prepare_ner_file.process_dataset(csv_file, output_filename)

def process_wikiner(paths, dataset):
    short_name = treebank_to_short_name(dataset)

    base_input_path = os.path.join(paths["NERBASE"], dataset)
    base_output_path = paths["NER_DATA_DIR"]

    expected_filename = "aij*wikiner*"
    input_files = [x for x in glob.glob(os.path.join(base_input_path, expected_filename)) if not x.endswith("bz2")]
    if len(input_files) == 0:
        raw_input_path = os.path.join(base_input_path, "raw")
        input_files = [x for x in glob.glob(os.path.join(raw_input_path, expected_filename)) if not x.endswith("bz2")]
        if len(input_files) > 1:
            raise FileNotFoundError("Found too many raw wikiner files in %s: %s" % (raw_input_path, ", ".join(input_files)))
    elif len(input_files) > 1:
        raise FileNotFoundError("Found too many raw wikiner files in %s: %s" % (base_input_path, ", ".join(input_files)))

    if len(input_files) == 0:
        raise FileNotFoundError("Could not find any raw wikiner files in %s or %s" % (base_input_path, raw_input_path))

    csv_file = os.path.join(base_output_path, short_name + "_csv")
    print("Converting raw input %s to space separated file in %s" % (input_files[0], csv_file))
    try:
        preprocess_wikiner(input_files[0], csv_file)
    except UnicodeDecodeError:
        preprocess_wikiner(input_files[0], csv_file, encoding="iso8859-1")

    # this should create train.bio, dev.bio, and test.bio
    print("Splitting %s to %s" % (csv_file, base_output_path))
    split_wikiner(base_output_path, csv_file, prefix=short_name)
    convert_bio_to_json(base_output_path, base_output_path, short_name)

def process_french_wikiner_gold(paths, dataset):
    short_name = treebank_to_short_name(dataset)

    base_input_path = os.path.join(paths["NERBASE"], "wikiner-fr-gold")
    base_output_path = paths["NER_DATA_DIR"]

    input_filename = os.path.join(base_input_path, "wikiner-fr-gold.conll")
    if not os.path.exists(input_filename):
        raise FileNotFoundError("Could not find the expected input file %s for dataset %s" % (input_filename, base_input_path))

    print("Reading %s" % input_filename)
    sentences = read_tsv(input_filename, text_column=0, annotation_column=2, separator=" ")
    print("Read %d sentences" % len(sentences))

    tags = [y for sentence in sentences for x, y in sentence]
    tags = sorted(set(tags))
    print("Found the following tags:\n%s" % tags)
    expected_tags = ['B-LOC', 'B-MISC', 'B-ORG', 'B-PER',
                     'E-LOC', 'E-MISC', 'E-ORG', 'E-PER',
                     'I-LOC', 'I-MISC', 'I-ORG', 'I-PER',
                     'O',
                     'S-LOC', 'S-MISC', 'S-ORG', 'S-PER']
    assert tags == expected_tags

    output_filename = os.path.join(base_output_path, "%s.full.bioes" % short_name)
    print("Writing BIOES to %s" % output_filename)
    write_sentences(output_filename, sentences)

    print("Splitting %s to %s" % (output_filename, base_output_path))
    split_wikiner(base_output_path, output_filename, prefix=short_name, suffix="bioes")
    convert_bioes_to_bio(base_output_path, base_output_path, short_name)
    convert_bio_to_json(base_output_path, base_output_path, short_name, suffix="bioes")

def process_french_wikiner_mixed(paths, dataset):
    """
    Build both the original and gold edited versions of WikiNER, then mix them

    First we eliminate any duplicates (with one exception), then we combine the data

    There are two main ways we could have done this:
      - mix it together without any restrictions
      - use the multi_ner mechanism to build a dataset which represents two prediction heads

    The second method seems to give slightly better results than the first method,
    but neither beat just using a transformer on the gold set alone

    On the randomly selected test set, using WV and charlm but not a transformer
    (this was on a previously published version of the dataset):

    one prediction head:
      INFO: Score by entity:
        Prec.   Rec.    F1
        89.32   89.26   89.29
      INFO: Score by token:
        Prec.   Rec.    F1
        89.43   86.88   88.14
      INFO: Weighted f1 for non-O tokens: 0.878855

    two prediction heads:
      INFO: Score by entity:
        Prec.   Rec.    F1
        89.83   89.76   89.79
      INFO: Score by token:
        Prec.   Rec.    F1
        89.17   88.15   88.66
      INFO: Weighted f1 for non-O tokens: 0.885675

    On a randomly selected dev set, using transformer:

    gold:
      INFO: Score by entity:
        Prec.   Rec.    F1
        93.63   93.98   93.81
      INFO: Score by token:
        Prec.   Rec.    F1
        92.80   92.79   92.80
      INFO: Weighted f1 for non-O tokens: 0.927548

    mixed:
      INFO: Score by entity:
        Prec.   Rec.    F1
        93.54   93.82   93.68
      INFO: Score by token:
        Prec.   Rec.    F1
        92.99   92.51   92.75
      INFO: Weighted f1 for non-O tokens: 0.926964
    """
    short_name = treebank_to_short_name(dataset)

    process_french_wikiner_gold(paths, "fr_wikinergold")
    process_wikiner(paths, "French-WikiNER")
    base_output_path = paths["NER_DATA_DIR"]

    with open(os.path.join(base_output_path, "fr_wikinergold.train.json")) as fin:
        gold_train = json.load(fin)
    with open(os.path.join(base_output_path, "fr_wikinergold.dev.json")) as fin:
        gold_dev = json.load(fin)
    with open(os.path.join(base_output_path, "fr_wikinergold.test.json")) as fin:
        gold_test = json.load(fin)

    gold = gold_train + gold_dev + gold_test
    print("%d total sentences in the gold relabeled dataset (randomly split)" % len(gold))
    gold = {tuple([x["text"] for x in sentence]): sentence for sentence in gold}
    print("  (%d after dedup)" % len(gold))

    original = (read_tsv(os.path.join(base_output_path, "fr_wikiner.train.bio"), text_column=0, annotation_column=1) +
                read_tsv(os.path.join(base_output_path, "fr_wikiner.dev.bio"), text_column=0, annotation_column=1) +
                read_tsv(os.path.join(base_output_path, "fr_wikiner.test.bio"), text_column=0, annotation_column=1))
    print("%d total sentences in the original wiki" % len(original))
    original_words = {tuple([x[0] for x in sentence]) for sentence in original}
    print("  (%d after dedup)" % len(original_words))

    missing = [sentence for sentence in gold if sentence not in original_words]
    for sentence in missing:
        # the capitalization of WisiGoths and OstroGoths is different
        # between the original and the new in some cases
        goths = tuple([x.replace("Goth", "goth") for x in sentence])
        if goths != sentence and goths in original_words:
            original_words.add(sentence)
    missing = [sentence for sentence in gold if sentence not in original_words]
    # currently this dataset doesn't find two sentences
    # one was dropped by the filter for incompletely tagged lines
    # the other is probably not a huge deal to have one duplicate
    print("Missing %d sentences" % len(missing))
    assert len(missing) <= 2
    for sent in missing:
        print(sent)

    skipped = 0
    silver = []
    silver_used = set()
    for sentence in original:
        words = tuple([x[0] for x in sentence])
        tags = tuple([x[1] for x in sentence])
        if words in gold or words in silver_used:
            skipped += 1
            continue
        tags = to_bio2(tags)
        tags = bio2_to_bioes(tags)
        sentence = [{"text": x, "ner": y, "multi_ner": ["-", y]} for x, y in zip(words, tags)]
        silver.append(sentence)
        silver_used.add(words)
    print("Using %d sentences from the original wikiner alongside the gold annotated train set" % len(silver))
    print("Skipped %d sentences" % skipped)

    gold_train = [[{"text": x["text"], "ner": x["ner"], "multi_ner": [x["ner"], "-"]} for x in sentence]
                  for sentence in gold_train]
    gold_dev = [[{"text": x["text"], "ner": x["ner"], "multi_ner": [x["ner"], "-"]} for x in sentence]
                  for sentence in gold_dev]
    gold_test = [[{"text": x["text"], "ner": x["ner"], "multi_ner": [x["ner"], "-"]} for x in sentence]
                  for sentence in gold_test]

    mixed_train = gold_train + silver
    print("Total sentences in the mixed training set: %d" % len(mixed_train))
    output_filename = os.path.join(base_output_path, "%s.train.json" % short_name)
    with open(output_filename, 'w', encoding='utf-8') as fout:
        json.dump(mixed_train, fout, indent=1)

    output_filename = os.path.join(base_output_path, "%s.dev.json" % short_name)
    with open(output_filename, 'w', encoding='utf-8') as fout:
        json.dump(gold_dev, fout, indent=1)
    output_filename = os.path.join(base_output_path, "%s.test.json" % short_name)
    with open(output_filename, 'w', encoding='utf-8') as fout:
        json.dump(gold_test, fout, indent=1)


def get_rgai_input_path(paths):
    return os.path.join(paths["NERBASE"], "hu_rgai")

def process_rgai(paths, short_name):
    base_output_path = paths["NER_DATA_DIR"]
    base_input_path = get_rgai_input_path(paths)

    if short_name == 'hu_rgai':
        use_business = True
        use_criminal = True
    elif short_name == 'hu_rgai_business':
        use_business = True
        use_criminal = False
    elif short_name == 'hu_rgai_criminal':
        use_business = False
        use_criminal = True
    else:
        raise UnknownDatasetError(short_name, "Unknown subset of hu_rgai data: %s" % short_name)

    convert_rgai.convert_rgai(base_input_path, base_output_path, short_name, use_business, use_criminal)
    convert_bio_to_json(base_output_path, base_output_path, short_name)

def get_nytk_input_path(paths):
    return os.path.join(paths["NERBASE"], "NYTK-NerKor")

def process_nytk(paths, short_name):
    """
    Process the NYTK dataset
    """
    assert short_name == "hu_nytk"
    base_output_path = paths["NER_DATA_DIR"]
    base_input_path = get_nytk_input_path(paths)

    convert_nytk.convert_nytk(base_input_path, base_output_path, short_name)
    convert_bio_to_json(base_output_path, base_output_path, short_name)

def concat_files(output_file, *input_files):
    input_lines = []
    for input_file in input_files:
        with open(input_file) as fin:
            lines = fin.readlines()
        if not len(lines):
            raise ValueError("Empty input file: %s" % input_file)
        if not lines[-1]:
            lines[-1] = "\n"
        elif lines[-1].strip():
            lines.append("\n")
        input_lines.append(lines)
    with open(output_file, "w") as fout:
        for lines in input_lines:
            for line in lines:
                fout.write(line)


def process_hu_combined(paths, short_name):
    assert short_name == "hu_combined"

    base_output_path = paths["NER_DATA_DIR"]
    rgai_input_path = get_rgai_input_path(paths)
    nytk_input_path = get_nytk_input_path(paths)

    with tempfile.TemporaryDirectory() as tmp_output_path:
        convert_rgai.convert_rgai(rgai_input_path, tmp_output_path, "hu_rgai", True, True)
        convert_nytk.convert_nytk(nytk_input_path, tmp_output_path, "hu_nytk")

        for shard in SHARDS:
            rgai_input = os.path.join(tmp_output_path, "hu_rgai.%s.bio" % shard)
            nytk_input = os.path.join(tmp_output_path, "hu_nytk.%s.bio" % shard)
            output_file = os.path.join(base_output_path, "hu_combined.%s.bio" % shard)
            concat_files(output_file, rgai_input, nytk_input)

    convert_bio_to_json(base_output_path, base_output_path, short_name)

def process_bsnlp(paths, short_name):
    """
    Process files downloaded from http://bsnlp.cs.helsinki.fi/bsnlp-2019/shared_task.html

    If you download the training and test data zip files and unzip
    them without rearranging in any way, the layout is somewhat weird.
    Training data goes into a specific subdirectory, but the test data
    goes into the top level directory.
    """
    base_input_path = os.path.join(paths["NERBASE"], "bsnlp2019")
    base_train_path = os.path.join(base_input_path, "training_pl_cs_ru_bg_rc1")
    base_test_path = base_input_path

    base_output_path = paths["NER_DATA_DIR"]

    output_train_filename = os.path.join(base_output_path, "%s.train.csv" % short_name)
    output_dev_filename   = os.path.join(base_output_path, "%s.dev.csv" % short_name)
    output_test_filename  = os.path.join(base_output_path, "%s.test.csv" % short_name)

    language = short_name.split("_")[0]

    convert_bsnlp.convert_bsnlp(language, base_test_path, output_test_filename)
    convert_bsnlp.convert_bsnlp(language, base_train_path, output_train_filename, output_dev_filename)

    for shard, csv_file in zip(SHARDS, (output_train_filename, output_dev_filename, output_test_filename)):
        output_filename = os.path.join(base_output_path, '%s.%s.json' % (short_name, shard))
        prepare_ner_file.process_dataset(csv_file, output_filename)

NCHLT_LANGUAGE_MAP = {
    "af":  "NCHLT Afrikaans Named Entity Annotated Corpus",
    # none of the following have UD datasets as of 2.8.  Until they
    # exist, we assume the language codes NCHTL are sufficient
    "nr":  "NCHLT isiNdebele Named Entity Annotated Corpus",
    "nso": "NCHLT Sepedi Named Entity Annotated Corpus",
    "ss":  "NCHLT Siswati Named Entity Annotated Corpus",
    "st":  "NCHLT Sesotho Named Entity Annotated Corpus",
    "tn":  "NCHLT Setswana Named Entity Annotated Corpus",
    "ts":  "NCHLT Xitsonga Named Entity Annotated Corpus",
    "ve":  "NCHLT Tshivenda Named Entity Annotated Corpus",
    "xh":  "NCHLT isiXhosa Named Entity Annotated Corpus",
    "zu":  "NCHLT isiZulu Named Entity Annotated Corpus",
}

def process_nchlt(paths, short_name):
    language = short_name.split("_")[0]
    if not language in NCHLT_LANGUAGE_MAP:
        raise UnknownDatasetError(short_name, "Language %s not part of NCHLT" % language)
    short_name = "%s_nchlt" % language

    base_input_path = os.path.join(paths["NERBASE"], "NCHLT", NCHLT_LANGUAGE_MAP[language], "*Full.txt")
    input_files = glob.glob(base_input_path)
    if len(input_files) == 0:
        raise FileNotFoundError("Cannot find NCHLT dataset in '%s'  Did you remember to download the file?" % base_input_path)

    if len(input_files) > 1:
        raise ValueError("Unexpected number of files matched '%s'  There should only be one" % base_input_path)

    base_output_path = paths["NER_DATA_DIR"]
    split_wikiner(base_output_path, input_files[0], prefix=short_name, remap={"OUT": "O"})
    convert_bio_to_json(base_output_path, base_output_path, short_name)

def process_my_ucsy(paths, short_name):
    assert short_name == "my_ucsy"
    language = "my"

    base_input_path = os.path.join(paths["NERBASE"], short_name)
    base_output_path = paths["NER_DATA_DIR"]
    convert_my_ucsy.convert_my_ucsy(base_input_path, base_output_path)
    convert_bio_to_json(base_output_path, base_output_path, short_name)

def process_fa_arman(paths, short_name):
    """
    Converts fa_arman dataset

    The conversion is quite simple, actually.
    Just need to split the train file and then convert bio -> json
    """
    assert short_name == "fa_arman"
    language = "fa"
    base_input_path = os.path.join(paths["NERBASE"], "PersianNER")
    train_input_file = os.path.join(base_input_path, "train_fold1.txt")
    test_input_file = os.path.join(base_input_path, "test_fold1.txt")
    if not os.path.exists(train_input_file) or not os.path.exists(test_input_file):
        full_corpus_file = os.path.join(base_input_path, "ArmanPersoNERCorpus.zip")
        if os.path.exists(full_corpus_file):
            raise FileNotFoundError("Please unzip the file {}".format(full_corpus_file))
        raise FileNotFoundError("Cannot find the arman corpus in the expected directory: {}".format(base_input_path))

    base_output_path = paths["NER_DATA_DIR"]
    test_output_file = os.path.join(base_output_path, "%s.test.bio" % short_name)

    split_wikiner(base_output_path, train_input_file, prefix=short_name, train_fraction=0.8, test_section=False)
    shutil.copy2(test_input_file, test_output_file)
    convert_bio_to_json(base_output_path, base_output_path, short_name)

def process_sv_suc3licensed(paths, short_name):
    """
    The .zip provided for SUC3 includes train/dev/test splits already

    This extracts those splits without needing to unzip the original file
    """
    assert short_name == "sv_suc3licensed"
    language = "sv"
    train_input_file = os.path.join(paths["NERBASE"], short_name, "SUC3.0.zip")
    if not os.path.exists(train_input_file):
        raise FileNotFoundError("Cannot find the officially licensed SUC3 dataset in %s" % train_input_file)

    base_output_path = paths["NER_DATA_DIR"]
    suc_conll_to_iob.process_suc3(train_input_file, short_name, base_output_path)
    convert_bio_to_json(base_output_path, base_output_path, short_name)

def process_sv_suc3shuffle(paths, short_name):
    """
    Uses an externally provided script to read the SUC3 XML file, then splits it
    """
    assert short_name == "sv_suc3shuffle"
    language = "sv"
    train_input_file = os.path.join(paths["NERBASE"], short_name, "suc3.xml.bz2")
    if not os.path.exists(train_input_file):
        train_input_file = train_input_file[:-4]
    if not os.path.exists(train_input_file):
        raise FileNotFoundError("Unable to find the SUC3 dataset in {}.bz2".format(train_input_file))

    base_output_path = paths["NER_DATA_DIR"]
    train_output_file = os.path.join(base_output_path, "sv_suc3shuffle.bio")
    suc_to_iob.main([train_input_file, train_output_file])
    split_wikiner(base_output_path, train_output_file, prefix=short_name)
    convert_bio_to_json(base_output_path, base_output_path, short_name)    
    
def process_da_ddt(paths, short_name):
    """
    Processes Danish DDT dataset

    This dataset is in a conll file with the "name" attribute in the
    misc column for the NER tag.  This function uses a script to
    convert such CoNLL files to .bio
    """
    assert short_name == "da_ddt"
    language = "da"
    IN_FILES = ("ddt.train.conllu", "ddt.dev.conllu", "ddt.test.conllu")

    base_output_path = paths["NER_DATA_DIR"]
    OUT_FILES = [os.path.join(base_output_path, "%s.%s.bio" % (short_name, shard)) for shard in SHARDS]

    zip_file = os.path.join(paths["NERBASE"], "da_ddt", "ddt.zip")
    if os.path.exists(zip_file):
        for in_filename, out_filename, shard in zip(IN_FILES, OUT_FILES, SHARDS):
            conll_to_iob.process_conll(in_filename, out_filename, zip_file)
    else:
        for in_filename, out_filename, shard in zip(IN_FILES, OUT_FILES, SHARDS):
            in_filename = os.path.join(paths["NERBASE"], "da_ddt", in_filename)
            if not os.path.exists(in_filename):
                raise FileNotFoundError("Could not find zip in expected location %s and could not file %s file in %s" % (zip_file, shard, in_filename))

            conll_to_iob.process_conll(in_filename, out_filename)
    convert_bio_to_json(base_output_path, base_output_path, short_name)


def process_norne(paths, short_name):
    """
    Processes Norwegian NorNE

    Can handle either Bokmål or Nynorsk

    Converts GPE_LOC and GPE_ORG to GPE
    """
    language, name = short_name.split("_", 1)
    assert language in ('nb', 'nn')
    assert name == 'norne'

    if language == 'nb':
        IN_FILES = ("nob/no_bokmaal-ud-train.conllu", "nob/no_bokmaal-ud-dev.conllu", "nob/no_bokmaal-ud-test.conllu")
    else:
        IN_FILES = ("nno/no_nynorsk-ud-train.conllu", "nno/no_nynorsk-ud-dev.conllu", "nno/no_nynorsk-ud-test.conllu")

    base_output_path = paths["NER_DATA_DIR"]
    OUT_FILES = [os.path.join(base_output_path, "%s.%s.bio" % (short_name, shard)) for shard in SHARDS]

    CONVERSION = { "GPE_LOC": "GPE", "GPE_ORG": "GPE" }

    for in_filename, out_filename, shard in zip(IN_FILES, OUT_FILES, SHARDS):
        in_filename = os.path.join(paths["NERBASE"], "norne", "ud", in_filename)
        if not os.path.exists(in_filename):
            raise FileNotFoundError("Could not find %s file in %s" % (shard, in_filename))

        conll_to_iob.process_conll(in_filename, out_filename, conversion=CONVERSION)

    convert_bio_to_json(base_output_path, base_output_path, short_name)

def process_ja_gsd(paths, short_name):
    """
    Convert ja_gsd from MegagonLabs

    for example, can download from https://github.com/megagonlabs/UD_Japanese-GSD/releases/tag/r2.9-NE
    """
    language, name = short_name.split("_", 1)
    assert language == 'ja'
    assert name == 'gsd'

    base_output_path = paths["NER_DATA_DIR"]
    output_files = [os.path.join(base_output_path, "%s.%s.bio" % (short_name, shard)) for shard in SHARDS]

    search_path = os.path.join(paths["NERBASE"], "ja_gsd", "UD_Japanese-GSD-r2.*-NE")
    versions = glob.glob(search_path)
    max_version = None
    base_input_path = None
    version_re = re.compile("GSD-r2.([0-9]+)-NE$")

    for ver in versions:
        match = version_re.search(ver)
        if not match:
            continue
        ver_num = int(match.groups(1)[0])
        if max_version is None or ver_num > max_version:
            max_version = ver_num
            base_input_path = ver

    if base_input_path is None:
        raise FileNotFoundError("Could not find any copies of the NE conversion of ja_gsd here: {}".format(search_path))
    print("Most recent version found: {}".format(base_input_path))

    input_files = ["ja_gsd-ud-train.ne.conllu", "ja_gsd-ud-dev.ne.conllu", "ja_gsd-ud-test.ne.conllu"]

    def conversion(x):
        if x[0] == 'L':
            return 'E' + x[1:]
        if x[0] == 'U':
            return 'S' + x[1:]
        # B, I unchanged
        return x

    for in_filename, out_filename, shard in zip(input_files, output_files, SHARDS):
        in_path = os.path.join(base_input_path, in_filename)
        if not os.path.exists(in_path):
            in_spacy = os.path.join(base_input_path, "spacy", in_filename)
            if not os.path.exists(in_spacy):
                raise FileNotFoundError("Could not find %s file in %s or %s" % (shard, in_path, in_spacy))
            in_path = in_spacy

        conll_to_iob.process_conll(in_path, out_filename, conversion=conversion, allow_empty=True, attr_prefix="NE")

    convert_bio_to_json(base_output_path, base_output_path, short_name)

def process_starlang(paths, short_name):
    """
    Process a Turkish dataset from Starlang
    """
    assert short_name == 'tr_starlang'

    PIECES = ["TurkishAnnotatedTreeBank-15",
              "TurkishAnnotatedTreeBank2-15",
              "TurkishAnnotatedTreeBank2-20"]

    chunk_paths = [os.path.join(paths["CONSTITUENCY_BASE"], "turkish", piece) for piece in PIECES]
    datasets = convert_starlang_ner.read_starlang(chunk_paths)

    write_dataset(datasets, paths["NER_DATA_DIR"], short_name)

def remap_germeval_tag(tag):
    """
    Simplify tags for GermEval2014 using a simple rubric

    all tags become their parent tag
    OTH becomes MISC
    """
    if tag == "O":
        return tag
    if tag[1:5] == "-LOC":
        return tag[:5]
    if tag[1:5] == "-PER":
        return tag[:5]
    if tag[1:5] == "-ORG":
        return tag[:5]
    if tag[1:5] == "-OTH":
        return tag[0] + "-MISC"
    raise ValueError("Unexpected tag: %s" % tag)

def process_de_germeval2014(paths, short_name):
    """
    Process the TSV of the GermEval2014 dataset
    """
    in_directory = os.path.join(paths["NERBASE"], "germeval2014")
    base_output_path = paths["NER_DATA_DIR"]
    datasets = []
    for shard in SHARDS:
        in_file = os.path.join(in_directory, "NER-de-%s.tsv" % shard)
        sentences = read_tsv(in_file, 1, 2, remap_tag_fn=remap_germeval_tag)
        datasets.append(sentences)
    tags = get_tags(datasets)
    print("Found the following tags: {}".format(sorted(tags)))
    write_dataset(datasets, base_output_path, short_name)

def process_hiner(paths, short_name):
    in_directory = os.path.join(paths["NERBASE"], "hindi", "HiNER", "data", "original")
    convert_bio_to_json(in_directory, paths["NER_DATA_DIR"], short_name, suffix="conll", shard_names=("train", "validation", "test"))

def process_hinercollapsed(paths, short_name):
    in_directory = os.path.join(paths["NERBASE"], "hindi", "HiNER", "data", "collapsed")
    convert_bio_to_json(in_directory, paths["NER_DATA_DIR"], short_name, suffix="conll", shard_names=("train", "validation", "test"))

def process_lst20(paths, short_name, include_space_char=True):
    convert_lst20.convert_lst20(paths, short_name, include_space_char)

def process_nner22(paths, short_name, include_space_char=True):
    convert_nner22.convert_nner22(paths, short_name, include_space_char)

def process_mr_l3cube(paths, short_name):
    base_output_path = paths["NER_DATA_DIR"]
    in_directory = os.path.join(paths["NERBASE"], "marathi", "MarathiNLP", "L3Cube-MahaNER", "IOB")
    input_files = ["train_iob.txt", "valid_iob.txt", "test_iob.txt"]
    input_files = [os.path.join(in_directory, x) for x in input_files]
    for input_file in input_files:
        if not os.path.exists(input_file):
            raise FileNotFoundError("Could not find the expected piece of the l3cube dataset %s" % input_file)

    datasets = [convert_mr_l3cube.convert(input_file) for input_file in input_files]
    write_dataset(datasets, base_output_path, short_name)

def process_bn_daffodil(paths, short_name):
    in_directory = os.path.join(paths["NERBASE"], "bangla", "Bengali-NER")
    out_directory = paths["NER_DATA_DIR"]
    convert_bn_daffodil.convert_dataset(in_directory, out_directory)

def process_pl_nkjp(paths, short_name):
    out_directory = paths["NER_DATA_DIR"]
    candidates = [os.path.join(paths["NERBASE"], "Polish-NKJP"),
                  os.path.join(paths["NERBASE"], "polish", "Polish-NKJP"),
                  os.path.join(paths["NERBASE"], "polish", "NKJP-PodkorpusMilionowy-1.2.tar.gz"),]
    for in_path in candidates:
        if os.path.exists(in_path):
            break
    else:
        raise FileNotFoundError("Could not find %s  Looked in %s" % (short_name, " ".join(candidates)))
    convert_nkjp.convert_nkjp(in_path, out_directory)

def process_kk_kazNERD(paths, short_name):
    in_directory = os.path.join(paths["NERBASE"], "kazakh", "KazNERD", "KazNERD")
    out_directory = paths["NER_DATA_DIR"]
    convert_kk_kazNERD.convert_dataset(in_directory, out_directory, short_name)

def process_masakhane(paths, dataset_name):
    """
    Converts Masakhane NER datasets to Stanza's .json format

    If we let N be the length of the first sentence, the NER files
    (in version 2, at least) are all of the form

    word tag
    ...
    word tag
      (blank line for sentence break)
    word tag
    ...

    Once the dataset is git cloned in $NERBASE, the directory structure is

    $NERBASE/masakhane-ner/MasakhaNER2.0/data/$lcode/{train,dev,test}.txt

    The only tricky thing here is that for some languages, we treat
    the 2 letter lcode as canonical thanks to UD, but Masakhane NER
    uses 3 letter lcodes for all languages.
    """
    language, dataset = dataset_name.split("_")
    lcode = lang_to_langcode(language)
    if lcode in two_to_three_letters:
        masakhane_lcode = two_to_three_letters[lcode]
    else:
        masakhane_lcode = lcode

    mn_directory = os.path.join(paths["NERBASE"], "masakhane-ner")
    if not os.path.exists(mn_directory):
        raise FileNotFoundError("Cannot find Masakhane NER repo.  Please check the setting of NERBASE or clone the repo to %s" % mn_directory)
    data_directory = os.path.join(mn_directory, "MasakhaNER2.0", "data")
    if not os.path.exists(data_directory):
        raise FileNotFoundError("Apparently found the repo at %s but the expected directory structure is not there - was looking for %s" % (mn_directory, data_directory))

    in_directory = os.path.join(data_directory, masakhane_lcode)
    if not os.path.exists(in_directory):
        raise UnknownDatasetError(dataset_name, "Found the Masakhane repo, but there was no %s in the repo at path %s" % (dataset_name, in_directory))
    convert_bio_to_json(in_directory, paths["NER_DATA_DIR"], "%s_masakhane" % lcode, "txt")

def process_sd_siner(paths, short_name):
    in_directory = os.path.join(paths["NERBASE"], "sindhi", "SiNER-dataset")
    if not os.path.exists(in_directory):
        raise FileNotFoundError("Cannot find SiNER checkout in $NERBASE/sindhi  Please git clone to repo in that directory")
    in_filename = os.path.join(in_directory, "SiNER-dataset.txt")
    if not os.path.exists(in_filename):
        in_filename = os.path.join(in_directory, "SiNER dataset.txt")
        if not os.path.exists(in_filename):
            raise FileNotFoundError("Found an SiNER directory at %s but the directory did not contain the dataset" % in_directory)
    convert_sindhi_siner.convert_sindhi_siner(in_filename, paths["NER_DATA_DIR"], short_name)

def process_en_worldwide_4class(paths, short_name):
    simplify_en_worldwide.main(args=['--simplify'])

    in_directory = os.path.join(paths["NERBASE"], "en_worldwide", "4class")
    out_directory = paths["NER_DATA_DIR"]

    destination_file = os.path.join(paths["NERBASE"], "en_worldwide", "en-worldwide-newswire", "regions.txt")
    prefix_map = read_prefix_file(destination_file)

    random_shuffle_by_prefixes(in_directory, out_directory, short_name, prefix_map)

def process_en_worldwide_9class(paths, short_name):
    simplify_en_worldwide.main(args=['--no_simplify'])

    in_directory = os.path.join(paths["NERBASE"], "en_worldwide", "9class")
    out_directory = paths["NER_DATA_DIR"]

    destination_file = os.path.join(paths["NERBASE"], "en_worldwide", "en-worldwide-newswire", "regions.txt")
    prefix_map = read_prefix_file(destination_file)

    random_shuffle_by_prefixes(in_directory, out_directory, short_name, prefix_map)

def process_en_ontonotes(paths, short_name):
    ner_input_path = paths['NERBASE']
    ontonotes_path = os.path.join(ner_input_path, "english", "en_ontonotes")
    ner_output_path = paths['NER_DATA_DIR']
    convert_ontonotes.process_dataset("en_ontonotes", ontonotes_path, ner_output_path)

def process_zh_ontonotes(paths, short_name):
    ner_input_path = paths['NERBASE']
    ontonotes_path = os.path.join(ner_input_path, "chinese", "zh_ontonotes")
    ner_output_path = paths['NER_DATA_DIR']
    convert_ontonotes.process_dataset(short_name, ontonotes_path, ner_output_path)

def process_en_conll03(paths, short_name):
    ner_input_path = paths['NERBASE']
    conll_path = os.path.join(ner_input_path, "english", "en_conll03")
    ner_output_path = paths['NER_DATA_DIR']
    convert_en_conll03.process_dataset("en_conll03", conll_path, ner_output_path)

def process_en_conll03_worldwide(paths, short_name):
    """
    Adds the training data for conll03 and worldwide together
    """
    print("============== Preparing CoNLL 2003 ===================")
    process_en_conll03(paths, "en_conll03")
    print("========== Preparing 4 Class Worldwide ================")
    process_en_worldwide_4class(paths, "en_worldwide-4class")
    print("============== Combined Train Data ====================")
    input_files = [os.path.join(paths['NER_DATA_DIR'], "en_conll03.train.json"),
                   os.path.join(paths['NER_DATA_DIR'], "en_worldwide-4class.train.json")]
    output_file = os.path.join(paths['NER_DATA_DIR'], "%s.train.json" % short_name)
    combine_files(output_file, *input_files)
    shutil.copyfile(os.path.join(paths['NER_DATA_DIR'], "en_conll03.dev.json"),
                    os.path.join(paths['NER_DATA_DIR'], "%s.dev.json" % short_name))
    shutil.copyfile(os.path.join(paths['NER_DATA_DIR'], "en_conll03.test.json"),
                    os.path.join(paths['NER_DATA_DIR'], "%s.test.json" % short_name))

def process_en_ontonotes_ww_multi(paths, short_name):
    """
    Combine the worldwide data with the OntoNotes data in a multi channel format
    """
    print("=============== Preparing OntoNotes ===============")
    process_en_ontonotes(paths, "en_ontonotes")
    print("========== Preparing 9 Class Worldwide ================")
    process_en_worldwide_9class(paths, "en_worldwide-9class")
    # TODO: pass in options?
    ontonotes_multitag.build_multitag_dataset(paths['NER_DATA_DIR'], short_name, True, True)

def process_en_combined(paths, short_name):
    """
    Combine WW, OntoNotes, and CoNLL into a 3 channel dataset
    """
    print("================= Preparing OntoNotes =================")
    process_en_ontonotes(paths, "en_ontonotes")
    print("========== Preparing 9 Class Worldwide ================")
    process_en_worldwide_9class(paths, "en_worldwide-9class")
    print("=============== Preparing CoNLL 03 ====================")
    process_en_conll03(paths, "en_conll03")
    build_en_combined.build_combined_dataset(paths['NER_DATA_DIR'], short_name)


def process_en_conllpp(paths, short_name):
    """
    This is ONLY a test set

    the test set has entities start with I- instead of B- unless they
    are in the middle of a sentence, but that should be find, as
    process_tags in the NER model converts those to B- in a BIOES
    conversion
    """
    base_input_path = os.path.join(paths["NERBASE"], "acl2023_conllpp", "dataset", "conllpp.txt")
    base_output_path = paths["NER_DATA_DIR"]
    sentences = read_tsv(base_input_path, 0, 3, separator=None)
    sentences = [sent for sent in sentences if len(sent) > 1 or sent[0][0] != '-DOCSTART-']
    write_dataset([sentences], base_output_path, short_name, shard_names=["test"], shards=["test"])

def process_armtdp(paths, short_name):
    assert short_name == 'hy_armtdp'
    base_input_path = os.path.join(paths["NERBASE"], "armenian", "ArmTDP-NER")
    base_output_path = paths["NER_DATA_DIR"]
    convert_hy_armtdp.convert_dataset(base_input_path, base_output_path, short_name)
    for shard in SHARDS:
        input_filename = os.path.join(base_output_path, f'{short_name}.{shard}.tsv')
        if not os.path.exists(input_filename):
            raise FileNotFoundError('Cannot find %s component of %s in %s' % (shard, short_name, input_filename))
        output_filename = os.path.join(base_output_path, '%s.%s.json' % (short_name, shard))
        prepare_ner_file.process_dataset(input_filename, output_filename)

def process_toy_dataset(paths, short_name):
    convert_bio_to_json(os.path.join(paths["NERBASE"], "English-SAMPLE"), paths["NER_DATA_DIR"], short_name)

def process_ar_aqmar(paths, short_name):
    base_input_path = os.path.join(paths["NERBASE"], "arabic", "AQMAR", "AQMAR_Arabic_NER_corpus-1.0.zip")
    base_output_path = paths["NER_DATA_DIR"]
    convert_ar_aqmar.convert_shuffle(base_input_path, base_output_path, short_name)

def process_he_iahlt(paths, short_name):
    assert short_name == 'he_iahlt'
    # for now, need to use UDBASE_GIT until IAHLTknesset is added to UD
    udbase = paths["UDBASE_GIT"]
    base_output_path = paths["NER_DATA_DIR"]
    convert_he_iahlt.convert_iahlt(udbase, base_output_path, "he_iahlt")

def process_ang_ewt(paths, short_name):
    assert short_name == 'ang_ewt'
    base_input_path = os.path.join(paths["NERBASE"], "ang", "Old_English-OEDT")
    convert_bio_to_json(base_input_path, paths["NER_DATA_DIR"], short_name)

DATASET_MAPPING = {
    "ang_ewt":           process_ang_ewt,
    "ar_aqmar":          process_ar_aqmar,
    "bn_daffodil":       process_bn_daffodil,
    "da_ddt":            process_da_ddt,
    "de_germeval2014":   process_de_germeval2014,
    "en_conll03":        process_en_conll03,
    "en_conll03ww":      process_en_conll03_worldwide,
    "en_conllpp":        process_en_conllpp,
    "en_ontonotes":      process_en_ontonotes,
    "en_ontonotes-ww-multi": process_en_ontonotes_ww_multi,
    "en_combined":       process_en_combined,
    "en_worldwide-4class": process_en_worldwide_4class,
    "en_worldwide-9class": process_en_worldwide_9class,
    "fa_arman":          process_fa_arman,
    "fi_turku":          process_turku,
    "fr_wikinergold":    process_french_wikiner_gold,
    "fr_wikinermixed":   process_french_wikiner_mixed,
    "hi_hiner":          process_hiner,
    "hi_hinercollapsed": process_hinercollapsed,
    "hi_ijc":            process_ijc,
    "he_iahlt":          process_he_iahlt,
    "hu_nytk":           process_nytk,
    "hu_combined":       process_hu_combined,
    "hy_armtdp":         process_armtdp,
    "it_fbk":            process_it_fbk,
    "ja_gsd":            process_ja_gsd,
    "kk_kazNERD":        process_kk_kazNERD,
    "mr_l3cube":         process_mr_l3cube,
    "my_ucsy":           process_my_ucsy,
    "pl_nkjp":           process_pl_nkjp,
    "sd_siner":          process_sd_siner,
    "sv_suc3licensed":   process_sv_suc3licensed,
    "sv_suc3shuffle":    process_sv_suc3shuffle,
    "tr_starlang":       process_starlang,
    "th_lst20":          process_lst20,
    "th_nner22":         process_nner22,
    "zh-hans_ontonotes": process_zh_ontonotes,
}

SUFFIX_MAPPING = {
    "_ilner":            process_il_ner,
    "_suralk":           process_suralk_multiner,
}

def main(dataset_name):
    paths = default_paths.get_default_paths()
    print("Processing %s" % dataset_name)

    random.seed(1234)

    if dataset_name in DATASET_MAPPING:
        DATASET_MAPPING[dataset_name](paths, dataset_name)
    elif dataset_name in ('uk_languk', 'Ukranian_languk', 'Ukranian-languk'):
        process_languk(paths, 'uk_languk')
    elif dataset_name.endswith("FIRE2013") or dataset_name.endswith("fire2013"):
        process_fire_2013(paths, dataset_name)
    elif dataset_name.endswith('WikiNER'):
        process_wikiner(paths, dataset_name)
    elif dataset_name.startswith('hu_rgai'):
        process_rgai(paths, dataset_name)
    elif dataset_name.endswith("_bsnlp19"):
        process_bsnlp(paths, dataset_name)
    elif dataset_name.endswith("_nchlt"):
        process_nchlt(paths, dataset_name)
    elif dataset_name in ("nb_norne", "nn_norne"):
        process_norne(paths, dataset_name)
    elif dataset_name == 'en_sample':
        process_toy_dataset(paths, dataset_name)
    elif dataset_name.lower().endswith("_masakhane"):
        process_masakhane(paths, dataset_name)
    else:
        for ending in SUFFIX_MAPPING:
            if dataset_name.endswith(ending):
                SUFFIX_MAPPING[ending](paths, dataset_name)
                break
        else:
            raise UnknownDatasetError(dataset_name, f"dataset {dataset_name} currently not handled by prepare_ner_dataset")
    print("Done processing %s" % dataset_name)

if __name__ == '__main__':
    main(sys.argv[1])
