# UralicNLP
UralicNLP is a natural language processing library for small Uralic languages. Currently its functionality is provided by sanat.csc.fi API which is also developed by [Mika Hämäläinen](https://mikakalevi.com).

UralicNLP can produce **morphological analysis**, **generate morphological forms**, **lemmatize words** and **give lexical information** about words in Uralic languages. At the time of writing, the following languages are supported: Skolt Sami, Ingrian, Meadow & Eastern Mari, Votic, Olonets-Karelian, Erzya, Moksha, Hill Mari, Udmurt, Tundra Nenets, Komi-Permyak and Finnish. This information originates from FST tools and dictionaries developed in the [Giellatekno infrastructure](http://giellatekno.uit.no/).

## Installation
The library can be installed from [PyPi](https://pypi.python.org/pypi/uralicNLP/).

    pip install uralicNLP
   
In case you want to use the Constraint Grammar features (*from uralicNLP.cg3 import Cg3*), you will also need to [install VISL CG-3](http://visl.sdu.dk/cg3/chunked/installation.html).

If you are using Linux and you run into problems with installing hfst dependency, you can find some help on [a blog post on installing hfst](https://mikalikes.men/using-hfst-on-python/)

## Usage

### List supported languages
The API is under constant development and new languages will be added to the Sanat infrastructure. That's why UralicNLP provides a functionality for looking up the list of currently supported languages. The method returns 3 letter ISO codes for the languages.

    >>from uralicNLP import uralicApi
    >>uralicApi.supported_languages()
    {'languages': ['sms', 'izh', 'mhr', 'vot', 'olo', 'myv', 'mdf', 'mrj', 'udm', 'yrk', 'koi', 'fin']}
  
### Lemmatize words
A word form can be lemmatized with UralicNLP. This does not do any disambiguation but rather returns a list of all the possible lemmas.

    >>from uralicNLP import uralicApi
    >>uralicApi.lemmatize("вирев", "myv")
    ['вирев', 'вирь']
  
  An example of lemmatizing the word *вирев* in Erzya (myv).

### Morphological analysis
Apart from just getting the lemmas, it's also possible to perform a complete morphological analysis.

    >>from uralicNLP import uralicApi
    >>uralicApi.analyze("voita", "fin")
    [['voi+N+Sg+Par', 0.0], ['voi+N+Pl+Par', 0.0], ['voitaa+V+Act+Imprt+Prs+ConNeg+Sg2', 0.0], ['voitaa+V+Act+Imprt+Sg2', 0.0], ['voitaa+V+Act+Ind+Prs+ConNeg', 0.0], ['voittaa+V+Act+Imprt+Prs+ConNeg+Sg2', 0.0], ['voittaa+V+Act+Imprt+Sg2', 0.0], ['voittaa+V+Act+Ind+Prs+ConNeg', 0.0], ['vuo+N+Pl+Par', 0.0]]
  
An example of analyzing the word *voita* in Finnish (fin).

### Morphological generation
From a lemma and a morphological analysis, it's possible to generate the desired word form. 


    >>from uralicNLP import uralicApi
    >>uralicApi.generate("käsi+N+Sg+Par", "fin")
    [['kättä', 0.0]]
  
An example of generating the singular partitive form for the Finnish noun *käsi*. The result is *kättä*.

### Download the models to speed things up

If you have a lot of data to process, it might be a good idea to download the morphological models to your computer locally. This can be done easily.

    >>from uralicNLP import uralicApi
    >>uralicApi.download("fin")

When models are installed, *generate()*, *analyze()* and *lemmatize()* methods will automatically use them instead of the server side API. [More information about the models](https://github.com/mikahama/uralicNLP/wiki/Models).

### Syntax - Constraint Grammar disambiguation

**Note** this requires the models to be installed (see above) and [VISL CG-3](http://visl.sdu.dk/cg3/chunked/installation.html). The disambiguation process is easy.

    >>from uralicNLP.cg3 import Cg3
    >>sentence = "Kissa voi nauraa"
    >>tokens = sentence.split(" ") #Do a simple tokenization for the sentence
    >>cg = Cg3("fin")
    >>print cg.disambiguate(tokens)
    [(u'Kissa', [<Kissa - N, Prop, Sg, Nom, <W:0.000000>>, <kissa - N, Sg, Nom, <W:0.000000>>]), (u'voi', [<voida - V, Act, Ind, Prs, Sg3, <W:0.000000>>]), (u'nauraa', [<nauraa - V, Act, InfA, Sg, Lat, <W:0.000000>>])]
    
The return object is a list of tuples. The first item in each tuple is the word form used in the sentence, the second item is a list of *Cg3Word* objects. In the case of a full disambiguation, these lists have only one Cg3Word object, but some times the result of the disambiguation still has some ambiguity. Each Cg3Word object has three variables *lemma*, *form* and *morphology*.

    >>disambiguations = cg.disambiguate(tokens)
    >> for disambiguation in disambiguations:
    ...     possible_words = disambiguation[1]
    ...     for possible_word in possible_words:
    ...         print possible_word.lemma, possible_word.morphology
    Kissa [u'N', u'Prop', u'Sg', u'Nom', u'<W:0.000000>']
    kissa [u'N', u'Sg', u'Nom', u'<W:0.000000>']
    voida [u'V', u'Act', u'Ind', u'Prs', u'Sg3', u'<W:0.000000>']
    nauraa [u'V', u'Act', u'InfA', u'Sg', u'Lat', u'<W:0.000000>']

### Lexical information
UralicNLP makes it possible to obtain the information available in sanat.csc.fi entries in JSON format. The information can contain data such as translations, example sentences, semantic tags, morphological information and so on. You have to define the language code of the dictionary. 

For example, "sms" selects the Skolt Sami dictionary. However, the word used to query can appear in any language. If the word is a lemma in Skolt Sami, the result will appear in "exact_match", if it's a word form for a Skolt Sami word, the results will appear in "lemmatized", and if it's a word in some other language, the results will appear in "other\_languages" under the language code of that language. I.e if you search for *cat* in the Skolt Sami dictionary, you will get a result of a form {"other\_languages": "eng": [Skolt Sami lexical items that translate to cat]}



    >>from uralicNLP import uralicApi
    >>uralicApi.dictionary_search("car", "sms")
    {'lemmatized': [], 'query': 'car', 'exact_match': {}, 'other_languages': {'eng': [{'lemma': 'autt', ...}]}, 'language': 'sms'}
  
An example of querying the Skolt Sami dictionary with *car*.

### Parsing UD CoNLL-U annotated TreeBank data

UralicNLP comes with tools for parsing and searching CoNLL-U formatted data. Please refer to [the Wiki for the UD parser documentation](https://github.com/mikahama/uralicNLP/wiki/UD-parser-(experimental)).

# Cite

If you use UralicNLP in an academic publication, please cite it as follows:

Mika Hämäläinen. (2018, January 9). UralicNLP (Version v1.0). Zenodo. http://doi.org/10.5281/zenodo.1143638

[![DOI](https://zenodo.org/badge/DOI/10.5281/zenodo.1143638.svg)](https://doi.org/10.5281/zenodo.1143638)

