= HXLM: Multilingual Terminology in Humanitarian Language Exchange
:toc: preamble
:icons: font
:source-highlighter: highlight.js
:highlightjs-languages: yaml

https://hdp.etica.ai/hxltm[image:https://img.shields.io/badge/Site-hdp.etica.ai%2Fhxltm-blue[Site]]
https://github.com/EticaAI/HXL-Data-Science-file-formats[image:https://img.shields.io/badge/GitHub-EticaAI%2FHXL--Data--Science--file--formats-lightgrey?logo=github&style=social[EticaAI/HXL-Data-Science-file-formats]]
https://pypi.org/project/hdp-toolchain/[image:https://img.shields.io/badge/python%20package-hdp--toolchain-brightgreen[Python
Package: hdp-toolchain]]
https://hxlstandard.org/[image:https://img.shields.io/badge/Standard-HXL-%23F26459[Standard
HXL]]
image:https://img.shields.io/github/license/EticaAI/HXL-Data-Science-file-formats[License]
https://docs.google.com/spreadsheets/d/1ih3ouvx_n8W5ntNcYBqoyZ2NRMdaA0LRg5F9mGriZm4/edit#gid=470146486[image:https://img.shields.io/badge/Google%20Drive-HXL--CPLP--Vocab_Auxilium--Humanitarium--API-yellowgreen[Google
Drive]]

The _Multilingual Terminology in Humanitarian Language Exchange_
(abbreviation: "`HXLTM`") is an HXLated valid tabular format (stricly, a
documented subset of https://hxlstandard.org/[HXL]) by
https://github.com/HXL-CPLP[HXL-CPLP] with strong focus to store community
contributed translations and glossaries while maximizing portability for
implementers (like via use of `+hxltmcli+` to export to
Excel, +++<abbr title="XML Localization Interchange File Format">XLIFF</abbr>+++,
+++<abbr title="Translation Memory eXchange">TMX</abbr>+++,
+++<abbr title="TermBase eXchange">TBX</abbr>+++,
+++<abbr title="Universal Terminology eXchange">UTX</abbr>+++, JSON, CSV,
https://hdp.etica.ai/hxltm/archivum/[and more!].

____
> *Why HXLTM was created?*
>
> HXLTM is...
>
> *I'm in the middle of something urgent! What to do?!*
>
> Okay, If it is your first time here, and you're like a software developer
> volunteer in the middle of a disaster response (or like someone in a
> Hackathon) you either will need to:
>
> . *As localization implementer*:
>   .. use existing HXLTM public datasets by exporting to a format your
>      software undestand. The ones done by HXL-CPLP and Etica.AI are on
>      public domain dedication.
>   .. To **convert from HXLTM to several other data formats, like TMX, TBX,
>      UTX, XLIFF, CSV, Excel, etc, etc, etc** check the dedicated page
>      https://hdp.etica.ai/hxltm/archivum/
> . *As translator collaborator*: be yourself a **translator manager** (even
>   if a single person team) and either
>   .. *Simple case*: "fork" someone else work (e.g. copy Hapi spreadsheets
>      <https://hapi.etica.ai/>) and do everything from your version;
>      you're welcome to donate back your work!
>   .. **Complex, lot's of texts**: export a bilingual XLIFF file from HXLTM to
>      a target language, then use an collaborative tool like
>     <https://www.matecat.com/> (check this https://www.youtube.com/watch?v=FCgOjBry6Ec[MateCat tutorial]).
>       ... Protip A: 2 (two) native languaguage speakers (one as translator,
>           plus other to review) is the perfect case to be accepted by others
>           immediately.
>       ... Protip B: The MateCat site, in special for translators, do not need
>           an account; you can just share the private link to one or more
>           collaborators.
> . *"If it's a good idea, go ahead and do it. It is much easier to apologize
>    than it is to get permission."* — Grace Hopper
>
> *(Again) Why HXLTM was created?*
> ++++
> See <a href="#why-HXLTM">Preface: Why HXLTM was created?</a>
> ++++

____

////

>
> Keep in mind that you do not need to know what HXLTM is if is here just to
> export as soon as possible content generated by others (like software
> developer volunteers in the middle of a disaster response).
>

> While `hxltmcli`, even early versions, already can be used as
> non-Humanitarian general propose to exchange translations stored
> (or entirely managed) "with Excel" or "Google Sheets", the (...TODO: continue)

also tooling for bilingual
> formats like XLIFF is great, like
> https://www.youtube.com/watch?v=FCgOjBry6Ec[online free CAT tool, MateCat]


> TODO: see also <https://github.com/idimitriadis0/TranslateOnLinux/blob/master/TranslateOnLinux.md>

- Standard: **Translation Memory eXchange (TMX) v1.4b**
  - https://www.gala-global.org/lisa-oscar-standards
  - https://en.wikipedia.org/wiki/Translation_Memory_eXchange
  - Example of usages
    - https://cloud.google.com/translate/automl/docs/prepare
    - https://mymemory.translated.net/doc/from-empty-tm.php
    - https://site.matecat.com/faq/translation-memory/
- Issues:
  - **HXL-CPLP/forum/issues/**
    - [**_HXL-CPLP/forum/issues/58: Convenção de tags HXL em conjunto de dados para armazenar Memória de Tradução (eng: HXL translation memory TM) \#58_**](https://github.com/HXL-CPLP/forum/issues/58)
  - **HXL-CPLP/Auxilium-Humanitarium-API**
    - **[HXL-CPLP/Auxilium-Humanitarium-API: [Hapi versão Alpha] Fluxo de trabalho de de traduções até geração do Hapi (do website, dos schemas e das OpenAPI)](https://github.com/HXL-CPLP/Auxilium-Humanitarium-API/issues/13)**
    - **[HXL-CPLP/Auxilium-Humanitarium-API: [MVP] Exportar de formato "HXL TM" (eng: HXL translation memory) para um ou mais formatos já usados por softwares de localização](https://github.com/HXL-CPLP/Auxilium-Humanitarium-API/issues/16)**
  - **EticaAI/HXL-Data-Science-file-formats**
    - _**hxltm2xliff: HXL Trānslātiōnem Memoriam -> XLIFF Version 2.1 #19**_
- Test projects
  - https://github.com/UNMigration/HTCDS
  - https://docs.google.com/spreadsheets/d/1ih3ouvx_n8W5ntNcYBqoyZ2NRMdaA0LRg5F9mGriZm4/edit#gid=1292720422

////

== Quickstart

=== Installation

`+hxltmcli+` uses Python 3. While is possible to just copy the
`+hxltmcli+` file and install manually dependencies, like the
https://github.com/HXLStandard/libhxl-python[HXLStandard/libhxl-python],
you can install with the
https://pypi.org/project/hdp-toolchain/[hdp-toolchain].

[source,bash]
----
# hxltmcli is installed with the hdp-toolchain, no extras required.
# @see https://pypi.org/project/hdp-toolchain/
pip install hdp-toolchain[hxltm]

hxltmcli --help

----

=== HXLTM: supported file types

NOTE: Check the dedicated page: https://hdp.etica.ai/hxltm/archivum/

== Preface

[#why-HXLTM]
=== Why HXLTM was created?

HXLTM is, by design, focused on storage and exchange of multilingual content
while keeping minimum compatibility with average localization file formats
used by software (TL;DR: they are mono or bilingual).

Unless we talk with a format like TBX (TermBase eXchange), and ignoring the
unsung hero UTX (Universal Terminology eXchange), the current standards to
exchange multilingual content are almost 20 years stuck on the past without
any update (check TMX and the sad fact that LISA was declared insolvent).
It’s a long history. And the problem is not if you don’t like XML-like formats
instead of JSON, because even the XML-like standards, as of 2021, still have
a poor feature set. The thing are so bad, SO BAD, that people use
gettext .po or bilingual JSON to handle multilingual translations/glossaries.

It’s a long history, but not just because HXL-CPLP, as a community user group,
would want to promote HXL (and, by the way, HXLTM is the format used on the
HXL-CPLP/Auxilium-Humanitarium-API if you want to collaborate 😀), but even
if you someone else want to coordinate volunteers to handle translations
without always have to enforce a single source translation language
(like English, as is mostly used for software localization), there is
simply no standard (and we mean eve paid services) to let you do it.

> Do not attribute to malice that which is adequately explained by lack of
  how other languages work and lack of tooling done by previous developers.

On this topic, one easily observable fact is that actually most maintainers
of software primarily written in English are not even native English speakers.
But the lack of knowledge of how other natural languages work can play a role
in initial decisions: it is also common, when asked for feedback, native
speakers of other languages (that care about the issue) don’t know what do
propose and the ones who speak up (often new to the software development)
may tell that this is not necessary.

TIP: @TODO this section still a draft. Needs to be improved later.

=== The need for optional different source language for translations

While `+hxltmcli+`, even early versions, already can be used as
non-Humanitarian general propose to exchange translations stored
(or entirely managed) "with Excel" or "Google Sheets" one of what would
become initial goals of HXL-CPLP is a data format and minimal software
implementation that could allow receiving collaborative suggestions
from people who don’t know whatever would be the initial working language.

If you look at the context of the HXL-CPLP/Auxilium-Humanitarium-API by
initial working language we don’t mean English. Good source material could in
fact be created by experts (let’s say, medical field, on a country like
Brazil, on a subject like the Covid pandemic) and in special outside of
software development, these experts, when do know more than their native
language (as is Portuguese on Brazil) they could do know Spanish, or French,
or Italian, but not English.


////

== HXLTM working file formats

The `+hxltmcli+` is an _(initial reference)_ of an public domain python
cli tool allow reuse by others interested in export HXLTM files to
common formats used by professional translators. But software developers
interested in promote use cases of HXL are encouraged to either
collaborate to `+hxltmcli+` or create other tools.

TIP: @TODO this section still a draft. Needs to be improved later.


== FAQ

=== Save entire Translations Memory on Excel files

==== Example data

* `+HXLTM-Exemplum+`: Generic test files:
** Input files: link:/tests/hxltm/[tests/hxltm/]
*** Live spreadsheet:
https://docs.google.com/spreadsheets/d/1isOgjeRJw__nky-YY-IR_EAZqLI6xQ96DKbD4tf0ZO8/edit#gid=0
** Output files: tests/hxltm/resultatum/
* Production files:
** `+HXL-CPLP-Vocab_Auxilium-Humanitarium-API+`: Hapi project
*** GitHub:
**** https://github.com/HXL-CPLP/Auxilium-Humanitarium-API
*** Live Spreadsheet:
**** https://docs.google.com/spreadsheets/d/1ih3ouvx_n8W5ntNcYBqoyZ2NRMdaA0LRg5F9mGriZm4/edit#gid=470146486
**** Note: the project may eventually use other sources of data (and
this link here may eventually not be up to date)

=== Advanced filter with HXL cli tools

++++
<a id="HXLTM-libhxl-cli-tools" href="#HXLTM-libhxl-cli-tools">§ HXLTM-libhxl-cli-tools</a>
++++

* See **https://github.com/HXLStandard/libhxl-python/wiki/HXL-cookbook**

Since a HXLTM (before export) is a valid HXL file, advanced seleting is
possible by, instead of `hxltmcli input.hxl.csv output.hxl.csv` use
`hxlcut input.hxl.csv --exclude (...) | hxltmclioutput.hxl.csv`.

[source,bash]
----
# libhxl already is installed with hdp-toolchain

hxlselect --help
#    Filter rows in a HXL dataset. (...)
hxlcut --help
#    Cut columns from a HXL dataset.

## Examples with HXL TM (used before pass data to hxltmcli)
hxlcut --exclude item+i_la+i_lat+is_Latn --sheet 6 HXL-CPLP-Vocab_Auxilium-Humanitarium-API.xlsx | hxltmcli
# Excludes Latin before pass to hxltmcli, from Microsoft Excel

hxlcut --exclude item+i_la+i_lat+is_Latn https://docs.google.com/spreadsheets/d/1ih3ouvx_n8W5ntNcYBqoyZ2NRMdaA0LRg5F9mGriZm4/edit#gid=1292720422 | hxltmcli
# Excludes Latin before pass to hxltmcli, from Google Sheets
----

=== Advanced filter with HXL-proxy (integration with Google Sheets and CSV/XLSX/etc avalible on web)
++++
<a id="HXLTM-HXL-Proxy" href="#HXLTM-HXL-Proxy">§ HXLTM-HXL-Proxy</a>
++++

In special if you are contributing for either tools for HXL, testing this tool
or helping in production (e.g. real time disaster response) please consider
usage of the public HXL-Proxy on https://proxy.hxlstandard.org/.

Most advanced features of the libhxl cli tools are availible via HXL-proxy.

==== Note about heavy usage: use cache
Both https://hapi.etica.ai/ and https://github.com/HXL-CPLP/Auxilium-Humanitarium-API
(and some links used on this documentation) may use the HXL-Proxy default
1 hour cache disabled. This is necessary because the HXL-proxy is used to build
static content based on latest translations.

It's a good practice if you are not only testing, but deployng in production,
to not disable HXL-Proxy cache (it's the default option if not copy and pasting
HXL-CPLP/Auxilium-Humanitarium-API internal build script links).

Also, even if you do not use the HXL-Proxy (but is using `hxltm` directly to
your own Google Spreadsheets) if you keep doing too much calls in short time
eventually the Google Docs may raise 400 errors since `hxltm` are not
authenticated requests. **Our recomendations on this case is:**

1. **download the entire Spreadsheet as .xlsx file and process the .xlsx file locally.**
2. **Download individual sheets as CSV files and save locally (this consumes less CPU than process .xlsx)**

////

[#presentations]
== Presentations

[#bootstrapping-hxltm]
=== Bootstrapping-HXLTM
==== eng-Latn: Bootstrapping technical translations and multilingual controlled vocabularies with HXLTM
https://docs.google.com/presentation/d/11t8EMTtT8mV9NM0Dy7kCHDZEYkmSeyh7OA5CX50_OEQ/edit?usp=sharing[eng-Latn: Bootstrapping technical translations and multilingual controlled vocabularies with HXLTM]

==== por-Latn: Como criar do zero traduções técnicas e vocabulários controlados multilíngues com HXLTM
https://docs.google.com/presentation/d/1wEw9MmEAQypn20XLSDETbV3q2Nlf9ICwn3q970Du1es/edit?usp=sharing[por-Latn: Como criar do zero traduções técnicas e vocabulários controlados multilíngues com HXLTM]

[#ontologia]
== Ontologia

NOTE: Most features feasible to not hardcode (id est, require you to hack the
      python code) on initial reference implementations use an ontology file
      as single source of truth.

CAUTION: While the exporting features (like `+normam.TMX.formatum.initiale+`,
         `+normam.TMX.formatum.corporeum+` and `+normam.TMX.formatum.finale+`)
         allow great customization only by fine tunning your ontology file,
         (TIP: the syntax is based on Liquid, 
         https://shopify.github.io/liquid/) importing data from other formats 
         like convert TMX, TBX or XLIFF) may require change not only the
         ontology file but also the Python code.

Ontology, YAML::
- https://hdp.etica.ai/ontologia/cor.hxltm.yml

Ontology, JSON::
- https://hdp.etica.ai/ontologia/json/cor.hxltm.json

[source,yaml]
----
include::../ontologia/cor.hxltm.yml[]
----

== License

link:UNLICENSE[image:../img/public-domain.png[Public Domain Dedication]]

The https://github.com/EticaAI[EticaAI] has dedicated the work to the
link:../UNLICENSE[public domain] by waiving all of their rights to the
work worldwide under copyright law, including all related and
neighboring rights, to the extent allowed by law. You can copy, modify,
distribute and perform the work, even for commercial purposes, all
without asking permission.
