Altangerel, C., Purev, J., Yesyenbyek, K., & Hansakunbuntheung, C. (2013). An evaluation of Mongolian data-driven Text-to-Speech. In 2013 International Conference Oriental COCOSDA held jointly with 2013 Conference on Asian Spoken Language Research and Evaluation (O-COCOSDA/CASLRE) (pp. 1–4). IEEE. 2013 International Conference Oriental COCOSDA held jointly with 2013 Conference on Asian Spoken Language Research and Evaluation (O-COCOSDA/CASLRE). https://doi.org/10.1109/icsda.2013.6709881 # AN EVALUATION OF MONGOLIAN DATA-DRIVEN TEXT-TO-SPEECH Chagnaa Altangerel, Jaimai Purev, Kerey Yesyenbyek Center for Research on Language Processing National University of Mongolia, NUM Ulaanbaatar, Mongolia {altangerel, purev,esenbek}@num.edu.mn Abstract— This paper presents a first attempt to evaluate data-driven speech synthesis of Mongolian trained on 1500- sentence female speech corpus. The speech corpus contains nearly 6 hours of Mongolian female speech that is designed to cover all Mongolian phones. The evaluation is done on two levels. In overall quality evaluation, we generated 25 sentences and asked raters about their quality based on Mean Opinion Score (MOS). The second evaluation uses Phoneme confusion test, which contains all possible phoneme set in Mongolian. Keywords— Speech corpus, Mongolian, Text-to-Speech Synthesis, Evaluation Chatchawarn Hansakunbuntheung Speech and Audio Technology Lab. National Electronics and Computer Technology Center Pathumthani, Thailand chatchawarn.hansakunbuntheung@nectec.or.th In Mongolian, a high quality Text-to-Speech synthesizer (TTS) is required for many applications such as speech to speech translation systems, screen readers etc. A few TTS development were implemented to support the purposed in the past several years [1-5,15]. Due to the lack of large highquality speech corpus designed for TTS, only few TTS systems based on unit concatenation approach i.e. diphone-based speech synthesizers have been successfully developed [1, 2]. However, the quality of the TTS systems required more intelligibility and naturalness to be practical. # I. INTRODUCTION This paper evaluates the HMM-based speech synthesis system using the 6-hour female speech corpus built within the framework of International Telecommunication Union (ITU) and Advanced Micro Devices (AMD) cooperation [15]. The goal of this paper is to present an evaluation of the first Mongolian female voice and extension of speech corpus for Mongolian TTS. In this paper, we first, introduce Mongolian language briefly in Section 2. Section 3 will introduce a review on Mongolian resources. Then, the design of evaluation and its results is shown in Section 4. And, finally Section 5 will finalize the work and conclude. # II. MONGOLIAN LANGUAGE Mongolian is one of the Altaic family languages. It is spoken by nearly 8 million speakers living mainly in Mongolia, parts of China and Russia, and neighboring countries. Due to its historical and geographical background, Mongolian has several dialectal variations in its linguistic, phonetic and graphic expressions such as Khalkha, Oirat, Buryat, Khalmyc and Inner Mongolia. Hence, we focus on Khalkha (Halh) dialect that is the national language of Mongolia. Over 2.7 million people speak Mongolian throughout Mongolia. [6, 7] In morphology, Mongolian is a highly agglutinative language. So, it has a rich number of morphemes to build up more complex words from simple roots [6]. Historically Mongolia has many written systems dating back to 1000 AD. The most recent written systems in Mongolian language including traditional Mongolian (Uygur Mongolian scripts), Latin, and, Cyrillic scripts. Official Mongolian script is Mongolian Cyrillic script which is generally used in daily life and computer systems [8, 9]. Genuine Mongolian words use the following syllable structures: V, CV, CVC, VVC, CVCC, VCCC and CVCCC; where C is consonant and $\mathrm { V }$ is vowel. Mongolian has seven short vowels, seven long vowels, five diphthongs and five ‘y’- vowels. Because of variations in dialect and conflicts in theories of Mongolian phonology, there are approximately 19- 23 consonants [15]. # III. REVIEW ON MONGOLIAN LANGUAGE RESOURCE # A. Mongolian text resource A large Mongolian text corpus of 5 million words is collected at National University of Mongolia [10, 11]. The content of the corpus is mostly from domains like daily online or printed newspapers, literature, and laws. The text corpus is already cleaned and tagged semi-automatically by spell checker and parsing tools. # B. Mongolian speech resource Existing Mongolian speech corpora have been collected for various purposes. A Multi-dialectal speech corpus of Mongolia (MDSCM) attempted to collect and to preserve variation of Mongolian text and speech dialects. However, the specification of MDSCM is not designed to consider phone variation and coverage for speech synthesis implementation. Another existing speech corpus is designed for automatic speech recognition (ASR). Because of its goal for ASR, it collected speech data of multi-speakers. Lists of 10 words were recorded repeatedly 10 times by 10 speakers. In total, 80 speakers’ speech data were collected and consists of 60,000 wave files [13]. The female speech corpus of 1500 sentences built for Mongolian text-to-speech system done in [15] is used in this evaluation. The contents of the speech corpus are selected from the 5-million text data [10, 11] using phone balanced criterion to cover all Mongolian consonant and vowel phones. # C. Tools The same speech corpus, tools and phonetic systems described in [15] are used in this work. Please refer to it for more detail. # IV. THE EVALUATION The evaluation of the speech synthesis is done at two levels. In overall quality evaluation, we generated 25 sentences and asked raters about their quality based on Mean Opinion Score (MOS). The generated sentences are selected from Mongolian online news covering various topics such as business, weather, sports, etc. In the second test, we have generated syllables for each phoneme (33 consonants and 20 vowels) and asked if listener can identify the phoneme. Listeners consisted of 80 people about 23-30 years old, with equal gender ratio. # A. Overall quality Overall quality evaluation used MOS and asked raters’ opinion about the sound quality/naturalness/smoothness of the speech. The score ranges from 1 (very bad) to 5 (very good). As shown in Fig. 1, the most raters’ score tend to be on the positive side with the highest rated score of 4 (Good), while, the percentage of the negative scores is very low. Average score of the whole sentences is 3.7 ( $74 \%$ according to 5). This score is relatively higher than previously done diphone based speech synthesis [2] system with the average score of fair level. ![](images/a579f9c6235f43cd143ad7a7e47d371b3daa89eb9da4acdda63a09c81db019df.jpg) Fig. 1. Overall quality score. # B. Phoneme confusion test We tried to include all possible Mongolian phonemes in this test. For consonant set each consonant phoneme is combined with long “aa” /a:/ vowel. Total 33 consonants out of 34 are combined in this test set, since consonant “ng” $/ \mathrm { N } /$ is not possible to come before a vowel. Table 1 shows the syllable set used in consonant phoneme test set. For consonants, Table 2 presents the confusion matrix of consonants. It clearly shows an important issue to improve the system i.e. the raters did not identify any of palatalized consonants such as B, D, G, K, L, M, N, etc. So, these columns are removed from confusion matrix, as we also didn’t have enough space in this paper. This result shows that all palatalized consonants are the most problematic sounds that extremely need further improvement to increase intelligibility of the system. We found that the whole palatalized ones were perceived as non-palatalized ones. So, the acoustic modeling seems to be failed to capture sound characteristics of palatalized sounds. Consonants $g , k , l , m , r , t , \nu , f , j , w , z$ are misclassified. Among them, $z$ is more close to $c$ and $h$ is more close to $w$ in nature. Some confusions among phones with similar places or manners can be found in the results. For examples, 1) $p$ and $k$ sounds, voiceless plosive sounds, were mostly perceived as $t$ sound, 2) confusion between $^ { b }$ and $g$ voiced plosive sounds. For vowels, there are 20 vowels in Mongolian and all of them are used in this test, as shown in Table 3. Raters could not classify correctly all of short vowels. Short vowels $e$ and $o$ are not identified. But $y$ is identified as $i$ which is a feminine counterpart of it. Raters also did not identify correctly any of the diphthongs and long vowels. Long vowels are identified as short vowels. Some of them are classified as its short counterparts such as a:, i:,^: etc. We may count them as correct answer, since we did not explain the raters enough about the differences of the phonemes duration. Long vowel $@$ : is classified as $u$ , which is a short feminine counterpart of it. Among long vowels, $e .$ : is weakly recognized. Some of the diphthongs are classified as $i ,$ , such as $i .$ , uy and ey. Two diphthongs ay and oy are classified as $o$ . TABLE I. CONSONANT PHONEME SET

No | Phone | Syll.
1 | ph | b | B | Баа
2 | ts | с | ц | Цаа
3 | th | d | Д | Даа
4 | g | g | Γ | Гаа
5 | kh | k | K | Kaa
6 | 13 | 1 | л | Лаа
7 | m | m | M | Maa
8 | n | n | H | Haa
9 | р | р | п | Паа
10 | r | r | P | Paa
11 | s | s | C | Caa
12 | t | t | т | Taa
13 | W | V | B | Baa
14 | x | x | X | Xaa
15 | f | f | Ф | Фаа
16 | tf | h | ч | Чаа
17 | j | ј | я,Ё,ЮO | Яа

No | Phone | Syll.
18 | N | ng | Hr | NA
19 | t | w | ж | Жаа
20 | tsh | Z | 3 | 3aa
21 | pih | B | Бь | Биа
22 | g | C | Гь | Гиа
23 | tih | D | Дь | Дia
24 | G | G | Га | Гиа
25 | kjh | K | Кь | Киа
26 | j | L | Ль | Лиa
27 | mj | M | Мь | Миа
28 | n | N | Нь | Ниa
29 | pj | P | Пь | Пиa
30 | ri | R | Pь | Риa
31 | L | S | ш | Шиа
32 | ti | т | Ть | Тиа
33 | wj | V | Bь | Виa
34 | xj | X | Хь | Хиа

Raters' classification (%)

| b | с | d | g | k | I | т | n | р | г | s | |t | v | х | f | h | |j | w | Z
b | 21 | о | 3 | 21 | 7 | о | 3 | 0 | 3 | 3 | з | 10 | о | 17 | 3 | 3 | о | 0 | 0
с | 2 | 55 | 0 | 2 | 12 | о | 0 | 0 | 0 | 2 | 14 | 2 | о | 8 | о | о | о | 0 | 2
d | 0 | 5 | 40 | 15 | 3 | 3 | 0 | 3 | 0 | о | 5 | 13 | о | 5 | 0 | 8 | о | 0 | 3
g | 23 | 3 | 7 | 17 | 3 | 0 | 0 | 0 | 10 | 3 | 7 | 7 | о | 10 | 0 | 7 | о | 0 | 3
k | 0 | 17 | о | 0 | .3 | 0 | 3 | 0 | 8 | 0 | 3 | 56 | 0 | 6 | 3 | 3 | 0 | 0 | 0
1 | 3 | 19 | 0 | 6 | 6 | о | 3 | 3 | 0 | о | 6 | 10 | о | 0 | 0 | 42 | 0 | о | о
m | 2 | 2 | о | 4 | 4 | 0 | 22 | 49 | 9 | 0 | 2 | 4 | о | о | 0 | о | 0 |  | о
n | о | о | 0 | о | 6 | 0 | 11 | 60 | 9 | о | о | 11 | 3 | о | о | 0 | о | 0 | о
p | 0 | 19 | 3 | о | 8 | 3 | 0 | 0 | 22 | 0 | 6 | 25 | о | 6 | 3 | 3 | 0 | о | 3
r | 8 | 8 | 0 | 12 | 15 | 0 | 8 | 4| | 15 | 4 | 4 | 15 | о | о | о | 8 | 0 | о | о
§ | 0 | 6 | 0 | 2 | 2 | 0 | 0 | 0 | 2 | 0 | 86 | 2 | 0 | о | 0 | 0 | о | о | о
t | 2 | 33 | 0 | о | 4 | о | 0 | 0 | 2 | 4 | 4 | 27 | о | 4 | 4 | 13 | 0 | о | о
V | 7 | 2 | 0 | о | 2 | 7 | 9 | 16 | 11 | 14 | 2 | 7 | 2 | 11 | 5 | 5 | о | 0 | о
X | о | 0 | 2 | о | 3 | 0 | 0 | о | 2 | 2 | о | 3 | о | 88 | о | о | о | о | о
f | 0 | 8 | 0 | 0 | 5 | 3 | 0 | 0 | 10 | о | о | 41 | о | 8 | 5 | 21 | о | о | о
h | 0 | 0 | 0 | 6 | о | 6 | 3 | 3 | 0 | 0 | 3 | 0 | о | 3 | 0 | 78 | о | 0 | о
j | о | 0 | 4 | 7 | о | 17 | 2 | 41 | 0 | 2 | о | 2 | 2 | о | 0 | 15 | 4 | 2 | о
w | 0 | 4 | 0 | 8 | о | 2 | 0 | 2 | 0 | о | 4 | 4 | о | о | 0 | 62 | о | 15 | о
z | о | 39 | о | 4 | о | о | 0 | о | 0 | о | 25 | о | 2 | о | 0 | 0 | о |  | 30
B | о | 48 | 0 | 2 | о | о | 10 | 10 | 2 | 2 | 2 | 7 | о | 7 | 2 | 2 | о | 0 | 5
с | 3 | 24 | 12 | 3 | о | 3 | 12 | 15 | 3 | 0 | 0 | 12 | о | 6 | 0 | 0 | 0 | 0 | 9
D | 4 | 11 | 4 | 7 | о | 4 | 4 | 21 | 18 | 7 | 0 | 7 | 4 | 4 | 4 | 0 | о | о | 4
G | 12 | 9 | 27 | 15 | 3 | 3 | 9 | 15 | 0 | 0 | 0 | 3 | 0 | о | 0 | 0 | о | о | 3
K | 0 | 18 | 5 | 5 | о | о | 3 | 3 | 3 | 3 | 3 | 28 | о | о | 0 | 25 | о | 8 | 0
L | 0 | 0 | 2 | 4 | 2 | о | 4 | 0 | 0 | 0 | о | 4 | о | о | 0 | 67 | о | 15 | 0
M | 5 | 3 | 0 | з | о | 3 | 16 | 11 | 5 | 43 | 0 | 3 | 3 | 3 | о | 3 | о | 0 | 0
N | 6 | 9 | 3 | 3 | 0 | о | 15 | 27 | 6 | 15 | 0 | о | 3 | о | 0 | 0 | 0 | 0 | 12
P | 3 | 36 | 0 | 3 | о | 3 | 0 | 0 | 9 | 3 | 0 | 9 | о | 0 | 3 | 0 | з | о | 0 27
R | 11 | 4 | 4 | 4 | 0 | 0 | 7 | 11 | 11 | 7 | 0 | 21 | о | о | 7 | 0 | о | о | 014
S | 0 | 25 | 0 | о | о | 25 | 0 | 25 | 0 | 0 | 0 | о | о | о | 0 | 0 | 0 |  | 25
т | 10 | 15 | о | 2 | о | 2 | 2 | 10 | 25 | 4 | о | 6 | 2 | 2 | 2 | 0 | 0 | 2 | 15
V | 5 | 5 | 10 | 5 | о | 3 | 5 | 8 | 5 | 41 | 0 | 0 | 10 | о | 0 | о | о | 0 | 3
X | 0 | 12 | о | 0 | 0 | 3 | 3 |  | 324 | 9 | 6 | 9 | о | 12 | 3 | 15 | о | о | 3

TABLE III.

No | Phone | Syll.
1 | a | A | a
2 | е | E | э
3 | I | I | H
4 | э | 0 | о
5 | U | U | y
6 | 0 | @ | θ
7 | U | ^ | Y
8 | i | Y | bI
9 | a: | a: | aa
10 | e: | e: | ээ

VOWEL PHONEME SET

No | Phone | Syll.
11 | i: | i: | ий
12 | э: | о: | 00
13 | u: | u: | y
14 | 0: | @: | ө
15 | U: | ^: | Y
16 | Ay | Ay | ай
17 | ∃у | Oy | Ой
18 | Uy | Uy | У
19 | Ey | Ey | Эй
20 | оу | ^y | Үй

Raters' classification (%)

| aei o u @ ^ y a: e: i: o: u: @: n: ay oy uy e ^y |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  |  |  |  |  |  |  |  |  |  |  |  | a 50 0 0 50 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 |  |  |  |  |  |
|  |  |  |  |  |  |  |  |  |  |  |  | 0 |  |  |  |  |  |
|  |  |  |  |  |  |  |  |  |  |  | 2900 70000000 |  |  |  |  |  |  |
|  |  |  |  |  |  |  |  |  |  |  | 0000 0 0 0 |  |  |  |  |  |  |
| 0 0 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| @ 10 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| 0 0 10 0 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  |  |  |  |  |  |  |  |  |  |  | У 1 1 96 0 0 0 1 0 0 0 0 0 0 0 0 0 0 |  |  |  |  |  |  |
|  |  |  |  |  |  |  |  |  |  |  | a: 99 0 1 00000000000 0 00 000 |  |  |  |  |  |  |
| 0 0100 00 00 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| 2 4 8 0 6 0 0 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  |  |  |  |  |  |  |  |  |  |  |  | 0: 1 0 0 9 0000 0 00 0 0 0 0 0 |  |  |  |  |  |
|  |  |  |  |  |  |  |  |  |  |  |  | : 0 3 0 0 78 17 3 0 0 0 0 0 0 0 0 0 0 0 0 |  |  |  |  |  |
|  |  |  |  |  |  |  |  |  |  |  |  | @ 4 1 60 28 6 0 0 0 0 |  |  |  |  |  |
|  |  |  |  |  |  |  |  |  |  |  |  | 3 0 0 0 10 7 78 1 0 0 0 0 0 000 0 00 |  |  |  |  |  |
|  |  |  |  |  |  |  |  |  |  |  |  | 2y 33 8 2 50 0 4 2 0 0 0 0 0 0 0 0 00 0 0 0 |  |  |  |  |  |
|  |  |  |  |  |  |  |  |  |  |  |  | 00 4 16 2 6 6 12 0 0 0 0 0 0 0 0 0 0 0 0 |  |  |  |  |  |
|  |  |  |  |  |  |  |  |  |  |  |  | uy 0 11 70 0 3 5 10 2 0 0 0 0 0 0 0 0 0 0 0 0 |  |  |  |  |  |
|  |  |  |  |  |  |  |  |  |  |  |  | ey 2 25 73 0 0 0 0 0 0 0 0 0 0 0 0 0 0 000 |  |  |  |  |  |
|  |  |  |  |  |  |  |  |  |  |  | 0 0 10 0 0 0 00 0 0 0 0 00 0 0 |  |  |  |  |  |  |

# ACKNOWLEDGMENT This work is partially supported by ITU-AMD and the Australian Government particularly the Department of Communications, Broadband and the Digital Economy (DBCDE). It is a collaborative project of the following partners including 1) School of Information Technology (SIT), National University of Mongolia (NUM) in Mongolia, and, 2) National Electronics and Computer Technology Center (NECTEC) in Thailand. The authors also wish to thank the speakers for recording their voices. # V. CONCLUSION AND FUTURE WORK This paper presented the evaluation of Mongolian text-tospeech system. Overall quality evaluation gives the best result among other Mongolian TTS systems. On phonetic level, phone confusion test gives not good result, thus we need to improve it. These results will be useful for future improvement of the Mongolian text-to-speech development. To increase intelligibility of phone quality, we first need to increase training speech corpus size for improving modeling accuracy of the Mongolian acoustic models. Currently we have increased the 1,500 sentence speech corpus to 3,000 sentences, which is about 10 hours of speech of both female and male voice. Another point to be considered is optimizing modeling parameters and features of the acoustic models. # VI. REFERENCES [1] B. Batchuluun, D. Batchuluun, and, G. Gombosuren, “Mongolian text to speech conversion tool”, Project Ref.No: 0501A2_L13, Interim report, InfoCon Co., Ltd, Mongolia, http://www.infocon.mn/tts/, 2006. [2] M. Davaatsagaan, and, K. Paliwal, “Diphone-Based Concatenative Speech Synthesis System for Mongolian”, Int’l MultiConference of Engineers and Computer Scientists (IMECS) 2008, Vol I, 2008. [3] B.Sukhbaatar, and, D.Munkhtuya, “A New Approach For The Mongolian Text-To-Speech Conversion”, Proc. Int’l Conference on Electronics, Information, and Communication (ICEIC 2006), Vol. 2, pp.303 – 306, 2006. [4] Otgonbayar, B., “Study of all necessary parameters and characteristics of Mongolian speech synthesis system and establishing functional model for Mongolian spoken language”, Ph.D. thesis, Mongolian University of Science and Technology, Mongolia, 1996. [5] O. BatEnkh, “Investigation on New Approach to Build Speech Synthesis System for Mongolian Language”, Ph.D. thesis, Mongolian University of Science and Technology, Mongolia, 2001. [6] D. Bayanduuren, “Mongolian Speech Recognition”, PAN localization Project Phase II , Mongolia, 2007. [7] I.Dawa, Husal, L. Yue, Y. Y. Ming, Uulang, B. S. Cheng, Batsaihan, Y.Arai, M. Mitsunaga, H. Isahara, and, S. Nakamura, “Multilingual Text – Speech Corpus of Mongolian”, Int’l Symposium on Chinese Spoken Language Processing (ISCSLP 2006), 2006. [8] A.Altangerel, and, B.Damdinsuren, “Research Report on ASR Development for Mongolian”, PAN localization Project Phase II , 2009. [9] Ts., Damdinsuren, “Mongol shine usgiin tovch durem”, Ulaanbaatar, 1946. [10] P. Jaimai, and, A. Chagnaa, “Language Resources for Mongolian”, Human Language Technology for Development (HLTD 2011), pp. 56- 61, 2011. [11] P. Jaimai, and, O. Chimeddorj, “Corpus Building for Mongolian Language”, 6th Workshop on Asian Language Resources, 2008. [12] I. Dawa, S. Okawa, and K. Shirai, “Design and assessment of Mongolian dialectal speech input system,” Proc. EALREW99, Taipei, pp. 45-48, 1999. [13] A. Ayush, and, D. Bayanduren, “Mongolian Speech corpus”, PAN localization Project Phase II, 2009. [14] A. M. Karlsson, and, J. Svantesson, “What happens to consonant clusters in Mongolian speech?”, Rossijskaja akademija nauk / Russian Academy of Science, 2007. [15] H. Chatchawarn, T. Ausdang, T. Nattanun, Ch. Altangerel, “Mongolian speech corpus for text-to-speech development”, proc. Oriental COCOSDA 2011, pp , Hsinchu, Taiwan,26-28 Oct, 2011.