{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "# ProMo - A Speech Prosody Morphing library for Python\n",
    "\n",
    "*An introduction and tutorial*\n",
    "\n",
    "# Section 1.1 An introduction"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr>\n",
    "TABLE OF CONTENTS\n",
    "\n",
    "**Section 1.1: An introduction**\n",
    "- <a href=\"#what_is_promo\">What is ProMo?</a>\n",
    "- <a href=\"#what_is_prosody\">What is Prosody?</a>\n",
    "    - <a href=\"#duration_more_info\">Considerations when resynthesizing duration</a>\n",
    "    - <a href=\"#pitch_more_info\">Considerations when resynthesizing pitch</a>\n",
    "- <a href=\"#requirements.txt\">Requirements</a>\n",
    "- <a href=\"#installing\">Installing ProMo</a>\n",
    "\n",
    "**[Section 1.2: Pitch manipulation tutorial](tutorial1_2_pitch_manipulations.ipynb)** (opens in a new notebook)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Settings things up\n",
    "from os.path import join\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "\n",
    "import IPython\n",
    "inputPath = join('..', 'examples', 'files')\n",
    "\n",
    "from praatio import data_points\n",
    "\n",
    "# Some convenience functions -- we'll be using this a lot\n",
    "def pitchForPlots(pitchFN):\n",
    "    pitchTier = data_points.open2DPointObject(pitchFN)\n",
    "    x, y = zip(*pitchTier.pointList)\n",
    "    return x, y\n",
    "\n",
    "def doPlot(axis, title, pitchFN):\n",
    "    axis.plot(*pitchForPlots(pitchFN))\n",
    "    axis.set_title(title)\n",
    "    axis.set_xlabel(\"time(s)\")\n",
    "    axis.set_ylabel(\"F0(hz)\")\n",
    "    \n",
    "def displayAudio(audioTuple):\n",
    "    for title, wavFN in audioTuple:\n",
    "        print(title)\n",
    "        IPython.display.display(IPython.display.Audio(wavFN))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## An Introduction\n",
    "\n",
    "<a id=\"what_is_promo\"></a>\n",
    "### What is ProMo\n",
    "\n",
    "ProMo is a library for making some complicated prosody manipulations simple.  ProMo comes with code for working with duration and pitch manipulations.\n",
    "\n",
    "The PSOLA algorithm is used to resynthesize the speech.  However, ProMo doesn't implement PSOLA.  It offloads that work to Praat via the praatIO library.\n",
    "\n",
    "Instead, ProMo automates the nitty-gritty work necessary to do certain kinds of manipulations.  Given an audio recording and a duration target or a pitch target, ProMo will output the original audio with a pitch or duration replaced with the target pitch or duration.  **Or** given a source and target audio file, the target pitch or duration can be automatically determined and the desired audio output.  Intermediate steps between the source and the target can also be generated.\n",
    "\n",
    "This is how ProMo got its name (morphing from one target to another) although ProMo's manipulation functions don't actually have to be used for morphing.\n",
    "\n",
    "<a id=\"what_is_prosody\"></a>\n",
    "### What is Prosody?\n",
    "\n",
    "\n",
    "ProMo is a library for manipulating prosody.  What is prosody?  Prosody is the melodic aspect of speech.  The pitch contour or intonation of an utterance.  The length of a syllable.  The loudness of a word.  These are aspects of prosody that speakers manipulate to alter the meaning of what they're saying.\n",
    "\n",
    "Different languages use prosody in different ways:\n",
    "\n",
    "* In Mandarin, each word is assigned one of five pitch contours or tones.  Words differing in tone have different meanings.\n",
    "\n",
    "    * 妈 'mother' [mā] (high tone)  vs 马 'horse' [mǎ] (falling rising tone) \n",
    "\n",
    "* In Arabic, there is a distinction between short and long vowels.  Words differing only in different vowel length have different meaning.\n",
    "\n",
    "    * أكل 'He ate' [akala] vs أكلا \"Those two ate\" [akala:]\n",
    "\n",
    "* In English multisyllabic words, one word has greater emphasis than others--the stressed syllable.  Stressed syllables are longer than unstressed syllables and are pronounced more carefully.\n",
    "\n",
    "    * The test for stress in English is to say a word as if you are surprised.  The stressed syllable is ususally obvious for native speakers.  Eg: banana:'ba.NA.na' or pineapple:'PI.na.pl', \n",
    "    \n",
    "* In English use of pause can alter the interpretation of how pieces of a sentence are connected\n",
    "\n",
    "    * The sentence 'The woman saw the man with the telescope' is ambiguous.  Which scenario is correct?  The woman used a telescope to see a man or a woman saw a man who was holding a telescope.  A slight pause between 'man' and 'with the telescope' leads to a stronger intepretation that the woman has the telescope.  A slight pause before 'the man' leads to a stronger intepretation that the man has the telescope.\n",
    "\n",
    "* In many languages, a rising intonation on a sentence designates a question while a falling intonation designates a statement.\n",
    "\n",
    "    * E.g. 'You're going to the store?' vs 'You're going to the store.'\n",
    "\n",
    "In addition to linguistic meaning, prosody can also be used to convey information about the speaker such as their emotional state or social identity."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"duration_more_info\"></a>\n",
    "#### Considerations when resynthesizing duration\n",
    "\n",
    "To understand duration manipulation, you should have a thorough understanding of consonants and vowels.  We won't be reviewing those here, but you might find it handy to reference the international phonetic alphabet <a href=\"http://www.ipachart.com/international\">with sounds</a> or <a href=\"https://commons.wikimedia.org/wiki/General_phonetics\">with links to wikipedia pages for more detailed information</a>\n",
    "\n",
    "Intuitively, some sounds can easily be lengthened naturally: vowels (a, i, u), fricatives (s, sh), nasals (m, n), liquids (r, l) \n",
    "\n",
    "Other sounds, however, cannot be lengthened naturally: stops (t, k), affricates (ch, dj).\n",
    "\n",
    "In some languages, the duration of speech segments is relatively fixed (Japanese) while in others there is a lot more flexibility (English, Dutch).  If we want to lengthen a word, do we equally lengthen each segment? Or only part of it?  In Japanese, we could equally lengthen all parts of a word.  For English, it might be necessary to weight some segments more than others.  \n",
    "\n",
    "Similarly, what about entire words?  In English, function words like 'if', 'on', or 'the' are generally reduced and pronounced quicker relative to content words (nouns, verbs, adjectives, etc).  If we wanted to increase the length of an utterance, it might sound better if only the content words are increased.  Or maybe not.  It's something to keep in mind.\n",
    "\n",
    "In summary, in a duration resynthesis task:\n",
    "\n",
    "    * Not all sounds are equally lengthened\n",
    "    * There may be language-specific considerations at the syllable or word level.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The original audio file:\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "                <audio controls=\"controls\" >\n",
       "                    <source src=\"data:audio/wav;base64,[encoded data, 49128 characters]\" type=\"audio/wav\" />\n",
       "                    Your browser does not support the audio element.\n",
       "                </audio>\n",
       "              "
      ],
      "text/plain": [
       "<IPython.lib.display.Audio object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The same file doubled in length:\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "                <audio controls=\"controls\" >\n",
       "                    <source src=\"data:audio/wav;base64,[encoded data, 102004 characters]\" type=\"audio/wav\" />\n",
       "                    Your browser does not support the audio element.\n",
       "                </audio>\n",
       "              "
      ],
      "text/plain": [
       "<IPython.lib.display.Audio object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "displayAudio(((\"The original audio file:\", join(inputPath, \"mary1.wav\")),\n",
    "              (\"The same file doubled in length:\", join(inputPath, \"mary1_double_length.wav\"))))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"pitch_more_info\"></a>\n",
    "#### Considerations when resynthesizing pitch\n",
    "\n",
    "Pitch is the fundamental frequency of speech--the lowest frequency produced by the vocal chords during speech.  Speech contains both voiced and unvoiced segments.  There is no voicing for consonants such as [t, k, sh, f] and thus, there is no pitch information in this segments.\n",
    "\n",
    "When it comes to resynthesizing pitch, only voiced segments will be affected by pitch resynthesis.\n",
    "\n",
    "The resynthesis process also depends on the pitch tracker.  For very low pitch or in creaky voice, automatic pitch tracking software may struggle to accurately follow the pitch.  Low pitch is very common at the end of an utterance, when the vocal folds tend to \"wind down\".  Speakers with deep voices might also have more problems having their pitch resynthesized because their voice will more regularly be 'too low' than speakers with higher pitch voices.\n",
    "\n",
    "Furthermore, pitch tracking software can be configured via a number of parameters (such as the minimum or maximum pitch values to consider).  Changing the parameters can lead to drammatically different results.  For these reasons, the output of a pitch tracker should never be assumed to be the absolute 'true' pitch.  \n",
    "\n",
    "Noisy speech environments can also cause problems for the pitch tracking software.  In these cases, the resynthesized pitch quality can be poor.  By poor I mean that either the desired effect will not be perceived (an utterance final rising pitch to signal a question) or the output will be distorted with the speaker sounding robotic or with the presence of \"pops\" in the output.\n",
    "\n",
    "If the pitch is too dramatic of a difference compared with the original audio, \n",
    "\n",
    "In summary, in a pitch resynthesis task:\n",
    "\n",
    "    * Voiced sounds are affected by resynthesis; unvoiced sounds aren't\n",
    "    * Pitch resynthesis in praat depends on pitch tracking software\n",
    "    * Pitch trackers estimate pitch--extracted pitch should not be treated as 'truth'.\n",
    "        * Changing the parameters to the pitch tracker can lead to very different analysis\n",
    "        * Poor recording quality, deep voices, and large differences in pitch can lead to less accurate pitch\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr>\n",
    "# A Tutorial\n",
    "\n",
    "<a id=\"requirements.txt\"></a>\n",
    "## Requirements\n",
    "\n",
    "ProMo requires my python library <a href=\"https://github.com/timmahrt/PraatIO\">PraatIO</a> and <a href=\"http://www.fon.hum.uva.nl/praat/\">Praat</a>.  Praat doesn't require installation.\n",
    "\n",
    "ProMo can also create visualizations of the morph process if you have **matplotlib** installed.\n",
    "\n",
    "Finally, if you want to use the pitch interpolation feature, you need to have **scipy** installed.\n",
    "\n",
    "matplotlib and scipy are both complicated to install, unless you use pip, like so:\n",
    "\n",
    "    pip install matplotlib\n",
    "    pip install scipy\n",
    "    \n",
    "or\n",
    "\n",
    "    python -m pip install matplotlib\n",
    "    python -m pip install scipy\n",
    "\n",
    "<hr>\n",
    "<a id=\"installing\"></a>\n",
    "## Installing ProMo\n",
    "\n",
    "ProMo can easily be installed with pip.  If you have trouble installing ProMo with pip, please visit the github <a href=\"https://github.com/timmahrt/ProMo\">ProMo page</a> for other installation options."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already up-to-date: promo in c:\\python36\\lib\\site-packages\n"
     ]
    }
   ],
   "source": [
    "!pip install promo --upgrade"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "### What's next\n",
    "\n",
    "[Section 1.2: Pitch manipulation](tutorial1_2_pitch_manipulations.ipynb)  (opens in a new notebook)\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
