**Natural Language Processing in Action**

**Hobson Lane, Cole Howard, and Hannes Hapke**

**1. Tell us about yourself.**

-   Hobson has more than 15 years of experience building autonomous
    > systems that make important decisions on behalf of humans. At
    > [*Talentpair*](http://talentpair.com) Hobson teaches machines to
    > read and understand resumes with more patience, consistency, and
    > less bias than most humans. Hobson is passionate about openness
    > and AI. He’s an active contributor to Open Source, Open Science
    > projects such as Keras, SciKit Learn, PyBrain, PUGNLP,
    > and ChatterBot. He has published papers and presented talks at
    > [*AIAA*](http://mstl.atl.calpoly.edu/~workshop/archive/2006/Spring/12-Lane-Nanosat.pdf),
    > [*PyCon*](https://us.pycon.org/2016/schedule/presentation/1778/),
    > [*PAIS*](https://www.academia.edu/attachments/36858143/download_file?st=MTQ3NDQ4ODkzMyw2Ny4xMzYuMTI5LjE5NiwxMTMzNDc5NQ%3D%3D&s=swp-preview-selector-dropdown),
    > and [*IEEE*](http://rjwagner49.com/Iris/SRW-Paper-13.pdf) and has
    > been awarded [*numerous
    > patents*](http://patents.justia.com/inventor/hobson-lane) in
    > Robotics and Automation.

-   Hobson’s first childhood dream was to sail around the world. Having
    > accomplished that, he’s now pursuing his second dream, building
    > friendly bots that can have intelligent conversations with humans
    > and each other.

-   [*Hannes*](https://www.linkedin.com/in/hanneshapke) is an Electrical
    > Engineer turned Data Scientist with *deep* experience in
    > *deep* learning. Neural networks changed his life. He became
    > fascinated with neural networks while investigating novel ways to
    > control renewable energy power plants effectively in High School.
    > Hannes loves to automate software development and machine
    > learning pipelines. He’s the one you can thank for all the
    > time-saving tips and “scaling” advice in this book as well as
    > integration of the Chatbot microservices with SMS messaging. He’s
    > a regular speaker at conferences like PyCon, Hack University, and
    > [*Open Source Bridge*](http://opensourcebridge.org/users/1588),
    > and he’s continually climbing the leaderboard at Kaggle using the
    > tools he developed for this book.

-   [*Cole*](https://www.linkedin.com/in/uglyboxer) is a carpenter and
    > writer turned Data Scientist and Deep Learning expert. A lifelong
    > hunter of patterns, he found his true home in the world of
    > Artificial Neural Networks. He has developed large-scale
    > e-commerce recommendation engines and state-of-the-art neural nets
    > for hyperdimensional machine intelligence systems (“deep learning”
    > neural nets) which perform at the top of the leader board for the
    > Kaggle competitions. He has presented talks on Convolutional
    > Neural Nets, Recurrent Neural Nets and their roles in natural
    > language processing at the [*Open Source Bridge
    > Conference*](http://opensourcebridge.org/users/2606) and
    > Hack University. He currently is deeply fascinated by the shape of
    > memory in a neural network.

-   Like our target audience, the authors are all self-taught experts in
    > Machine Intelligence (Artificial Intelligence, Deep Learning, pick
    > your buzzword) and achieved success by teaching those around them
    > and exploring resources available from publishers such as Manning.

**2. Tell us about the book.**

-   We want to show the reader how to build a system that can read,
    > "understand", and compose meaningful, English text, and we want to
    > explain why contributing to an ecosystem of these prosocial bots
    > is so important.

-   This book will show developers (and budding data scientists) how to
    > build machines that understand and act intelligently on
    > information gleaned from natural language. These machines will:

    -   Compose tweets

    -   Summarize text

    -   Participate in conversations on Twitter

    -   Answer questions

    -   Gage the sentiment of English statements

    -   Gage the sensitivity and kindness of a statement

    -   Estimate the era or “cultural context” of a music lyric or
        > colloquial phrase

    -   Predict the human reaction to a natural language statement

    -   Compose passages for books such as this

> Both the “how” and “why” of these machines will be explained to the
> reader. Readers will build the intuition and skills required to extend
> the capability of the tools demonstrated in this book.

-   Machines must be taught to understand human conversation and
    > communicate with us in helpful ways if we can expect them to be
    > supportive of us in the future. Propagating and “diversifying the
    > genome” of prosocial, natural-language-aware machines helps ensure
    > a bright future for humanity in cooperation with machines, and may
    > accelerate the discovery of technological solutions to pressing
    > societal challenges.

-   There are 2 mathematical techniques for compressing high dimensional
    > data (the millions of dimensions of data contained in all the
    > possible natural language phrases and their meaning) into a small
    > number of topics or “meanings” that can be acted on by a machine.

    -   Statistical or Data-Based: A large amount of data (texts) can be
        > processed by a machine without human supervision to train
        > machines to behave intelligently, but the volume of text
        > required grows with the complexity/intelligence of the
        > behavior desired

    -   Ontology and Grammar-Based: Grammar-based techniques depend on a
        > large and flexible ontology of information about the world and
        > words to describe that information as well as the grammar
        > rules for communication of those facts. This ontology usually
        > requires significant human curation, but smaller ontologies
        > can be assembled from open sources without much effort and
        > significantly improve the performance of chat bots.

> This book will show the reader how to build a pipeline built on both
> of these two approaches and includes a sufficient volume of accurate
> data to achieve satisfactory results on each of the tasks outlined.
> Other books about natural language processing leave the training and
> data acquisition as an “exercise for the reader” and often only
> discuss toy problems or simplified applications. In contrast, this
> book will show the reader how to build fully-trained end-to-end
> natural language pipelines capable of prediction tasks with reasonable
> accuracy and interactive dialog in domains sufficiently broad to be
> useful and interesting. It will explain each step required to build an
> intelligent, [*prosocial
> machine*](https://docs.google.com/document/d/1Gn3w2OOyHaCfQxOGJwaLbou_-PdUVNZyrtvujm8-1w4/edit?ts=57be90a9),
> including the “answer” in the form of a library of open source
> software packages that accomplishes the training and system
> implementation described in the book. Each software package will
> include links to the open data sources (machine-readable corpora and
> training sets) that were used to train it as well as fully functional
> online APIs (microservices) to each of the pieces of the pipeline.
> Only by building and interacting with this intelligent machine can the
> reader develop an intuition and understanding of the underlying
> principles that make it possible.

-   This book will be an **“In Action”** tutorial best read with a
    > laptop nearby.

    -   Example implementations of the tools the book describes will be
        > open sourced under a liberal license (MIT) and deployed as web
        > apps and APIs for the reader to play with and modify while
        > reading the book

    -   The tools the book describes will be used to write the book!
        > Some of the text within the book will have been corrected and
        > edited by these tools. It may even be possible to have the
        > closing and opening paragraphs written entirely by one of
        > these tools (the text summarizer), without any human
        > interaction, justifying a sensational title such as “A
        > Book-Writing Machine In Action” or “Learn How to Build a
        > Machine that can Write a Book Like This”

**3. Tasks in the domain of this book**

-   Acquire, clean, and label a “reading list” for your machine

-   Visualize statistics and patterns in your corpus (reading list)

-   Train the machine to glean “meaning” from this reading list

-   Visualize the domain of knowledge your machine has attained

-   Train the machine to perform useful NL tasks (such as composing new
    > text or predicting reader reaction to an English statement)

-   Construct a neural network appropriate for training on natural
    > language texts for performing many of the tasks in this book.

-   Generate a summary of a large document (such as this book)

-   Generate phrases with the meaning requested by a user in the style
    > of those from a subset of one’s reading list (corpus).

-   Suggest phrasing and rewording of existing texts to optimize for
    > some measure of quality such as readability, grammaticality, an
    > empathetic tone, style-similarity to another text, or conciseness.

**4. The minimally-qualified reader (MQR)**

The MQR will have knowledge of High School algebra and have had some
exposure to vectors and matrices as well as simple statistical measures
such as mean and variance (or standard deviation). They will also have
programmed in some modern programming language, such as C++, Python,
Java, or Javascript. They will have seen a “curve fit” or linear
regression and understand that it is possible for a simple machine (such
as a programmable calculator) to calculate the slope and “offset” of a
line that most closely “fits” data presented in a 2-D scatter plot. They
will have seen a formula for a line such as \`y = a \* x + b\` before.
They will have an aptitude for interpreting graphs and connecting the
patterns they see to the statistics they can compute in a computer
programming language (like the min, max, mean and variance).

-   The ideal MQR will be a University or advanced high-school student
    > with a passion for solving technical (math and science) problems.

**The MQR will know basic math + computer science concepts**

-   Vectors (arrays) & Matrices (arrays of arrays)

-   Sets (unordered arrays when all elements are unique within the set)

-   Strings

-   Arrays, Functions, Classes, Loops, Conditionals

**The MQR will know basic English language concepts**

-   Words, Nouns, Verbs, Proper Nouns

-   Sentences, Phrases

-   Contractions, Abbreviations, Acronyms

-   Possessive Word Forms, Plural Forms

-   A grammar rule or two (such as one or two requirements for a
    > valid sentence)

**The MQR will not be required to be a proficient python programmer, but
should know (or quickly figure out) how to install and run python
packages on their machine.**

**5. Q&A**

How is it possible to build a machine that can compose valid sentences
without teaching it all the rules of English grammar and the logic
behind the meaning of all the words in the English language? How is it
possible for a machine to interact with other humans in a chatroom
without sounding mechanical, repetitive, “programmed”?

**6. Tell us about the competition and the ecosystem.**

-   Numerous books on Natural Language Processing are available,
    > including several that teach the use of python for NLP?

-   It provides a working state-of-the-art system including the data
    > required to make it viable and intelligent. And it provides
    > insights into human cognition and analogous machine intelligence
    > processes that other books neglect.

-   My favorite books on the topic:

    -   Jurafsky’s *Speech and Language Processing* Manning’s
        > *Foundations of Statistical NLP*

    -   Grus’s *Data Science from Scratch*

    -   Russel & Norvig’s *Artificial Intelligence a Modern Approach*

    -   Bird’s *Natural Language Processing with Python*

    -   Perkins’ *Python 3 Text Processing with NLTK 3 Cookbook*

-   What are the most important web sites and companies associated with
    > this topic?

    -   Coursera: e.g. [*intro to
        > NLP*](https://www.coursera.org/learn/natural-language-processing)

    -   Kaggle: e.g. [*word2vec on
        > movies*](https://www.kaggle.com/c/word2vec-nlp-tutorial)

    -   Udacity: e.g.
        > [*cs271*](http://catalog.oregonstate.edu/CourseDetail.aspx?subjectcode=CS&coursenumber=271)

    -   MIT Open Courseware: e.g.
        > [*6-864*](http://ocw.mit.edu/courses/electrical-engineering-and-computer-science/6-864-advanced-natural-language-processing-fall-2005/)

    -   Carnegie Mellon Univ.: [*Never Ending Language
        > Learning*](http://rtw.ml.cmu.edu/rtw/kbbrowser/)

    -   Princeton Univ.: Chomsky,
        > [*WordNet*](https://wordnet.princeton.edu/)

    -   Stanford Univ.:
        > [*cs224n*](http://web.stanford.edu/class/cs224n/), Jurafsky’s
        > [*cs124*](http://web.stanford.edu/class/cs124/lec/chatbot.pdf)

    -   Google:

        -   [*cloud.google.com/natural-language/*](https://cloud.google.com/natural-language/)

        -   [*W*](https://radimrehurek.com/gensim/models/word2vec.html)[*ord2vec*](https://radimrehurek.com/gensim/models/word2vec.html)

        -   [*Norvig’s spelling
            > corrector*](http://norvig.com/spell-correct.html)

        -   [*NGram Viewer
            > API*](http://storage.googleapis.com/books/ngrams/books/datasetsv2.html)
            > 15% of all books ever!

    -   Gensim package: [*gensim
        > tutorial*](https://radimrehurek.com/gensim/tutorial.html)

    -   [*Lexica Links*](http://saifmohammad.com/WebPages/lexicons.html)

    -   [*Gutenberg
        > project*](https://www.gutenberg.org/wiki/Gutenberg:Offline_Catalogs)
        > All books before 1940

    -   Microsoft: Tai

-   Where do others interested in this topic gather online?

    -   Coursera

    -   Kaggle

    -   Udacity

    -   Hack University

    -   Local Python User Group Meetups

**7. Book size**

-   300 pages

-   80 diagrams, tables, pictures, graphics

-   150 short code snippets

**8. Contact information**

> Names on contract: Hobson Lane, Cole Howard, Hannes Hapke
>
> Names on cover: (same)
>
> Total Good
>
> 2004 SW Jefferson St \#614
>
> Portland, OR 97201
>
> Phone: 503.974.6274
>
> Skype: codeninja1
>
> Nonprofit:
> [*total*](http://totalgood.com/)[*≡*](http://totalgood.com/)[***GOOD***](http://totalgood.com/)
>
> Blog: [*hobsonlane.com*](https://github.com/hobson)
>
> LinkedIn:
> [*linkedin.com/in/hobsonlane*](https://www.linkedin.com/in/hobsonlane)
>
> Github: [*github.com/totalgood*](https://github.com/totalgood)
>
> Twitter: [*@hobsonlane*](https://twitter.com/hobsonlane) and
> [*@goodtotal*](https://twitter.com/goodtotal)
>
> Stack Exchange: [*@hobs*](http://stackoverflow.com/users/623735/hobs)

**9. Schedule**

-   14 Chapters

    -   Ch 1: Oct 15th

    -   Ch 2: Oct 31st

    -   ⅓ manuscript (Ch 1-5): Jan 31st

    -   ⅔ manuscript (Ch 6-10): March 31st

    -   Complete manuscript (Ch 1-14): May 18th

-   Are there any critical deadlines for the completion of this book?

    -   Several competitors likely to publish on this topic in 2017

    -   PyCon 2017 May 18th (when we will present excerpts from
        > the book)

**10. Table of Contents**

> **Natural Language Processing in Action**

Part 1 First steps

1 Natural Language Processing

> 1.1 What is Natural Language Processing
>
> 1.2 Grammar-based Approaches (historical perspective)
>
> 1.3 Statistical Approaches

2 Machines that Understand Text

> 2.1 The Magic behind Google Search
>
> 2.2 Math with Words (Word2Vec)
>
> 2.3 Writing Assistance (Hemingway App, Resume Writing Coaches)

Part 2 Your Bot’s Summer Reading List

3 Acquiring Words

> 3.1 Finding Corpora and Lexica without undue copyright restrictions
>
> 3.2 Labeled Corpora and Lexica
>
> 3.3 Scraping and cleaning text
>
> 3.4 Twitter
>
> 3.5 Google NGram API
>
> 3.6 Gutenberg Project
>
> 3.7 Storage

4 Build Your Vocabulary

> 4.1 Overview of a NLP pipeline
>
> 4.2 Tokenization and Character-Level Models
>
> 4.2 Word Stemming and Case Normalization
>
> 4.3 Semantic stemming

4.3 Transcoding (e.g. slang words/phrases; POS tagging; word
disambiguation)

> 4.4 N-Grams and Bags of Words
>
> 4.5 TFIDF: Frequency Counting and Statistical Filters
>
> 4.6 Rare words, Proper Nouns, and IDs (URLs)
>
> 4.7 Common words (stop words)
>
> 4.8 Zipf’s Law
>
> 4.9 Interesting Word-Usage Stats, History

5 Chunking

> 5.1 Sentence Segmentation
>
> 5.2 Paragraph/Chapter/Topic Segmentation
>
> 5.3 What makes a Document? Context?
>
> 5.4 Unsupervised Labeling/Clustering

6 Semantics

> 6.1 SVD and PCA
>
> 6.2 Latent Semantic Indexing, Principle Component Regression
>
> 6.3 Topic Vector Magic

7 You Slice, I Choose

> 7.1 Linear Regression
>
> 7.2 Nonlinear Regression on Large Data Sets (SGD)
>
> 7.3 Logistic Regression
>
> 7.4 Support Vector Machines

Part 3 Communicating in the Wild

8 Style and Tone

> 8.1 Sentiment Analysis
>
> 8.2 Sarcasm Detection

8.3 [*Choosing Favorites*](https://github.com/totalgood/twip)

8.4 Imitating the Manning Corpus of Technical Books

9 Making Decisions (Finite State Machines)

> 9.1 [*Regular Expressions*](http://stackoverflow.com/a/525029/623735)
>
> 9.2 [*Generative Grammar*](http://www.nltk.org/book/ch08.html)
>
> 9.3 [*Finite State
> Machines*](http://web.stanford.edu/class/cs124/lec/chatbot.pdf) for
> dialog management (AKA phone trees)
>
> 9.4 Took the Words Out of My Mouth ([*predictive
> text*](https://veekaybee.github.io/markov-in-python/) with a [*Markov
> Models*](http://www.stat.purdue.edu/~mdw/CSOI/MarkovLab.html))

10 Special Agent Chatty

10.1 Goal-based dialog (e.g. x.ai, MS paperclip, Google Search)

10.2 [*Question Answering*](http://www.aclweb.org/anthology/Y13-1042)

10.2 [*Let’s Chat*](https://github.com/gunthercox/ChatterBot)
(conversation agents)

10.3 [*Sarcasm*](http://www.thesarcasmdetector.com/about/) and Humor

11 [*Neural
Nets*](http://www.wildml.com/2016/04/deep-learning-for-chatbots-part-1-introduction/)

> 11.1 [*Deep
> Learning*](https://adeshpande3.github.io/adeshpande3.github.io/The-9-Deep-Learning-Papers-You-Need-To-Know-About.html)
>
> 11.2 CNNs
>
> 11.3 RNNs
>
> 11.4 LSTMs
>
> 11.5 Word2Vec

12 A Sounding Board

> 12.1 Grammar filters
>
> 12.2 Style/Sentiment filters
>
> 12.3 Adversarial Neural Networks

13 Hyperparameter Optimization

> 13.1 Heuristics
>
> 13.2 Semi-random Search
>
> 13.3 Never-Ending Learning

14 Playing Out of Your League (Scaling)

> 14.1 Distributed Approaches
>
> 14.2 Hyper parallelization (GPUs)
>
> 14.3 Problem segmentation (analogous to brain lobes)

15\. Summary

15.1 What You’ve Learned

15.2 What Your Bot Has Learned (My Bot Wrote this Section)

15.3 What Next?
