---
layout: post
title: Bioinformatics FAQ Bot
author: Hobson Lane, Travis Harper
date: Dec 19, 2019
subtitle: SD Python Monthly Meetup
# theme: dark  # default, simple, sky, beige, serif, solarized, blood, moon, night, black, league , white
highlight: pygments
center: false
# css: styles.css

# Rendering:
#
# Put `reveal.js/` in the same directory as this markdown file then
# build with `pandoc -s -t revealjs 2019-12-19--FAQ-Bot.md -o 2019-12-19--Faq-Bot.html`
---

# Bioinformatics FAQ Bot

San Diego Python Monthly Meetup
by Hobson Lane, Travis Harper
Oct 24, 2019

# Bioinformatics for Real

UCSD Extension **Data Science for Digital Health**
Enroll at: **[bit.ly/ucsd-ds](http://bit.ly/ucsd-ds)**
Discount Code ($100 off): UCSDDSDHWI20

# Thank you Travis!

- Analyzing WikiQA
- Architecting a Transformer

# QA Bot

- WikiQA
- ANSQ

# SOA

![WikiQA State of the Art](media/wikiqa-soa.png)

# Basic Search QA

- Find a question DB (`key`)
- Respond with associated answer (`value`)

# Infinite Search QA

- Translate question to a statement
- Search Wikipedia

# Example: Q -> A

```python
>>> question = "Who discovered radiation?"
>>> statement = question.replace('Who', '[MASK] [MASK]')
>>> statement
"[MASK] [MASK] discovered radiation."
```

# Search Results

DuckDuckGo: ["discovered radiation"](https://duckduckgo.com/?q=%22discovered+radiation.%22&t=h_&ia=web)

![Person that discovered radiation](media/discovered-radiation.png)

# Answer

## Marie Curie

# Scalable Search: `O(log(N))`

- Discrete index
- Sparse BOW vectors

# Synonyms & Typos

- Stemming
- Lemmatizing
- Spelling Corrector
- BPE (bytepair encoding)

# Examples

- Full text search in Postgres
- Trigram indexes in Databases
- Ellastic Search

# Prefilter

- Page rank
- Sparse TFIDF vectors

# Examples

- Full Text (keywords): O(log(N))
- TFIDF (Ellastic Search): O(log(N))
- TFIDF + Semantic Search: O(L)

# Academic Search Approachs

- Edit distance


# Knowledg-based QA

- Extract information from Wikipedia
- Build *Knowledge Graph* in DB
- Query Knowledge Graph
- Inference on Knowledge Graph

<aside class="notes">


If you just need to find an edge in your graph, like an "is-a" relationship you can do a simple query of a relational database for the node you're looking for.
Inference is only required when you need to go deeper into the graph to find an answer to a question that doesn't have a direct answer in your graph edges.

</aside>


# Transformer

![Transformer Test Example Output](media/transformer-test.png)

# Transformer

![Transformer Wizard of Oz Question Answers](media/transformer-woz-output.png)

# `qary` imports

```python
from qary.etl import glossaries
from qary import spacy_language_model

nlp = spacy_language_model.load('en_core_web_md')
```

# `qary.glossary_bots.Bot`

```python
class Bot:

    def __init__(self, domains=('dsdh',)):
        global nlp
        self.nlp = nlp
        self.glossary = glossaries.load(domains=domains)
        self.glossary.fillna('', inplace=True)
        self.vector = dict()
        self.vector['term'] = pd.DataFrame({s: nlp(s or '').vector for s in self.glossary['term']})
        self.vector['definition'] = pd.DataFrame({s: nlp(s or '').vector for s in self.glossary['definition']})
```

#  `qary.glossary_bots.Bot.reply`

```python
    def reply(self, statement):
        """ Suggest responses to a user statement string with [(score, reply_string)..]"""
        responses = []
        match = re.match(r'\b(what\s(is|are))\b([^\?]*)(\?*)', statement.lower())
        if match:
            responses.append((1, str(match.groups())))
        else:
            responses = [(1.0, "I don't understand")]
        return responses
```

# `glossary_bots` test

```python
>>> from qary.skills import glossary_bots
>>> bot = glossary_bots.Bot()
>>> bot.nlp.lang
'en'
>>> list(bot.vector['term'].keys())
['ACP (American College of Physicians)',
 'AKI (Acute Kidney Injury)',
 'Allele',
 ...
 'Xiaoice']
```

# `glossary_bots` test

```python
>>> bot.reply('Nucleotide')
[(1.0, "I don't understand")]
>>> bot.reply('What is a Nucleotide')
[(1, "('what is', 'is', ' a nucleotide', '')")]
```

Now strip whitespace and stop words and look up the definition in `bot.glossary`.

Or use the semantic vectors...

# regex hack

```python
match = re.match(
    r"\b(what\s+(is|are)\s*(not|n't)?\s+(a|an|the))\b([^\?]*)(\?*)",
    statement.lower())
if match:
    try:
        responses.append((1,
            self.glossary['definition'][match.groups()[-2].strip().lower()]))
    except KeyError:
        responses.append((1,
            str(match.groups())))
```

# `glossary_bots` works!

```python
>>> bot = Bot()
>>> bot.reply('allele')
[(1.0, "I don't understand")]
>>> bot.reply('What is a nucleotide?')
[(1,
  'The basic building blocks of DNA and ... Guanine (G), Cytosine ... ')]
```
