# Olipy

Olipy is a Python library for artistic text generation. Unlike most
software packages, which have a single, unifying purpose. Olipy is
more like a set of art supplies. Each module is designed to help you
achieve a different aesthetic effect.

# Setup

Olipy is distributed as the `olipy` package on PyPI. Here's how to
quickly get started from a command line:

```
# Create a virtual environment.
virtualenv env

# Activate the virtual environment.
source env/bin/activate

# Install Olipy within the virtual envirionment.
pip install olipy

# Run an example script.
olipy.apollo
```

Olipy uses the [`TextBlob`](https://textblob.readthedocs.org/) library
to parse text. Installing Olipy through `pip` will install
TextBlob as a dependency, but `TextBlob` has extra dependencies (text corpora) which
are _not_ installed by `pip`.  Instructions for installing the extra
dependencies are on the `TextBlob` site, but they boil down to running
[this Python
script](https://raw.github.com/sloria/TextBlob/master/download_corpora.py).

# Example scripts

Olipy is packaged with a number of  scripts which do fun things with
the data and algorithms. You can run any of these scripts from a
virtual environment that has the `olipy` package installed.

* `olipy.apollo`: Generates dialogue between astronauts and Mission
  Control. Demonstrates Queneau assembly on dialogue.
* `olipy.board_games`: Generates board game names and
  descriptions. Demonstrates complex Queneau assemblies.
* `olipy.corrupt` "Corrupts" whatever text is typed in by adding
  increasing numbers of diacritical marks. Demonstrates the
  `gibberish.Corruptor` class.
* `olipy.dinosaurs`: Generates dinosaur names. Demonstrates Queneau
  assembly on parts of a word.
* `olipy.ebooks`: Selects some lines from a public domain text using
  the *_ebooks algorithm. Demonstrates the
  `olipy.gutenberg.ProjectGutenbergText`
  and `olipy.ebooks.EbooksQuotes` classes.
* `olipy.gibberish`: Prints out 140-character string of aesthetically
  pleasing(?) gibberish. Demonstrates the `gibberish.Gibberish` class.
* `olipy.mashteroids`: Generates names and IAU citations for minor
  planets. Demonstrates Queneau assembly on sentences.
* `olipy.sonnet`: Generates Shakespearean sonnets using Queneau assembly.
* `olipy.typewriter`: Retypes whatever you type into it, with added typoes.
* `olipy.words`: Generates common-looking and obscure-looking English
  words.

# Module guide

## alphabet.py

A list of interesting groups of Unicode characters -- alphabets, shapes, and so on.

```
from olipy.alphabet import Alphabet
print(Alphabet.default().random_choice())
# 𝔄𝔅ℭ𝔇𝔈𝔉𝔊ℌℑ𝔍𝔎𝔏𝔐𝔑𝔒𝔓𝔔ℜ𝔖𝔗𝔘𝔙𝔚𝔛𝔜ℨ𝔞𝔟𝔠𝔡𝔢𝔣𝔤𝔥𝔦𝔧𝔨𝔩𝔪𝔫𝔬𝔭𝔮𝔯𝔰𝔱𝔲𝔳𝔴𝔵𝔶𝔷
print(Alphabet.default().random_choice())
# ┌┐└┘├┤┬┴┼═║╒╓╔╕╖╗╘╙╚╛╜╝╞╟╠╡╢╣╤╥╦╧╨╩╪╫╬╴╵╶╷
```

This module is used heavily by gibberish.py.

# corpora.py

This module makes it easy to load datasets from Darius
Kazemi's [Corpora Project](https://github.com/dariusk/corpora), as
well as additional datasets specific to Olipy -- mostly large word
lists which the Corpora Project considers out of scope. (These new
datasets are discussed at the end of this document.)

Olipy is packaged with a complete copy of the data from the Corpora
Project, so you don't have to install anything extra. However,
installing the Corpora Project data some other way can give you
datasets created since the Olipy package was updated.

The interface of the `corpora` module is that used by Allison Parrish's
[`pycorpora`](https://github.com/aparrish/pycorpora/) project. The
datasets show up as Python modules which contain Python data
structures:

```
from olipy import corpora
for city in corpora.geography.large_cities['cities']:
    print(city)
# Akron
# Albequerque
# Anchorage
# ...
```

You can use `from corpora import` ... to import a particular Corpora
Project category:

```
from olipy.corpora import governments
print(governments.nsa_projects["codenames"][0] # prints "ARTIFICE")

from olipy.pycorpora import humans
print(humans.occupations["occupations"][0] # prints "accountant")
```

Additionally, corpora supports an API similar to that provided by the Corpora Project node package:

```
from olipy import corpora

# get a list of all categories
corpora.get_categories() # ["animals", "archetypes"...]

# get a list of subcategories for a particular category
corpora.get_categories("words") # ["literature", "word_clues"...]

# get a list of all files in a particular category
corpora.get_files("animals") # ["birds_antarctica", "birds_uk", ...]

# get data deserialized from the JSON data in a particular file
corpora.get_file("animals", "birds_antarctica") # returns dict w/data

# get file in a subcategory
corpora.get_file("words/literature", "shakespeare_words")
```

## ebooks.py

A module for incongruously sampling texts in the style of the infamous
[https://twitter.com/horse_ebooks](@horse_ebooks). Based on the
[https://twitter.com/zzt_ebooks](@zzt_ebooks) algorithm by Allison
Parrish.

```
from olipy.ebooks import EbooksQuotes
from olipy import corpora
data = corpora.words.literature.fiction.pride_and_prejudice
for quote in EbooksQuotes().quotes_in(data['text']):
    print(quote)
# They attacked him  in various ways--with barefaced
# An invitation to dinner
# Mrs. Bennet
# ...
```

Example scripts for ebooks.py:

* example.ebooks.py: Selects some lines from a Project Gutenberg
  text, with a bias towards the keywords you give it as command-line
  arguments.

## gibberish.py

A module for those interested in the appearance of Unicode
glyphs. Its main use is generating aesthetically pleasing gibberish
using selected combinations of Unicode code charts.

```
from olipy.gibberish import Gibberish
print(Gibberish.random().tweet().encode("utf8"))
# ৠ𐒧𐒇দ𐒔𐒜ৗ𐒃𐒝𐒓আ৭৭উ𐒇৶০ধপ𐒤৯ৰ৪ড়ঐবননত৲ফঌ𐒓৴ৄু০েএঠৰ𐒔𐒥গনি৶ঘ𐒋উঙ𐒤ঙছতাৃীফ৮৬৸উকফ𐒘ইমঢ৭ূণঌঊ𐒇𐒋ীঁিৃ𐒌𐒒৺𐒤৺ভ𐒖৭𐒤ৡৰল𐒊ঢ়ৎ𐒅যথখৱঌ
# ঈঔ৫ঽ𐒔৩়দ𐒋ৠসুয়ঊশ𐒆𐒖𐒁ঔৰসঈ𐒆অ𐒋𐒑𐒨়দ৯ৄ৫ 😘
```

## gutenberg.py

A module for dealing with texts from Project Gutenberg. Strips headers
and footers, and parses the text.

```
from olipy import corpora
from olipy.gutenberg import ProjectGutenbergText
text = corpora.words.literature.nonfiction.literary_shrines['text']
text = ProjectGutenbergText(text)
print(len(text.paragraphs))
# 1258
```

## ia.py

A module for dealing with texts from Internet Archive.

```
import random
from olipy.ia import Text

# Print a URL to the web reader for a specific title in the IA collection.
item = Text("yorkchronicle1946poqu")
print(item.reader_url(10))
# https://archive.org/details/yorkchronicle1946poqu/page/n10

# Pick a random page from a specific title, and print a URL to a
# reusable image of that page.
identifier = "TNM_Radio_equipment_catalog_fall__winter_1963_-_H_20180117_0150"
item = Text(identifier)
page = random.randint(0, item.pages-1)
print(item.image_url(page, scale=8))
# https://ia600106.us.archive.org/BookReader/BookReaderImages.php?zip=/30/items/TNM_Radio_equipment_catalog_fall__winter_1963_-_H_20180117_0150/TNM_Radio_equipment_catalog_fall__winter_1963_-_H_20180117_0150_jp2.zip&file=TNM_Radio_equipment_catalog_fall__winter_1963_-_H_20180117_0150_jp2/TNM_Radio_equipment_catalog_fall__winter_1963_-_H_20180117_0150_0007.jp2&scale=8
```

## letterforms.py

A module that knows things about the shapes of Unicode glyphs.

`alternate_spelling` translates from letters of the English alphabet
to similar-looking characters.

```
from olipy.letterforms import alternate_spelling
print(alternate_spelling("I love alternate letterforms."))
# ヱ 𝑳𝖮Ⓥ𝙀 𝚊𝓵┯⒠┌𝐍ａ⫪𝖊 𝐋𝖾ߙ𝓉ᥱ𝙧ߓ𝕠┍ጠ𝑆.
```

## markov.py

A module for generating new token lists from old token lists using a
Markov chain.

Olipy's primary purpose is to promote alternatives to
Markov chains (such as Queneau assembly and the *_ebooks algorithm),
but sometimes you really do want a Markov chain. Queneau assembly is
usually better than a Markov chain above the word level (constructing
paragraphs from sentences) and below the word level (constructing
words from phonemes), but Markov chains are usually better when
assembling sequences of words.

markov.py was originally written by Allison "A. A." Parrish.

```
from olipy.markov import MarkovGenerator
from olipy import corpora
text = corpora.words.literature.nonfiction.literary_shrines['text']
g = MarkovGenerator(order=1, max=100)
g.add(text)
print(" ".join(g.assemble()))
# The Project Gutenberg-tm trademark.                    Canst thou, e'en thus, thy own savings, went as the gardens, the club. The quarrel occurred between
# him and his essay on the tea-table. In these that, in Lamb's day, for a stray
# relic or four years ago, taken with only Adam and _The
# Corsair_. Writing to his home on his new purple and the young man you might
# mean nothing on Christmas sports and art seriously instead of references to
# the heart'--allowed--yet I got out and more convenient.... Mr.
```

## mosaic.py

Tiles Unicode characters together to create symmetrical mosaics.
gibberish.py uses this module as one of its techniques. Includes
information on Unicode characters whose glyphs appear to be mirror
images.

```
from olipy.mosaic import MirroredMosaicGibberish
mosaic = MirroredMosaicGibberish()
print(mosaic.tweet())
# ▛▞ ▙▞▙▟▚▟ ▚▜
# ▛▞▞ ▞▛▜▚ ▚▚▜
#  ▞▙  ▞▚  ▟▚ 
# ▙▚▚ ▚▙▟▞ ▞▞▟
# ▙▚ ▛▚▛▜▞▜ ▞▟

print(gibberish.tweet())
# 🙌🙌😯📶🙌👍👍🙌📶😯🙌🙌
#  📶🙌😯🙌🕠🕠🙌😯🙌📶 
# 🚂💈🎈🔒🚲🕃🕃🚲🔒🎈💈🚂
#  📶🙌😯🙌🕠🕠🙌😯🙌📶 
# 🙌🙌😯📶🙌👍👍🙌📶😯🙌🙌

```

## queneau.py

A module for Queneau assembly, a technique pioneered by Raymond
Queneau in his 1961 book "Cent mille milliards de poèmes" ("One
hundred million million poems"). Queneau assembly randomly creates new
texts from a collection of existing texts with identical structure.

```
from olipy.queneau import WordAssembler
from olipy.corpus import Corpus
assembler = WordAssembler(Corpus.load("dinosaurs"))
print(assembler.assemble_word())
# Trilusmiasunaus
```

## randomness.py

Techniques for generating random patterns that are more sophisticated
than `random.choice`.

### `Gradient`

The `Gradient` class generates a string of random choices that are
weighted towards one set of options near the start, and weighted
towards another set of options near the end.

Here's a gradient from lowercase letters to uppercase letters:

```
from olipy.randomness import Gradient
import string
print("".join(Gradient.gradient(string.lowercase, string.uppercase, 40)))
# rkwyobijqQOzKfdcSHIhYINGrQkBRddEWPHYtORB
```

### `WanderingMonsterTable`

The `WanderingMonsterTable` class lets you make a weighted random selection from 
one of four buckets. A random selection from the "common" bucket will show up 65% of the time, a 
selection from the "uncommon" bucket 20% of the time, "rare" 11% of the time, and "very rare" 4% of 
the time. (It uses the same probabilities as the first edition of Advanced Dungeons & Dragons.)

```
from olipy.randomness import WanderingMonsterTable

monsters = WanderingMonsterTable(
         common=["Giant rat", "Alligator"],
         uncommon=["Orc", "Hobgoblin"],
         rare=["Mind flayer", "Neo-otyugh"],
         very_rare=["Flumph", "Ygorl, Lord of Entropy"],
)
for i in range(5):
    print monsters.choice()
# Giant rat
# Alligator
# Alligator
# Orc
# Giant rat
```

tokenizer.py
------------

A word tokenizer that performs better than NLTK's default tokenizers
on some common types of English.

```
from nltk.tokenize.treebank import TreebankWordTokenizer
s = '''Good muffins cost $3.88\\nin New York. Email: muffins@example.com'''
TreebankWordTokenizer().tokenize(s)
# ['Good', 'muffins', 'cost', '$', '3.88', 'in', 'New', 'York.', 'Email', ':', 'muffins', '@', 'example.com']
WordTokenizer().tokenize(s)
# ['Good', 'muffins', 'cost', '$', '3.88', 'in', 'New', 'York.', 'Email:', 'muffins@example.com']
```

typewriter.py
-------------

Simulates the Adler Universal 39 typewriter used in _The Shining_ and
the sorts of typos that would be made on that typewriter. Originally
written for [@a_dull_bot](https://botsin.space/@adullbot).

```
from olipy.typewriter import Typewriter
typewriter = Typewriter()
typewriter.type("All work and no play makes Jack a dull boy.")
# 'All work and no play makes Jack a dull bo6.'
```

# Extra corpora

Olipy makes available several word lists and datasets that aren't in
the Corpora Project. These datasets (as well as the standard Corpora
Project datasets) can be accessed through the `corpora` module. Just
write code like this:

```
from olipy import corpora
nouns = corpora.words.common_nouns['abstract_nouns']
```

### `corpora.geography.large_cities`

Names of large U.S. and world cities.

### `corpora.geography.us_states`

The fifty U.S. states.

### `corpora.language.languages`

Names of languages defined in ISO-639-1

### `corpora.language.unicode_code_sheets`

The name of every Unicode code sheet, each with the characters found on that sheet.

### `corpora.science.minor_planet_details`

'name', 'number' and IAU 'citation' for named minor planets
(e.g. asteroids) as of July 2013. The 'discovery' field contains
discovery circumstances. The 'suggested_by' field, when present, has
been split out from the end of the original IAU citation with a simple
heuristic. The 'citation' field has then been tokenized into sentences
using NLTK's Punkt tokenizer and a set of custom abbreviations.

Data sources: 
 http://www.minorplanetcenter.net/iau/lists/NumberedMPs.html
 http://ssd.jpl.nasa.gov/sbdb.cgi

This is more complete than the Corpora Project's `minor_planets`,
which only lists the names of the first 1000 minor planets.

### `corpora.words.adjectives`

About 5000 English adjectives, sorted roughly by frequency of occurrence.

### `corpora.words.common_nouns`

Lists of English nouns, sorted roughly by frequency of occurrence.

Includes:

* `abstract_nouns` like "work" and "love".
* `concrete_nouns` like "face" and "house".
* `adjectival_nouns` -- nouns that can also act as adjectives -- like "chance" and "light".

### `corpora.words.common_verbs`

Lists of English verbs, sorted roughly by frequency of occurrence.

* `present_tense` verbs like "get" and "want".
* `past_tense` verbs like "said" and "found".
* `gerund` forms like "holding" and "leaving".

### `corpora.words.english_words`

A consolidated list of about 73,000 English words from the FRELI
project. (http://www.nkuitse.com/freli/)

### `corpora.words.scribblenauts`

The top 4000 nouns that were 'concrete' enough to be summonable in the
2009 game _Scribblenauts_. As always, this list is ordered with more common
words towards the front.

### `corpora.words.literature.board_games`

Information about board games, collected from BoardGameGeek in July
2013. One JSON object per line.

Data source:
 http://boardgamegeek.com/wiki/page/BGG_XML_API2


### `corpora.words.literature.fiction.pride_and_prejudice`

The complete text of a public domain novel ("Pride and Prejudice"
by Jane Austen).

### `corpora.words.literature.nonfiction.apollo_11`

Transcripts of the Apollo 11 mission, presented as dialogue, tokenized
into sentences using NLTK's Punkt tokenizer. One JSON object per line.

Data sources:
 The Apollo 11 Flight Journal: http://history.nasa.gov/ap11fj/
 The Apollo 11 Surface Journal: http://history.nasa.gov/alsj/
 "Intended to be a resource for all those interested in the Apollo
  program, whether in a passing or scholarly capacity."

### `corpora.words.literature.nonfiction.literary_shrines`

The complete text of a public domain nonfiction book ("Famous Houses
and Literary Shrines of London" by A. St. John Adcock).

### `corpora.words.literature.gutenberg_id_mapping`

Maps old-style (pre-2007) Project Gutenberg filenames to the new-style
ebook IDs. For example, "/etext95/3boat10.zip" is mapped to the
number 308 (see http://www.gutenberg.org/ebooks/308). Pretty much
nobody needs this.
