Metadata-Version: 2.0
Name: textacy
Version: 0.5.0
Summary: Higher-level text processing, built on spaCy
Home-page: https://github.com/chartbeat-labs/textacy
Author: Burton DeWilde
Author-email: burtdewilde@gmail.com
License: Apache
Download-URL: https://pypi.python.org/pypi/textacy
Description-Content-Type: UNKNOWN
Keywords: textacy,spacy,nlp,text processing,linguistics
Platform: UNKNOWN
Classifier: Development Status :: 4 - Beta
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python
Classifier: Programming Language :: Python :: 2
Classifier: Programming Language :: Python :: 2.7
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.4
Classifier: Programming Language :: Python :: 3.5
Classifier: Programming Language :: Python :: 3.6
Classifier: Natural Language :: English
Classifier: Topic :: Text Processing :: Linguistic
Requires-Dist: cachetools (>=2.0.0)
Requires-Dist: cytoolz (>=0.8.0)
Requires-Dist: ftfy (<5.0.0,>=4.2.0)
Requires-Dist: ijson (>=2.3)
Requires-Dist: networkx (>=1.11)
Requires-Dist: numpy (<2.0.0,>=1.9.0)
Requires-Dist: pyemd (>=0.3.0)
Requires-Dist: pyphen (>=0.9.4)
Requires-Dist: python-levenshtein (>=0.12.0)
Requires-Dist: requests (>=2.10.0)
Requires-Dist: scipy (>=0.17.0)
Requires-Dist: scikit-learn (>=0.17.0)
Requires-Dist: spacy (>=2.0.0)
Requires-Dist: tqdm (>=4.11.1)
Requires-Dist: unidecode (>=0.04.19)
Requires-Dist: backports.csv (>=1.0.1); python_version=="2.7"
Provides-Extra: all
Requires-Dist: cld2-cffi (>=0.1.4); extra == 'all'
Requires-Dist: matplotlib (>=1.5.0); extra == 'all'
Provides-Extra: lang
Requires-Dist: cld2-cffi (>=0.1.4); extra == 'lang'
Provides-Extra: viz
Requires-Dist: matplotlib (>=1.5.0); extra == 'viz'

========================================
textacy: higher-level NLP built on spaCy
========================================

``textacy`` is a Python library for performing higher-level natural language
processing (NLP) tasks, built on the high-performance spaCy_ library. With the
fundamentals --- tokenization, part-of-speech tagging, dependency parsing, etc. ---
offloaded to another library, ``textacy`` focuses on tasks facilitated by the
ready availability of tokenized, POS-tagged, and parsed text.

.. image:: https://img.shields.io/travis/chartbeat-labs/textacy/master.svg?style=flat-square
    :target: https://travis-ci.org/chartbeat-labs/textacy
    :alt: build status

.. image:: https://img.shields.io/github/release/chartbeat-labs/textacy.svg?style=flat-square
    :target: https://github.com/chartbeat-labs/textacy/releases
    :alt: current release version

.. image:: https://img.shields.io/pypi/v/textacy.svg?style=flat-square
    :target: https://pypi.python.org/pypi/textacy
    :alt: pypi version

.. image:: https://anaconda.org/conda-forge/textacy/badges/version.svg
    :target: https://anaconda.org/conda-forge/textacy
    :alt: conda version

Features
--------

- Stream text, json, csv, and spaCy binary data to and from disk
- Clean and normalize raw text, *before* analyzing it
- Explore a variety of included datasets, with both text data and metadata from
  Congressional speeches to historical literature to Reddit comments
- Access and filter basic linguistic elements, such as words and ngrams, noun
  chunks and sentences
- Extract named entities, acronyms and their definitions, direct quotations,
  key terms, and more from documents
- Compare strings, sets, and documents by a variety of similarity metrics
- Transform documents and corpora into vectorized and semantic network representations
- Train, interpret, visualize, and save ``sklearn``-style topic models using
  LSA, LDA, or NMF methods
- Identify a text's language, display key words in context (KWIC), true-case words,
  and navigate a parse tree

... and more!


Project Links
-------------

- `textacy @ PyPi <https://pypi.python.org/pypi/textacy>`_
- `textacy @ GitHub <https://github.com/chartbeat-labs/textacy>`_
- `textacy @ ReadTheDocs <http://textacy.readthedocs.io/en/latest/>`_


Installation
------------

The simple way to install ``textacy`` is via ``pip``:

.. code-block:: console

    $ pip install textacy

or ``conda``:

.. code-block:: console

    $ conda install -c conda-forge textacy

**Note:** If you use ``pip``, some dependencies have been made optional, because
they can be difficult to install and/or are only needed in certain uses cases.
To use visualization functions, you'll need ``matplotlib`` installed; you can do
so via ``pip install textacy[viz]``. For automatic language detection, you'll
need ``cld2-cffi`` installed; do ``pip install textacy[lang]``. To install all
optional dependencies:

.. code-block:: console

    $ pip install textacy[all]

Otherwise, you can download and unzip the source ``tar.gz`` from  PyPi_,
then install manually:

.. code-block:: console

    $ python setup.py install


Downloading Data
~~~~~~~~~~~~~~~~

For most uses of ``textacy``, language-specific model data in ``spacy`` is
required. Follow the directions `here <https://spacy.io/docs/usage/models>`_
to download the necessary files. Currently available language models are listed
`here <https://spacy.io/usage/models#section-available>`_.

``textacy`` features convenient access to several datasets comprised of thousands
of text + metadata records. Data can be downloaded via the ``.download()`` method
on corresponding dataset classes, *or* directly from the command line.
For example:

.. code-block:: console

    $ python -m textacy download capitol_words

will download and save a compressed json file with ~11k speeches given by the
main protagonists of the 2016 U.S. Presidential election (that had previously
served in the U.S. Congress). For more details, do

.. code-block:: console

    $ python -m textacy --help


Usage Example
-------------

.. code-block:: pycon

    >>> import textacy
    >>> import textacy.datasets

Efficiently stream documents from disk and into a processed corpus:

.. code-block:: pycon

    >>> cw = textacy.datasets.CapitolWords()
    >>> cw.download()
    >>> records = cw.records(speaker_name={'Hillary Clinton', 'Barack Obama'})
    >>> text_stream, metadata_stream = textacy.fileio.split_record_fields(
    ...     records, 'text')
    >>> corpus = textacy.Corpus('en', texts=text_stream, metadatas=metadata_stream)
    >>> corpus
    Corpus(1241 docs; 857058 tokens)

Represent corpus as a document-term matrix, with flexible weighting and filtering:

.. code-block:: pycon

    >>> vectorizer = textacy.Vectorizer(
    ...     weighting='tfidf', normalize=True, smooth_idf=True,
    ...     min_df=2, max_df=0.95)
    >>> doc_term_matrix = vectorizer.fit_transform(
    ...     (doc.to_terms_list(ngrams=1, named_entities=True, as_strings=True)
    ...      for doc in corpus))
    >>> print(repr(doc_term_matrix))
    <1241x11708 sparse matrix of type '<class 'numpy.float64'>'
        with 215182 stored elements in Compressed Sparse Row format>

Train and interpret a topic model:

.. code-block:: pycon

    >>> model = textacy.TopicModel('nmf', n_topics=10)
    >>> model.fit(doc_term_matrix)
    >>> doc_topic_matrix = model.transform(doc_term_matrix)
    >>> doc_topic_matrix.shape
    (1241, 10)
    >>> for topic_idx, top_terms in model.top_topic_terms(vectorizer.id_to_term, top_n=10):
    ...     print('topic', topic_idx, ':', '   '.join(top_terms))
    topic 0 : new   people   's   american   senate   need   iraq   york   americans   work
    topic 1 : rescind   quorum   order   consent   unanimous   ask   president   mr.   madam   aside
    topic 2 : dispense   reading   amendment   unanimous   consent   ask   president   mr.   pending   aside
    topic 3 : health   care   child   mental   quality   patient   medical   program   information   family
    topic 4 : student   school   education   college   child   teacher   high   program   loan   year
    topic 5 : senators   desiring   chamber   vote   4,600   amtrak   rail   airline   litigation   expedited
    topic 6 : senate   thursday   wednesday   session   unanimous   consent   authorize   p.m.   committee   ask
    topic 7 : medicare   drug   senior   medicaid   prescription   benefit   plan   cut   cost   fda
    topic 8 : flu   vaccine   avian   pandemic   roberts   influenza   seasonal   outbreak   health   cdc
    topic 9 : virginia   west virginia   west   senator   yield   question   thank   objection   inquiry   massachusetts

Basic indexing as well as flexible selection of documents in a corpus:

.. code-block:: pycon

    >>> obama_docs = list(corpus.get(
    ...     lambda doc: doc.metadata['speaker_name'] == 'Barack Obama'))
    >>> len(obama_docs)
    411
    >>> doc = corpus[-1]
    >>> doc
    Doc(2999 tokens; "In the Federalist Papers, we often hear the ref...")

Preprocess plain text, or highlight particular terms in it:

.. code-block:: pycon

    >>> textacy.preprocess_text(doc.text, lowercase=True, no_punct=True)[:70]
    'in the federalist papers we often hear the reference to the senates ro'
    >>> textacy.text_utils.keyword_in_context(doc.text, 'America', window_width=35)
    g on this tiny piece of Senate and  America n history. Some 10 years ago, I ask
    o do the hard work in New York and  America , who get up every day and do the v
    say: You know, you never can count  America  out. Whenever the chips are down,
     what we know will give our fellow  America ns a better shot at the kind of fut
    aith in this body and in my fellow  America ns. I remain an optimist, that Amer
    ricans. I remain an optimist, that  America 's best days are still ahead of us.

Extract various elements of interest from parsed documents:

.. code-block:: pycon

    >>> list(textacy.extract.ngrams(
    ...     doc, 2, filter_stops=True, filter_punct=True, filter_nums=False))[:15]
    [Federalist Papers,
     Senate's,
     's role,
     violent passions,
     pernicious resolutions,
     everlasting credit,
     common ground,
     8 years,
     tiny piece,
     American history,
     10 years,
     years ago,
     New York,
     fellow New,
     New Yorkers]
    >>> list(textacy.extract.ngrams(
    ...     doc, 3, filter_stops=True, filter_punct=True, min_freq=2))
    [fellow New Yorkers,
     World Trade Center,
     Senator from New,
     World Trade Center,
     Senator from New,
     lot of fun,
     fellow New Yorkers,
     lot of fun]
    >>> list(textacy.extract.named_entities(
    ...     doc, drop_determiners=True, exclude_types='numeric'))[:10]
    [Senate,
     Senate,
     American,
     New York,
     New Yorkers,
     Senate,
     Barbara Mikulski,
     Senate,
     Pennsylvania Avenue,
     Senate]
    >>> pattern = textacy.constants.POS_REGEX_PATTERNS['en']['NP']
    >>> pattern
    <DET>? <NUM>* (<ADJ> <PUNCT>? <CONJ>?)* (<NOUN>|<PROPN> <PART>?)+
    >>> list(textacy.extract.pos_regex_matches(doc, pattern))[:10]
    [the Federalist Papers,
     the reference,
     the Senate's role,
     the consequences,
     sudden and violent passions,
     intemperate and pernicious resolutions,
     the everlasting credit,
     wisdom,
     our Founders,
     an effort]
    >>> list(textacy.extract.semistructured_statements(doc, 'I', cue='be'))
    [(I, was, on the other end of Pennsylvania Avenue),
     (I, was, , a very new Senator, and my city and my State had been devastated),
     (I, am, grateful to have had Senator Schumer as my partner and my ally),
     (I, am, very excited about what can happen in the next 4 years),
     (I, been, a New Yorker, but I know I always will be one)]
    >>> textacy.keyterms.textrank(doc, n_keyterms=10)
    [('day', 0.01608508275877894),
     ('people', 0.015079868730811194),
     ('year', 0.012330783590843065),
     ('way', 0.011732786337383587),
     ('colleague', 0.010794482493897155),
     ('new', 0.0104941198408241),
     ('time', 0.010016582029543003),
     ('work', 0.0096498231660789),
     ('lot', 0.008960478625039818),
     ('great', 0.008552318032915361)]

Compute basic counts and readability statistics for a given text:

.. code-block:: pycon

    >>> ts = textacy.TextStats(doc)
    >>> ts.n_unique_words
    1107
    >>> ts.basic_counts
    {'n_chars': 11498,
     'n_long_words': 512,
     'n_monosyllable_words': 1785,
     'n_polysyllable_words': 222,
     'n_sents': 99,
     'n_syllables': 3525,
     'n_unique_words': 1107,
     'n_words': 2516}
    >>> ts.flesch_kincaid_grade_level
    10.853709110179697
    >>> ts.readability_stats
    {'automated_readability_index': 12.801546064781363,
     'coleman_liau_index': 9.905629258346586,
     'flesch_kincaid_grade_level': 10.853709110179697,
     'flesch_readability_ease': 62.51222198133965,
     'gulpease_index': 55.10492845786963,
     'gunning_fog_index': 13.69506833036245,
     'lix': 45.76390294037353,
     'smog_index': 11.683781121521076,
     'wiener_sachtextformel': 5.401029023140788}

Count terms individually, and represent documents as a bag-of-terms with flexible
weighting and inclusion criteria:

.. code-block:: pycon

    >>> doc.count('America')
    3
    >>> bot = doc.to_bag_of_terms(ngrams={2, 3}, as_strings=True)
    >>> sorted(bot.items(), key=lambda x: x[1], reverse=True)[:10]
    [('new york', 18),
     ('senate', 8),
     ('first', 6),
     ('state', 4),
     ('9/11', 3),
     ('look forward', 3),
     ('america', 3),
     ('new yorkers', 3),
     ('chuck', 3),
     ('lot of fun', 2)]

**Note:** In almost all cases, ``textacy`` expects to be working with unicode text.
Docstrings indicate this as ``str``, which is clear and correct for Python 3 but
not Python 2. In the latter case, users should cast ``str`` bytes to ``unicode``,
as needed.


Maintainer
----------

- Burton DeWilde (<burton@chartbeat.net>)


.. _spaCy: https://spacy.io/
.. _PyPi: https://pypi.python.org/pypi/textacy


