gensim: models.lsimodel – Latent Semantic Indexing
Description: Efficient topic modelling of text semantics in Python.

#

models.lsimodel – Latent Semantic Indexing

# `models.lsimodel` – Latent Semantic Indexing

Module for Latent Semantic Indexing.

This module actually contains several algorithms for decomposition of large corpora, a combination of which effectively and transparently allows building LSI models for:

- corpora much larger than RAM: only constant memory is needed, independent of the corpus size (though still dependent on the feature set size)
- corpora that are streamed: documents are only accessed sequentially, no random-access
- corpora that cannot be even temporarily stored: each document can only be seen once and must be processed immediately (one-pass algorithm)
- distributed computing for very large corpora, making use of a cluster of machines

Wall-clock performance on the English Wikipedia (2G corpus positions, 3.2M documents, 100K features, 0.5G non-zero entries in the final TF-IDF matrix), requesting the top 400 LSI factors:

algorithm | serial | distributed
--- | --- | ---
one-pass merge algorithm | 5h14m | 1h41m
multi-pass stochastic algo (with 2 power iterations) | 5h39m | N/A [1]

serial = Core 2 Duo MacBook Pro 2.53Ghz, 4GB RAM, libVec

distributed = cluster of four logical nodes on three physical machines, each with dual core Xeon 2.0GHz, 4GB RAM, ATLAS

[1] | The stochastic algo could be distributed too, but most time is already spent reading/decompressing the input from disk in its 4 passes. The extra network traffic due to data distribution across cluster nodes would likely make it slower.

class `gensim.models.lsimodel.` `LsiModel`(corpus=None, num_topics=200, id2word=None, chunksize=20000, decay=1.0, distributed=False, onepass=True, power_iters=2, extra_samples=100)

  Objects of this class allow building and maintaining a model for Latent Semantic Indexing (also known as Latent Semantic Analysis).

  The main methods are:

  1. constructor, which initializes the projection into latent topics space,
  2. the `[]` method, which returns representation of any input document in the latent space,
  3. add_documents() for incrementally updating the model with new documents.

  The left singular vectors are stored in lsi.projection.u, singular values in lsi.projection.s. Right singular vectors can be reconstructed from the output of lsi[training_corpus], if needed.

  Model persistency is achieved via its load/save methods.

  num_topics is the number of requested factors (latent dimensions).

  After the model has been trained, you can estimate topics for an arbitrary, unseen document, using the `topics = self[document]` dictionary notation. You can also add new training documents, with `self.add_documents`, so that training can be stopped and resumed at any time, and the LSI transformation is available at any point.

  If you specify a corpus, it will be used to train the model. See the method add_documents for a description of the chunksize and decay parameters.

  Turn onepass off to force a multi-pass stochastic algorithm.

  power_iters and extra_samples affect the accuracy of the stochastic multi-pass algorithm, which is used either internally (onepass=True) or as the front-end algorithm (onepass=False). Increasing the number of power iterations improves accuracy, but lowers performance. See [2] for some hard numbers.

  Turn on distributed to enable distributed computing.

  Example:

      >>> lsi = LsiModel(corpus, num_topics=10)
      >>> print lsi[doc_tfidf] # project some document into LSI space
      >>> lsi.add_documents(corpus2) # update LSI on additional documents
      >>> print lsi[doc_tfidf]

  [2] | http://nlp.fi.muni.cz/~xrehurek/nips/rehurek_nips.pdf

  `add_documents`(corpus, chunksize=None, decay=None)

    Update singular value decomposition to take into account a new corpus of documents.

    Training proceeds in chunks of chunksize documents at a time. The size of chunksize is a tradeoff between increased speed (bigger chunksize) vs. lower memory footprint (smaller chunksize). If the distributed mode is on, each chunk is sent to a different worker/computer.

    Setting decay < 1.0 causes re-orientation towards new data trends in the input document stream, by giving less emphasis to old observations. This allows LSA to gradually “forget” old observations (documents) and give more preference to new ones.

  classmethod `load`(fname)

    Load a previously saved object from file (also see save).

  `print_debug`(num_topics=5, num_words=10)

    Print (to log) the most salient words of the first num_topics topics.

    Unlike print_topics(), this looks for words that are significant for a particular topic and not for others. This should result in a more human-interpretable description of topics.

  `print_topics`(num_topics=5, num_words=10)

    Alias for show_topics() which prints the top 5 topics to log.

  `save`(fname)

    Override the default save (which uses cPickle), because that’s too inefficient and cPickle has bugs. Instead, single out the large transformation matrix and store that separately in binary format (that can be directly mmap’ed back in load()), under fname.npy.

  `show_topic`(topicno, topn=10)

    Return a specified topic (=left singular vector), 0 <= topicno < self.num_topics, as string.

    Return only the topn words which contribute the most to the direction of the topic (both negative and positive).

        >>> lsimodel.print_topic(10, topn=5)
        '-0.340 * "category" + 0.298 * "$M$" + 0.183 * "algebra" + -0.174 * "functor" + -0.168 * "operator"'

  `show_topics`(num_topics=-1, num_words=10, log=False, formatted=True)

    Show num_topics most significant topics (show all by default). For each topic, show num_words most significant words (10 words by defaults).

    Return the shown topics as a list – a list of strings if formatted is True, or a list of (value, word) 2-tuples if it’s False.

    If log is True, also output this result to log.

`gensim.models.lsimodel.` `clip_spectrum`(s, k, discard=0.001)

  Given eigenvalues s, return how many factors should be kept to avoid storing spurious (tiny, numerically instable) values.

  This will ignore the tail of the spectrum with relative combined mass < min(discard, 1/k).

  The returned value is clipped against k (= never return more than k).

`gensim.models.lsimodel.` `stochastic_svd`(corpus, rank, num_terms, chunksize=20000, extra_dims=None, power_iters=0, dtype=<type 'numpy.float64'>, eps=1e-06)

  Return (U, S): the left singular vectors and the singular values of the streamed input corpus corpus [3].

  This may actually return less than the requested number of top rank factors, in case the input is of lower rank. The extra_dims (oversampling) and especially power_iters (power iterations) parameters affect accuracy of the decomposition.

  This algorithm uses 2+power_iters passes over the data. In case you can only afford a single pass over the input corpus, set onepass=True in `LsiModel` and avoid using this algorithm directly.

  The decomposition algorithm is based on Halko, Martinsson, Tropp. Finding structure with randomness, 2009.

  [3] | If corpus is a scipy.sparse matrix instead, it is assumed the whole corpus fits into core memory and a different (more efficient) code path is chosen.