Cheshire3 Objects: DocumentFactory

|  | ## Cheshire3 Objects: DocumentFactory

## Description

DocumentFactories are the main means by which Documents are ingested into the system. Once the 'load' argument has been called, a DocumentFactory should be able to return, on request, one or more Documents. The way in which it does this will depend on the way in which it has been configured, and how 'load' was called. For example it may locate all documents, and cache them internally (e.g. for multiple XML documents within a single file), or it may crawl, locating and returning the documents one at a time (e.g. for many large files in a directory structure.)

## Implementations

The following implementations are pre-configured and ready to use.
They may be used out-of-the-box in configurations for Workflows, or in code by getting the object from a Server.

- defaultDocumentFactory

  A 'Smart' DocumentFactory that will attempt to do the most sensible thing to generate documents, based on the arguments passed to its 'load' method.

- defaultAccumulatingDocumentFactory

  A DocumentFactory that accumulates data across multiple 'load' calls to produce 1 or more Documents.
  The main difference between this and the defaultDocumentFactory is that 'load' can be called repeatedly without losing Documents already discovered.

The DocumentFactory will try to guess the format based on the data argument passed to it, however if you know the format, you can tell the documentFactory by using the format keyword argument. e.g.

`documentFacory.load(session, "/home/user/data/", format="dir")`
`documentFacory.load(session, "/home/user/data.zip", format="zip")`

A DocumentFactory will use an appropriate DocumentStream to deal with each format. Part of the 'smart'ness of DocumentFactories is that the DocumentStreams can be recursively called. e.g. You could call 'load' on a directory which contained a number of zip files, each of which were made up of a number of XML files. The DocumentFactory would use a DirectoryDocumentStream, a ZipDocumentStream, a FileDocumentStream and a XmlDocumentStream in turn to find and return XML Documents.

At the present time, the following formats are supported by defaultDocumentFactory and defaultAccumulatingDocumentFactory.
Note Well: DocumentStreams are only intended for use by DocumentFactories, and are unlikely to behave correctly if called directly by users' scripts.

Short name | DocumentStream used | Description
--- | --- | ---
xml | XmlDocumentStream | Given data, finds XML instances within it and treats each as a Document. By default the documentFactory will use the first tag that it encounters as the basis of all future Documents, but if you know the name of the tag to use, you can supply this with the tagName keyword argument. e.g. `documentFactory.load(session, "/home/user/myFile.xml", format="xml", tagName="myTag")`
marc | MarcDocumentStream | Given data containing MARC records, treats each MARC record as a Document (see also docs for MarcParser and MarcRecord.)
dir | DirectoryDocumentStream | Given a directory name, walks though all files and sub-directories within it looking for Documents.
tar | TarDocumentStream | Given the data which makes up a tar file, extract the files from it as Documents.
zip | ZipDocumentStream | Given the data which makes up a zip file, extract the files from it as Documents.
cluster | ClusterDocumentStream | Given the path to a raw cluster data file (as created by a ClusterExtractionDocumentFactory), merge and create documents.
locate | LocateDocumentStream | Given a name or pattern, locates files whose names match.
component | ComponentDocumentStream | Given a Record, finds component Documents using a configured Selector.
termHash | TermHashDocumentStream | Given data consisting of a hash of terms, treat each term as a Document
file | FileDocumentStream | Given the path to a file, open it, and read the contents.

## API

Module: cheshire3.documentFactory
Classes:

- SimpleDocumentFactory
  A 'Smart' DocumentFactory that will attempt to do the most sensible thing to generate documents, based on the arguments passed to its 'load' method. Additional DocumentStreams can be registered with the SimpleDocumentFactory class by calling the class method 'register_stream' (see the methods table for details). Such DocumentStreams could be user-defined, as long as they conform to the DocumentStream API.
- ComponentDocumentFactory
  A ComponentDocumentFactory is configured ahead of time with an XPath. When its 'load' method is called on a Record, the record is processed, and elements matching the configured XPath are extracted and treated as new documents in their own right.
- AccumulatingDocumentFactory
  A DocumentFactory that accumulates data across multiple 'load' calls to produce 1 or more documents.
- XmlDocumentStream
  Given data, finds XML instances within it and treats each as a Document. By default the documentFactory will use the first tag that it encounters as the basis of all future Documents, but if you know the name of the tag to use, you can supply this with the tagName keyword argument. e.g.
  `documentFactory.load(session, "/home/user/myFile.xml", format="xml", tagName="myTag")`
- MarcDocumentStream
  Given data containing MARC records, treats each MARC record as a Document (see also docs for MarcParser and MarcRecord.)
- DirectoryDocumentStream
  Given a directory name, walks though all files and sub-directories within it looking for Documents.
- TarDocumentStream
  Given the data which makes up a tar file, extract the files from it as Documents.
- ZipDocumentStream
  Given the data which makes up a zip file, extract the files from it as Documents.
- ClusterDocumentStream
  Given the path to a raw cluster data file (as created by a ClusterExtractionDocumentFactory), merge and create documents.
- LocateDocumentStream
  Given a name or pattern, locates files whose names match.
- ComponentDocumentStream
  Given a Record, finds component Documents using a configured Selector.
- TermHashDocumentStream
  Given data consisting of a hash of terms, treat each term as a Document
- FileDocumentStream
  Given the path to a file, open it, and read the contents.

DocumentFactory Methods:

Function | Parameters | Returns | Description
--- | --- | --- | ---
__init__ | session, config, parent |  | The constructor takes the config node for the object, and its parent (usually a database).
load | session, ?data, ?cache, ?format, ?tagName, ?codec |  | Load the data provide (or use the configured default if not provided). The way the data is loaded is dependent on the other parameters (or their configured defaults if absent): cache - should documents be cached in memory: 0 = No, 1 = Locations cached but not documents, 2 = Yes format - specifies how to treat the data parameter tagName - The XML tag to treat as the document root codec - specifies the codec to use to read the documents in
get_document | session, ?index | Document | Return the index'th document in the factory if index is provided, otherwise return the next document.
register_stream | session, format, class |  | Class method to register the supplied class of DocumentStream with the document factory for the given format. This class will be used the next time 'load' is called with this format.

DocumentStream Methods:

Function | Parameters | Returns | Description
--- | --- | --- | ---
__init__ | session, stream, format, ?tagName, ?codec, ?factory |  | The constructor takes the location of the data stream, the format. Optional arguments are the tagName to look for, the codec to use to read in data and the DocumentFactory that initialized the stream.
open_stream | streamLocation | data stream | Perform any operations needed before the data stream can be read (e.g. open files).
fetch_document | index | data/Document | Return the index'th piece of data or Document.
find_documents | session, cache |  | Find documents within the data stream.

### Sub-Packages

Sub-Package: web
Module: cheshire3.web.documentFactory
Classes:

- HttpDocumentStream
  Given an URL, find and download Documents.
- FtpDocumentStream
  Given a FTP URL, find and get (download) Documents.
- Z3950DocumentStream
  Given a Z3950 request, as a URL, perform the search and then map the results to Documents.
- SruDocumentStream
  Given a SRU request, as a URL, perform the request and then map the results to Documents.
- SrwDocumentStream
  Same as SruDocumentStream, but uses SOAP over HTTP POST to carry out the request rather than HTTP GET.
- OAIHarvestDocumentStream
  Given a an OAI-PMH request, as a URL, perform the request to harvest the records from the server and map them to Documents.
- OpensearchDocumentStream
  Given a tuple of OSD location and query, perform the search and then map the results to Documents.
- SyndicationDocumentStream
  Given a URL, filename or buffer, use universal feed parser to import rss, atom, etc. and map feeds to Documents.
- AccHttpDocumentStream
  AccumulatingDocumentFactory friendly version of HttpDocumentStream. Accumulate Documents from a number of URLs (e.g. when web-crawling).
- GoogleSearchDocumentStream
  Given a Google query, use the Google SOAP interface to search, fetch the actual files matched and map them to documents.
  OBSELETE - Google have withdrawn their SOAP interface.

Sub-Package: vdb
Module: cheshire3.vdb.documentFactory
Classes:

- VdbDocumentFactory
  Given a ZeeRex Record, create a summary Document for the remote service (e.g. XML representing the content of all indexes for a SRU server.)
- VdbDocumentStream
  Abstract Base Class for creating summary documents from remote services.
- SruVdbDocumentStream
  Harvest index terms from an SRU server.
- SrwVdbDocumentStream
  Harvest index terms from an SRW server. As SruVdbDocumentStream, but uses SOAP via HTTP POST.