translate.search.indexing.CommonIndexer.CommonDatabase

Package translate :: Package search :: Package indexing :: Module CommonIndexer :: Class CommonDatabase |
[hide private]
[frames] | no frames]

# Class CommonDatabase

source code

base class for indexing support

any real implementation must override most methods of this class

Instance Methods | [hide private]
__init__(self, basedir, analyzer=None, create_allowed=True) initialize or open an indexing database | source code
flush(self, optimize=False) flush the content of the database - to force changes to be written to disk | source code
query type of the specific implemention |
make_query(self, args, require_all=True, analyzer=None) create simple queries (strings or field searches) or combine multiple queries (AND/OR) | source code
xapian.Query | PyLucene.Query |
_create_query_for_query(self, query) generate a query based on an existing query object | source code
xapian.Query | PyLucene.Query |
_create_query_for_string(self, text, require_all=True, analyzer=None) generate a query for a plain term of a string query | source code
xapian.Query | PyLucene.Query |
_create_query_for_field(self, field, value, analyzer=None) generate a field query | source code
xapian.Query | PyLucene.Query |
_create_query_combined(self, queries, require_all=True) generate a combined query | source code
index_document(self, data) add the given data to the database | source code
xapian.Document | PyLucene.Document |
_create_empty_document(self) create an empty document to be filled and added to the index later | source code
_add_plain_term(self, document, term, tokenize=True) add a term to a document | source code
_add_field_term(self, document, field, term, tokenize=True) add a field term to a document | source code
_add_document_to_index(self, document) add a prepared document to the index database | source code
begin_transaction(self) begin a transaction | source code
cancel_transaction(self) cancel an ongoing transaction | source code
commit_transaction(self) submit the currently ongoing transaction and write changes to disk | source code
subclass of CommonEnquire |
get_query_result(self, query) return an object containing the results of a query | source code
delete_document_by_id(self, docid) delete a specified document | source code
list of dicts |
search(self, query, fieldnames) return a list of the contents of specified fields for all matches of a query | source code
delete_doc(self, ident) delete the documents returned by a query | source code
_walk_matches(self, query, function, arg_for_function=None) use this function if you want to do something with every single match of a query | source code
set_field_analyzers(self, field_analyzers) set the analyzers for different fields of the database documents | source code
int | dict |
get_field_analyzers(self, fieldnames=None) return the analyzer that was mapped to a specific field | source code
_decode(self, text) decode the string from utf-8 or charmap perform unicde normalization | source code
Inherited from `object`: `__delattr__`, `__getattribute__`, `__hash__`, `__new__`, `__reduce__`, `__reduce_ex__`, `__repr__`, `__setattr__`, `__str__`

Class Variables | [hide private]
| field_analyzers = ``{``}`` mapping of field names and analyzers - see 'set_field_analyzers'
| ANALYZER_EXACT = `0` exact matching: the query string must equal the whole term string
| ANALYZER_PARTIAL = `2` partial matching: a document matches, even if the query string only matches the beginning of the term value.
| ANALYZER_TOKENIZE = `4` tokenize terms and queries automatically
| ANALYZER_DEFAULT = `6` the default analyzer to be used if nothing is configured
| QUERY_TYPE = `None` override this with the query class of the implementation
| INDEX_DIRECTORY_NAME = `None` override this with a string to be used as the name of the indexing directory/file in the filesystem

Properties | [hide private]
Inherited from `object`: `__class__`

Method Details | [hide private]

### __init__(self, basedir, analyzer=None, create_allowed=True) (Constructor) | source code
initialize or open an indexing database Any derived class must override __init__. Any implementation can rely on the "self.location" attribute to be set by the __init__ function of the super class. Parameters: `basedir` (str) - the parent directory of the database `analyzer` (int) - bitwise combination of possible analyzer flags to be used as the default analyzer for this database. Leave it empty to use the system default analyzer (self.ANALYZER_DEFAULT). see self.ANALYZER_TOKENIZE, self.ANALYZER_PARTIAL, ... `create_allowed` (bool) - create the database, if necessary; default: True Raises: `ValueError` - the given location exists, but the database type is incompatible (e.g. created by a different indexing engine) `OSError` - the database failed to initialize Overrides: object.__init__

### flush(self, optimize=False) | source code
flush the content of the database - to force changes to be written to disk some databases also support index optimization Parameters: `optimize` (bool) - should the index be optimized if possible?

### make_query(self, args, require_all=True, analyzer=None) | source code
  - create simple queries (strings or field searches) or combine multiple queries (AND/OR) To specifiy rules for field searches, you may want to take a look at 'set_field_analyzers'. The parameter 'match_text_partial' can override the previously defined default setting. Parameters: `args` (list of queries | single query | str | dict) - queries or search string or description of field query examples:
           [xapian.Query("foo"), xapian.Query("bar")]
           xapian.Query("foo")
           "bar"
           {"foo": "bar", "foobar": "foo"}
`require_all` (boolean) - boolean operator (True -> AND (default) / False -> OR) `analyzer` (int) - (only applicable for 'dict' or 'str') Define query options (partial matching, exact matching, tokenizing, ...) as bitwise combinations of CommonIndexer.ANALYZER_???. This can override previously defined field analyzer settings. If analyzer is None (default), then the configured analyzer for the field is used. Returns: query type of the specific implemention the combined query

### _create_query_for_query(self, query) | source code
generate a query based on an existing query object basically this function should just create a copy of the original Parameters: `query` (xapian.Query) - the original query object Returns: xapian.Query | PyLucene.Query the resulting query object

### _create_query_for_string(self, text, require_all=True, analyzer=None) | source code
generate a query for a plain term of a string query basically this function parses the string and returns the resulting query Parameters: `text` (str) - the query string `require_all` (bool) - boolean operator (True -> AND (default) / False -> OR) `analyzer` (int) - Define query options (partial matching, exact matching, tokenizing, ...) as bitwise combinations of CommonIndexer.ANALYZER_???. This can override previously defined field analyzer settings. If analyzer is None (default), then the configured analyzer for the field is used. Returns: xapian.Query | PyLucene.Query resulting query object

### _create_query_for_field(self, field, value, analyzer=None) | source code
generate a field query this functions creates a field->value query Parameters: `field` (str) - the fieldname to be used `value` (str) - the wanted value of the field `analyzer` (int) - Define query options (partial matching, exact matching, tokenizing, ...) as bitwise combinations of CommonIndexer.ANALYZER_???. This can override previously defined field analyzer settings. If analyzer is None (default), then the configured analyzer for the field is used. Returns: xapian.Query | PyLucene.Query resulting query object

### _create_query_combined(self, queries, require_all=True) | source code
generate a combined query Parameters: `queries` (list of xapian.Query) - list of the original queries `require_all` (bool) - boolean operator (True -> AND (default) / False -> OR) Returns: xapian.Query | PyLucene.Query the resulting combined query object

### index_document(self, data) | source code
add the given data to the database Parameters: `data` (dict | list of str) - the data to be indexed. A dictionary will be treated as fieldname:value combinations. If the fieldname is None then the value will be interpreted as a plain term or as a list of plain terms. Lists of terms are indexed separately. Lists of strings are treated as plain terms.

### _create_empty_document(self) | source code
create an empty document to be filled and added to the index later Returns: xapian.Document | PyLucene.Document the new document object

### _add_plain_term(self, document, term, tokenize=True) | source code
add a term to a document Parameters: `document` (xapian.Document | PyLucene.Document) - the document to be changed `term` (str) - a single term to be added `tokenize` (bool) - should the term be tokenized automatically

### _add_field_term(self, document, field, term, tokenize=True) | source code
add a field term to a document Parameters: `document` (xapian.Document | PyLucene.Document) - the document to be changed `field` (str) - name of the field `term` (str) - term to be associated to the field `tokenize` (bool) - should the term be tokenized automatically

### _add_document_to_index(self, document) | source code
add a prepared document to the index database Parameters: `document` (xapian.Document | PyLucene.Document) - the document to be added

### begin_transaction(self) | source code
begin a transaction You can group multiple modifications of a database as a transaction. This prevents time-consuming database flushing and helps, if you want that a changeset is committed either completely or not at all. No changes will be written to disk until 'commit_transaction'. 'cancel_transaction' can be used to revert an ongoing transaction. Database types that do not support transactions may silently ignore it.

### cancel_transaction(self) | source code
cancel an ongoing transaction See 'start_transaction' for details.

### commit_transaction(self) | source code
submit the currently ongoing transaction and write changes to disk See 'start_transaction' for details.

### get_query_result(self, query) | source code
return an object containing the results of a query Parameters: `query` (a query object of the real implementation) - a pre-compiled query Returns: subclass of CommonEnquire an object that allows access to the results

### delete_document_by_id(self, docid) | source code
delete a specified document Parameters: `docid` (int) - the document ID to be deleted

### search(self, query, fieldnames) | source code
return a list of the contents of specified fields for all matches of a query Parameters: `query` (a query object of the real implementation) - the query to be issued `fieldnames` (string | list of strings) - the name(s) of a field of the document content Returns: list of dicts a list of dicts containing the specified field(s)

### delete_doc(self, ident) | source code
delete the documents returned by a query Parameters: `ident` (int | list of tuples | dict | list of dicts | query (e.g. xapian.Query) | list of queries) - [list of] document IDs | dict describing a query | query

### _walk_matches(self, query, function, arg_for_function=None) | source code
use this function if you want to do something with every single match of a query example:
        self._walk_matches(query, function_for_match, arg_for_func)
       'function_for_match' expects only one argument: the matched object
Parameters: `query` (xapian.Query | PyLucene.Query) - a query object of the real implementation `function` (function) - the function to execute with every match `arg_for_function` (anything) - an optional argument for the function

### set_field_analyzers(self, field_analyzers) | source code
set the analyzers for different fields of the database documents All bitwise combinations of CommonIndexer.ANALYZER_??? are possible. Parameters: `field_analyzers` (dict containing field names and analyzers) - mapping of field names and analyzers Raises: `TypeError` - invalid values in 'field_analyzers'

### get_field_analyzers(self, fieldnames=None) | source code
return the analyzer that was mapped to a specific field see 'set_field_analyzers' for details Parameters: `fieldnames` (str | list of str | None) - the analyzer of this field (or all/multiple fields) is requested; leave empty (or "None") to request all fields Returns: int | dict the analyzer setting of the field - see CommonDatabase.ANALYZER_??? or a dict of field names and analyzers