misc — Translate Toolkit 2.3.0 documentation

# misc

Miscellaneous modules for translate - including modules for backward compatibility with pre-2.3 versions of Python

## dictutils

Implements a case-insensitive (on keys) dictionary and order-sensitive dictionary

class `translate.misc.dictutils.` `cidict`(fromdict=None)
  `clear`() → None. Remove all items from D.
  `copy`() → a shallow copy of D
  `fromkeys`(S[, v]) → New dict with keys from S and values equal to v.

    v defaults to None.

  `get`(k[, d]) → D[k] if k in D, else d. d defaults to None.
  `has_key`(k) → True if D has a key k, else False
  `items`() → list of D's (key, value) pairs, as 2-tuples
  `iteritems`() → an iterator over the (key, value) items of D
  `iterkeys`() → an iterator over the keys of D
  `itervalues`() → an iterator over the values of D
  `keys`() → list of D's keys
  `pop`(k[, d]) → v, remove specified key and return the corresponding value.

    If key is not found, d is returned if given, otherwise KeyError is raised

  `popitem`() → (k, v), remove and return some (key, value) pair as a

    2-tuple; but raise KeyError if D is empty.

  `setdefault`(k[, d]) → D.get(k,d), also set D[k]=d if k not in D
  `update`(E) → None.

    Update D from E: for k in E.keys(): D[k] = E[k]

  `values`() → list of D's values
  `viewitems`() → a set-like object providing a view on D's items
  `viewkeys`() → a set-like object providing a view on D's keys
  `viewvalues`() → an object providing a view on D's values

## file_discovery

`translate.misc.file_discovery.` `get_abs_data_filename`(path_parts, basedirs=None)

  Get the absolute path to the given file- or directory name in the current running application’s data directory.

  Parameters: | path_parts (list) – The path parts that can be joined by `os.path.join()`.

## lru

class `translate.misc.lru.` `LRUCachingDict`(maxsize, cullsize=2, peakmult=10, aggressive_gc=True, *args, **kwargs)

  Caching dictionary like object that discards the least recently used objects when number of cached items exceeds maxsize.

  cullsize is the fraction of items that will be discarded when maxsize is reached.

  `cull`()

    free memory by deleting old items from cache

  `itervaluerefs`()

    Return an iterator that yields the weak references to the values.

    The references are not guaranteed to be ‘live’ at the time they are used, so the result of calling the references needs to be checked before being used. This can be used to avoid creating references that will cause the garbage collector to keep the values around longer than needed.

  `valuerefs`()

    Return a list of weak references to the values.

    The references are not guaranteed to be ‘live’ at the time they are used, so the result of calling the references needs to be checked before being used. This can be used to avoid creating references that will cause the garbage collector to keep the values around longer than needed.

## multistring

Supports a hybrid Unicode string that can also have a list of alternate strings in the strings attribute

class `translate.misc.multistring.` `multistring`(*args, **kwargs)
  `capitalize`() → unicode

    Return a capitalized version of S, i.e. make the first character have upper case and the rest lower case.

  `center`(width[, fillchar]) → unicode

    Return S centered in a Unicode string of length width. Padding is done using the specified fill character (default is a space)

  `count`(sub[, start[, end]]) → int

    Return the number of non-overlapping occurrences of substring sub in Unicode string S[start:end]. Optional arguments start and end are interpreted as in slice notation.

  `decode`([encoding[, errors]]) → string or unicode

    Decodes S using the codec registered for encoding. encoding defaults to the default encoding. errors may be given to set a different error handling scheme. Default is ‘strict’ meaning that encoding errors raise a UnicodeDecodeError. Other possible values are ‘ignore’ and ‘replace’ as well as any other name registered with codecs.register_error that is able to handle UnicodeDecodeErrors.

  `encode`([encoding[, errors]]) → string or unicode

    Encodes S using the codec registered for encoding. encoding defaults to the default encoding. errors may be given to set a different error handling scheme. Default is ‘strict’ meaning that encoding errors raise a UnicodeEncodeError. Other possible values are ‘ignore’, ‘replace’ and ‘xmlcharrefreplace’ as well as any other name registered with codecs.register_error that can handle UnicodeEncodeErrors.

  `endswith`(suffix[, start[, end]]) → bool

    Return True if S ends with the specified suffix, False otherwise. With optional start, test S beginning at that position. With optional end, stop comparing S at that position. suffix can also be a tuple of strings to try.

  `expandtabs`([tabsize]) → unicode

    Return a copy of S where all tab characters are expanded using spaces. If tabsize is not given, a tab size of 8 characters is assumed.

  `find`(sub[, start[, end]]) → int

    Return the lowest index in S where substring sub is found, such that sub is contained within S[start:end]. Optional arguments start and end are interpreted as in slice notation.

    Return -1 on failure.

  `format`(*args, **kwargs) → unicode

    Return a formatted version of S, using substitutions from args and kwargs. The substitutions are identified by braces (‘{‘ and ‘}’).

  `index`(sub[, start[, end]]) → int

    Like S.find() but raise ValueError when the substring is not found.

  `isalnum`() → bool

    Return True if all characters in S are alphanumeric and there is at least one character in S, False otherwise.

  `isalpha`() → bool

    Return True if all characters in S are alphabetic and there is at least one character in S, False otherwise.

  `isdecimal`() → bool

    Return True if there are only decimal characters in S, False otherwise.

  `isdigit`() → bool

    Return True if all characters in S are digits and there is at least one character in S, False otherwise.

  `islower`() → bool

    Return True if all cased characters in S are lowercase and there is at least one cased character in S, False otherwise.

  `isnumeric`() → bool

    Return True if there are only numeric characters in S, False otherwise.

  `isspace`() → bool

    Return True if all characters in S are whitespace and there is at least one character in S, False otherwise.

  `istitle`() → bool

    Return True if S is a titlecased string and there is at least one character in S, i.e. upper- and titlecase characters may only follow uncased characters and lowercase characters only cased ones. Return False otherwise.

  `isupper`() → bool

    Return True if all cased characters in S are uppercase and there is at least one cased character in S, False otherwise.

  `join`(iterable) → unicode

    Return a string which is the concatenation of the strings in the iterable. The separator between elements is S.

  `ljust`(width[, fillchar]) → int

    Return S left-justified in a Unicode string of length width. Padding is done using the specified fill character (default is a space).

  `lower`() → unicode

    Return a copy of the string S converted to lowercase.

  `lstrip`([chars]) → unicode

    Return a copy of the string S with leading whitespace removed. If chars is given and not None, remove characters in chars instead. If chars is a str, it will be converted to unicode before stripping

  `partition`(sep) -> (head, sep, tail)

    Search for the separator sep in S, and return the part before it, the separator itself, and the part after it. If the separator is not found, return S and two empty strings.

  `replace`(old, new[, count]) → unicode

    Return a copy of S with all occurrences of substring old replaced by new. If the optional argument count is given, only the first count occurrences are replaced.

  `rfind`(sub[, start[, end]]) → int

    Return the highest index in S where substring sub is found, such that sub is contained within S[start:end]. Optional arguments start and end are interpreted as in slice notation.

    Return -1 on failure.

  `rindex`(sub[, start[, end]]) → int

    Like S.rfind() but raise ValueError when the substring is not found.

  `rjust`(width[, fillchar]) → unicode

    Return S right-justified in a Unicode string of length width. Padding is done using the specified fill character (default is a space).

  `rpartition`(sep) -> (head, sep, tail)

    Search for the separator sep in S, starting at the end of S, and return the part before it, the separator itself, and the part after it. If the separator is not found, return two empty strings and S.

  `rsplit`([sep[, maxsplit]]) → list of strings

    Return a list of the words in S, using sep as the delimiter string, starting at the end of the string and working to the front. If maxsplit is given, at most maxsplit splits are done. If sep is not specified, any whitespace string is a separator.

  `rstrip`([chars]) → unicode

    Return a copy of the string S with trailing whitespace removed. If chars is given and not None, remove characters in chars instead. If chars is a str, it will be converted to unicode before stripping

  `split`([sep[, maxsplit]]) → list of strings

    Return a list of the words in S, using sep as the delimiter string. If maxsplit is given, at most maxsplit splits are done. If sep is not specified or is None, any whitespace string is a separator and empty strings are removed from the result.

  `splitlines`(keepends=False) → list of strings

    Return a list of the lines in S, breaking at line boundaries. Line breaks are not included in the resulting list unless keepends is given and true.

  `startswith`(prefix[, start[, end]]) → bool

    Return True if S starts with the specified prefix, False otherwise. With optional start, test S beginning at that position. With optional end, stop comparing S at that position. prefix can also be a tuple of strings to try.

  `strip`([chars]) → unicode

    Return a copy of the string S with leading and trailing whitespace removed. If chars is given and not None, remove characters in chars instead. If chars is a str, it will be converted to unicode before stripping

  `swapcase`() → unicode

    Return a copy of S with uppercase characters converted to lowercase and vice versa.

  `title`() → unicode

    Return a titlecased version of S, i.e. words start with title case characters, all remaining cased characters have lower case.

  `translate`(table) → unicode

    Return a copy of the string S, where all characters have been mapped through the given translation table, which must be a mapping of Unicode ordinals to Unicode ordinals, Unicode strings or None. Unmapped characters are left untouched. Characters mapped to None are deleted.

  `upper`() → unicode

    Return a copy of S converted to uppercase.

  `zfill`(width) → unicode

    Pad a numeric string S with zeros on the left, to fill a field of the specified width. The string S is never truncated.

## optrecurse

class `translate.misc.optrecurse.` `ManPageOption`(*opts, **attrs)
  `take_action`(action, dest, opt, value, values, parser)

    take_action that can handle manpage as well as standard actions

class `translate.misc.optrecurse.` `RecursiveOptionParser`(formats, usetemplates=False, allowmissingtemplate=False, description=None)

  A specialized Option Parser for recursing through directories.

  `add_option`(Option)

    add_option(opt_str, …, kwarg=val, …)

  `check_values`(values : Values, args : [string])

    -> (values : Values, args : [string])

    Check that the supplied option values and leftover arguments are valid. Returns the option values and leftover arguments (possibly adjusted, possibly completely new – whatever you like). Default implementation just returns the passed-in values; subclasses may override as desired.

  `checkoutputsubdir`(options, subdir)

    Checks to see if subdir under options.output needs to be created, creates if neccessary.

  `define_option`(option)

    Defines the given option, replacing an existing one of the same short name if neccessary…

  `destroy`()

    Declare that you are done with this OptionParser. This cleans up reference cycles so the OptionParser (and all objects referenced by it) can be garbage-collected promptly. After calling destroy(), the OptionParser is unusable.

  `disable_interspersed_args`()

    Set parsing to stop on the first non-option. Use this if you have a command processor which runs another command that has options of its own and you want to make sure these options don’t get confused.

  `enable_interspersed_args`()

    Set parsing to not stop on the first non-option, allowing interspersing switches with command arguments. This is the default behavior. See also disable_interspersed_args() and the class documentation description of the attribute allow_interspersed_args.

  `error`(msg : string)

    Print a usage message incorporating ‘msg’ to stderr and exit. If you override this in a subclass, it should not return – it should either exit or raise an exception.

  `finalizetempoutputfile`(options, outputfile, fulloutputpath)

    Write the temp outputfile to its final destination.

  `format_manpage`()

    returns a formatted manpage

  `getformathelp`(formats)

    Make a nice help string for describing formats…

  `getfullinputpath`(options, inputpath)

    Gets the absolute path to an input file.

  `getfulloutputpath`(options, outputpath)

    Gets the absolute path to an output file.

  `getfulltemplatepath`(options, templatepath)

    Gets the absolute path to a template file.

  `getoutputname`(options, inputname, outputformat)

    Gets an output filename based on the input filename.

  `getoutputoptions`(options, inputpath, templatepath)

    Works out which output format and processor method to use…

  `getpassthroughoptions`(options)

    Get the options required to pass to the filtermethod…

  `gettemplatename`(options, inputname)

    Gets an output filename based on the input filename.

  `getusageman`(option)

    returns the usage string for the given option

  `getusagestring`(option)

    returns the usage string for the given option

  `isexcluded`(options, inputpath)

    Checks if this path has been excluded.

  `isrecursive`(fileoption, filepurpose='input')

    Checks if fileoption is a recursive file.

  `isvalidinputname`(inputname)

    Checks if this is a valid input filename.

  `mkdir`(parent, subdir)

    Makes a subdirectory (recursively if neccessary).

  `openinputfile`(options, fullinputpath)

    Opens the input file.

  `openoutputfile`(options, fulloutputpath)

    Opens the output file.

  `opentemplatefile`(options, fulltemplatepath)

    Opens the template file (if required).

  `opentempoutputfile`(options, fulloutputpath)

    Opens a temporary output file.

  `parse_args`(args=None, values=None)

    Parses the command line options, handling implicit input/output args.

  `print_help`(file : file = stdout)

    Print an extended help message, listing all options and any help text provided with them, to ‘file’ (default stdout).

  `print_manpage`(file=None)

    outputs a manpage for the program using the help information

  `print_usage`(file : file = stdout)

    Print the usage message for the current program (self.usage) to ‘file’ (default stdout). Any occurrence of the string “%prog” in self.usage is replaced with the name of the current program (basename of sys.argv[0]). Does nothing if self.usage is empty or not defined.

  `print_version`(file : file = stdout)

    Print the version message for this program (self.version) to ‘file’ (default stdout). As with print_usage(), any occurrence of “%prog” in self.version is replaced by the current program’s name. Does nothing if self.version is empty or undefined.

  `processfile`(fileprocessor, options, fullinputpath, fulloutputpath, fulltemplatepath)

    Process an individual file.

  `recurseinputfilelist`(options)

    Use a list of files, and find a common base directory for them.

  `recurseinputfiles`(options)

    Recurse through directories and return files to be processed.

  `recursiveprocess`(options)

    Recurse through directories and process files.

  `run`()

    Parses the arguments, and runs recursiveprocess with the resulting options…

  `set_usage`(usage=None)

    sets the usage string - if usage not given, uses getusagestring for each option

  `seterrorleveloptions`()

    Sets the errorlevel options.

  `setformats`(formats, usetemplates)

    Sets the format options using the given format dictionary.

    Parameters: | formats (Dictionary) – The dictionary keys should be: Single strings (or 1-tuples) containing an input format (if not usetemplates) Tuples containing an input format and template format (if usetemplates) Formats can be None to indicate what to do with standard input The dictionary values should be tuples of outputformat (string) and processor method.

  `setmanpageoption`()

    creates a manpage option that allows the optionparser to generate a manpage

  `setprogressoptions`()

    Sets the progress options.

  `splitext`(pathname)

    Splits pathname into name and ext, and removes the extsep.

    Parameters: | pathname (string) – A file path
    Returns: | root, ext
    Return type: | tuple

  `splitinputext`(inputpath)

    Splits an inputpath into name and extension.

  `splittemplateext`(templatepath)

    Splits a templatepath into name and extension.

  `templateexists`(options, templatepath)

    Returns whether the given template exists…

  `warning`(msg, options=None, exc_info=None)

    Print a warning message incorporating ‘msg’ to stderr and exit.

## ourdom

module that provides modified DOM functionality for our needs

Note that users of ourdom should ensure that no code might still use classes directly from minidom, like minidom.Element, minidom.Document or methods such as minidom.parseString, since the functionality provided here will not be in those objects.

`translate.misc.ourdom.` `getElementsByTagName_helper`(parent, name, dummy=None)

  A reimplementation of getElementsByTagName as an iterator.

  Note that this is not compatible with getElementsByTagName that returns a list, therefore, the class below exposes this through yieldElementsByTagName

`translate.misc.ourdom.` `getnodetext`(node)

  returns the node’s text by iterating through the child nodes

`translate.misc.ourdom.` `parse`(file, parser=None, bufsize=None)

  Parse a file into a DOM by filename or file object.

`translate.misc.ourdom.` `parseString`(string, parser=None)

  Parse a file into a DOM from a string.

`translate.misc.ourdom.` `searchElementsByTagName_helper`(parent, name, onlysearch)

  limits the search to within tags occuring in onlysearch

`translate.misc.ourdom.` `writexml_helper`(self, writer, indent='', addindent='', newl='')

  A replacement for writexml that formats it like typical XML files. Nodes are intendented but text nodes, where whitespace can be significant, are not indented.

## progressbar

Progress bar utilities for reporting feedback on the progress of an application.

class `translate.misc.progressbar.` `DotsProgressBar`

  An ultra-simple progress indicator that just writes a dot for each action

  `show`(verbosemessage)

    show a dot for progress :-)

class `translate.misc.progressbar.` `HashProgressBar`(*args, **kwargs)

  A ProgressBar which knows how to go back to the beginning of the line.

class `translate.misc.progressbar.` `MessageProgressBar`(*args, **kwargs)

  A ProgressBar that just writes out the messages without any progress display

class `translate.misc.progressbar.` `NoProgressBar`

  An invisible indicator that does nothing.

  `show`(verbosemessage)

    show nothing for progress :-)

class `translate.misc.progressbar.` `ProgressBar`(minValue=0, maxValue=100, totalWidth=50)

  A plain progress bar that doesn’t know very much about output.

  `show`(verbosemessage)

    displays the progress bar

## quote

String processing utilities for extracting strings with various kinds of delimiters

`translate.misc.quote.` `entitydecode`(source, name2codepoint)

  Decode `source` using entities from `name2codepoint`.

  Parameters: | source (unicode) – Source string to decode name2codepoint (`dict()`) – Dictionary mapping entity names (without the the leading `&` or the trailing `;`) to code points

`translate.misc.quote.` `entityencode`(source, codepoint2name)

  Encode `source` using entities from `codepoint2name`.

  Parameters: | source (unicode) – Source string to encode codepoint2name (`dict()`) – Dictionary mapping code points to entity names (without the the leading `&` or the trailing `;`)

`translate.misc.quote.` `escapecontrols`(source)

  escape control characters in the given string

`translate.misc.quote.` `extract`(source, startdelim, enddelim, escape=None, startinstring=False, allowreentry=True)

  Extracts a doublequote-delimited string from a string, allowing for backslash-escaping returns tuple of (quoted string with quotes, still in string at end).

`translate.misc.quote.` `extractwithoutquotes`(source, startdelim, enddelim, escape=None, startinstring=False, includeescapes=True, allowreentry=True)

  Extracts a doublequote-delimited string from a string, allowing for backslash-escaping includeescapes can also be a function that takes the whole escaped string and returns the replaced version.

`translate.misc.quote.` `find_all`(searchin, substr)

  Returns a list of locations where substr occurs in searchin locations are not allowed to overlap

`translate.misc.quote.` `htmlentitydecode`(source)

  Decode source using HTML entities e.g. `&copy;` -> ©.

  Parameters: | source (unicode) – Source string to decode

`translate.misc.quote.` `htmlentityencode`(source)

  Encode `source` using HTML entities e.g. © -> `&copy;`

  Parameters: | source (unicode) – Source string to encode

`translate.misc.quote.` `java_utf8_properties_encode`(source)

  Encodes source in the escaped-unicode encoding used by java utf-8 .properties files.

`translate.misc.quote.` `javapropertiesencode`(source)

  Encodes source in the escaped-unicode encoding used by Java .properties files

`translate.misc.quote.` `mozillaescapemarginspaces`(source)

  Escape leading and trailing spaces for Mozilla .properties files.

`translate.misc.quote.` `propertiesdecode`(source)

  Decodes source from the escaped-unicode encoding used by .properties files.

  Java uses Latin1 by default, and Mozilla uses UTF-8 by default.

  Since the .decode(“unicode-escape”) routine decodes everything, and we don’t want to we reimplemented the algorithm from Python Objects/unicode.c in Python and modify it to retain escaped control characters.

## wsgi

Wrapper to launch the bundled CherryPy server.

`translate.misc.wsgi.` `launch_server`(host, port, app, **kwargs)

  Use CherryPy’s WSGI server, a multithreaded scallable server.

## wStringIO

A thin wrapper around BytesIO that accepts and auto-convert non bytes input

class `translate.misc.wStringIO.` `CatchStringOutput`(onclose)

  catches the output before it is closed and sends it to an onclose method

  `close`()

    wrap the underlying close method, to pass the value to onclose before it goes

  `slam`()

    use this method to force the closing of the stream if it isn’t closed yet

class `translate.misc.wStringIO.` `StringIO`(buf='')

## xml_helpers

Helper functions for working with XML.

`translate.misc.xml_helpers.` `getText`(node, xml_space='preserve')

  Extracts the plain text content out of the given node.

  This method checks the xml:space attribute of the given node, and takes an optional default to use in case nothing is specified in this node.

`translate.misc.xml_helpers.` `getXMLlang`(node)

  Gets the xml:lang attribute on node

`translate.misc.xml_helpers.` `getXMLspace`(node, default=None)

  Gets the xml:space attribute on node

`translate.misc.xml_helpers.` `namespaced`(namespace, name)

  Returns name in Clark notation within the given namespace.

  For example namespaced(“source”) in an XLIFF document might return:

      {urn:oasis:names:tc:xliff:document:1.1}source

  This is needed throughout lxml.

`translate.misc.xml_helpers.` `normalize_space`(text)

  Normalize the given text for implementation of `xml:space="default"`.

`translate.misc.xml_helpers.` `normalize_xml_space`(node, xml_space, remove_start=False)

  normalize spaces following the nodes xml:space, or alternatively the given xml_space parameter.

`translate.misc.xml_helpers.` `setXMLlang`(node, lang)

  Sets the xml:lang attribute on node

`translate.misc.xml_helpers.` `setXMLspace`(node, value)

  Sets the xml:space attribute on node

`translate.misc.xml_helpers.` `string_xpath` = string()

  Return a non-normalized string in the node subtree

`translate.misc.xml_helpers.` `string_xpath_normalized` = normalize-space()

  Return a (space) normalized string in the node subtree

`translate.misc.xml_helpers.` `xml_preserve_ancestors` = ancestor-or-self::*[attribute::xml:space='preserve']

  All ancestors with xml:space=’preserve’

`translate.misc.xml_helpers.` `xml_space_ancestors` = ancestor-or-self::*/attribute::xml:space

  All xml:space attributes in the ancestors