Usage — cygenja 0.5.1 documentation

# Usage

We briefly describe the use of cygenja. Basically, you register some filters, file extensions and actions before triggering the translation. In the Examples section, you can see cygenja in action as we detail its use to generate the CySparse library.

## The Generator class

The `Generator` class is the main class of the cygenja library. It also is the only class you need to interact with. Its constructor is really simple:

    from cygenja.generator import Generator
    ...

    logger = ...
    env = ...
    engine = Generator('root_directory', env, logger, True)

You provide a `root_directory` directory, a Jinja2 environment, a logger (from the standard `logging` library) and decide if warnings must raise Exceptions or not (default: `False`). The first two arguments are mandatory while the last two are optional. You don’t have to provide a logging engine. We describe the root directory a little further in The root directory, refer the reader to jinja2.Environment for more about the `env` argument and discuss the two last arguments in the next corresponding subsections.

### Logging

A logging engine can be used but is not mandatory. If you don’t want to log cygenja‘s behavior, simply pass None as the value of the logger argument in the constructor (this is the default). The logging engine is an object from Python’s logging library.

    from cygenja.generator import Generator
    import logging

    logger = logging.getLogger('Logger name')
    engine = Generator('root_directory', logger, True)

Three message logging helpers are provided:

    def log_info(self, msg)
    def log_warning(self, msg)
    def log_error(self, msg)

Their names and signatures are quite self-explanatory.

### Raising exceptions on Warnings

Errors always trigger RuntimeErrors while warnings may or may not trigger RuntimeErrors. To raise exceptions on warnings, set the `raise_exception_on_warning` to `True` in the constructor:

    engine = Generator('root_directory', logger=logger, raise_exception_on_warning=True)

By default, `raise_exception_on_warning` is set to `False`.

## Patterns

There are only two types of patterns:

- fnmatch patterns for file names and
- glob patterns for directory names.

This is a general rule for the whole library. When you register an action though, you must provide a directory name, not a directory name pattern.

We encourage the reader to (re)read the specifications of these two libraries.

## The root directory

The root directory is really the main working directory: all file generations can only be done inside subdirectories of this directory.

This is so important that it is worth a warning:

Warning

File generations can only be done inside subdirectories of the root directory.

This directory is given as the first parameter of `Generator`‘s constructor and can be absolute or relative. At any moment, you can retrieve this directory as an absolute path:

    engine = Generator('root_directory', ...)

    absolute_root_directory = engine.root_directory()

## Filters

Filters are simply Jinja2 filters. These filters are registered:

    def my_jinja2_filter(filter_argument):
        ...
        return filter_result

    engine = Generator(...)
    engine.register_filter('my_filter_name', my_jinja2_filter)

where `'my_filter_name'` is the name of the filter used inside your Jinja2 template files and `my_jinja2_filter` is a reference to the actual filter.

The signature of `register_filter` is:

    register_filter(self, filter_name, filter_ref, force=False)

allowing you to register a new filter under an already existing filter name. If you keep `force` set to `False`, a warning is triggered each time you try to register a new filter under an already existing filter name and this new filter is disregarded.

You also can register several filters at once with a dictionary of filters:

    engine = Generator(...)
    filters = { 'f1' : filter1,
                'f2' : filter2}

    engine.register_filters(filters, force=False)

At any time, you can list the registered filters:

    engine = Generator(...)
    print engine.filters_list()

This list also includes predefined Jinja2 filters (see builtin filter). If you only want the filters you registered, invoke:

    engine.registered_filters_list()

## File extensions

cygenja uses a correspondance table between template files and generated files. This table defines a correspondance between file extensions. For instance, to have *.cpd templates generate *.pxd files:

    engine = Generator(...)
    engine.register_extension(`.cpd`, `.pxd`)

Again, we use a `force` switch to force the redefinition of such a correspondance. By default, this switch is set to `False` and if you try to redefine an association with a given template extension, you will trigger a warning and this new correspondance will be disregarded.

You can use a `dict` to register several extensions at once:

    engine = Generator(...)
    ext_correspondance = { '.cpd' : '.pxd',
                           '.cpx' : 'pyx'}
    engine.register_extensions(ext_correspondance, force=False):

As with filters, you can retrieve the registered extensions:

    engine.registered_extensions_list()

Files with extensions registered as template file extensions are systematically parsed, i.e. you cannot use these extensions for files that are not templates because cygenja will try to parse them. What about generated file extensions? Files with these extensions can peacefully coexist with generated files, i.e. existing files, regardless of their extensions, can coexist with generated files and will not be plagued by cyjenja. This means that you can safely delete files: only generated files will be deleted [1].

Note

Only generated files are deleted. You can thus safely delete files with cygenja.

## Actions

Actions (defined in the `GeneratorAction` class) are really the core concept of cygenja: an action correspond to a translation rule. This translation rule makes a correspondance between a subdirectory and a file pattern and a user callback. Here is the signature of the `register_action` method:

    def register_action(self, relative_directory, file_pattern, action_function)

The `relative_directory` argument holds the name of a relative directory from the root directory. The separator is OS dependent. For instance, under linux, you can register the following:

    engine = Generator(...)

    def action_function(...):
        ...
        return ...

    engine.register_action('cysparse/sparse/utils', 'find*.cpy', action_function)

This means that all files corresponding to the `'find*.cpy'` fnmatch pattern inside the `cysparse/sparse/utils` directory can be dealt with the `action_function`.

Contrary to filters and file extensions, you cannot ask for a list of registered actions. But you can ask cygenja to perform a dry session: cygenja outputs what it would normaly do but without taking any action [2].

### User callback

The `action_function()` is a user-defined callback without argument. It returns a file suffix with a corresponding Jinja2 variables dict (this is a simple Python `dict`). Let’s illustrate this by an example:

    GENERAL_CONTEXT = {...}
    INDEX_TYPES = ['INT32', 'INT64']
    ELEMENT_TYPES = ['FLOAT32', 'FLOAT64']

    def generate_following_index_and_type():
        """

        """
        for index in INDEX_TYPES:
            GENERAL_CONTEXT['index'] = index
            for type in ELEMENT_TYPES:
                GENERAL_CONTEXT['type'] = type
                yield '_%s_%s' % (index, type), GENERAL_CONTEXT

The user-defined callback `generate_following_index_and_type()` doesn’t take any input argument and returns the `'_%s_%s'` suffix string together with the variables `dict` `GENERAL_CONTEXT`. This function allows cygenja to create files with this suffix from any matching template file. The `GENERAL_CONTEXT` is given to Jinja2 for the appropriate translation.

For instance, let’s use the `ext_correspondance` extensions `dict` discussed earlier (see File extensions):

    ext_correspondance = { '.cpd' : '.pxd',
                           '.cpx' : 'pyx'}

Any template file with a `.cpd` or `.cpx` extension will be translated into a `_index_type.pxd` or `_index_type.pyx` file respectively. For instance, the template file `my_template_code_file.cpd` will be translated to:

- `my_template_code_file_INT32_FLOAT32.cpd`
- `my_template_code_file_INT32_FLOAT64.cpd`
- `my_template_code_file_INT64_FLOAT32.cpd`
- `my_template_code_file_INT64_FLOAT64.cpd`

As this function is defined by the user, you have total control on what you want to generate or not. In our example, we redefine `GENERAL_CONTEXT['index']` and `GENERAL_CONTEXT['type']` for each index and element types.

We use generators (`yield`) but you could return a `list` if you prefer.

### Incompatible actions

You could register incompatible actions, i.e. register competing actions that would translate a file in different ways. Our approach is to only use the first compatible action and to disregard all the other actions, regardless if they could be applied or not. So the order in which you register your actions is important. A file will be dealt with the first compatible action found. This is worth a warning:

Warning

A template is translated with the first compatible action found and only that action.

### Default action

cygenja allows to define one default action that will be triggered when no other compatible action is found for a given template file that corresponds to a fnmatch pattern:

    engine = Generator(...)

    def default_action():
        return ...

    engine.register_default_action('*.*',  default_action)

Be careful when defining a default action. This action is applied to all template files (corresponding to the fnmatch pattern) for which no compatible action is found. You might want to prefer declaring explicit actions than relying on this implicit default action. That said, if you have lots of default cases, this default action can be very convenient and avoid lots of unnecessary action declarations.

## File generation

To generate the files from template files, there is only one method to invoke: generate(). Its signature is:

    def generate(self, dir_pattern, file_pattern, action_ch='g',
                 recursively=False, force=False)

`dir_pattern` is a `glob` pattern taken from the root directory and it is only used for directories while `file_pattern` is a `fnmatch` pattern taken from all matching directories and is only used for files. The `action_ch` is a character that triggers different behaviours:

- `g`: Generate all files that match both directory and file patterns. This is the default behavior.
- `d`: Same as g but with doing anything, i.e. dry run.
- `c`: Same as g but erasing the generated files instead, i.e. clean.

These actions can be done in a given directory or in all its corresponding subdirectories. To choose between these two options, use the `recursively` switch. Finally, by default, files are only generated if they are outdated, i.e. if they are older than the template they were originated from. You can force the generation with the `force` switch.

Footnotes

[1] | The user is responsible to not to define a translation rule that overwrites any existing files.

[2] | You also have access to the internal `TreeMap` object:
    engine = Generator(...)

    treemap = engine.registered_actions_treemap()
and thus you have access to all its methods. One interesting method is `to_string()`. It gives you a representation of all involved subdirectories.