Examples — cygenja 0.5.1 documentation

# Examples

In this section, we demonstrate the use of cygenja to generate the CySparse library.

## Init

We start by creating a cygenja engine:

    from cygenja.generator import Generator
    ...

    # read cysparse.cfg
    cysparse_config = ConfigParser.SafeConfigParser()
    cysparse_config.read('cysparse.cfg')

    # create logger
    logger = make_logger(cysparse_config=cysparse_config)

    # cygenja engine
    current_directory = os.path.dirname(os.path.abspath(__file__))
    cygenja_engine = Generator(current_directory, logger=logger)

`make_logger` is just a wrapper around a logging logger. This logger is not mandatory but can be quite handy to debug sessions. The `current_directory` can be absolute or relative. In this example, let’s say its value is `'cysparse'` [1], the main project directory.

We now define some variables:

    ELEMENT_TYPES = ['INT32_t', 'INT64_t',
                     'FLOAT32_t', 'FLOAT64_t', 'FLOAT128_t',
                     'COMPLEX64_t', 'COMPLEX128_t', 'COMPLEX256_t']
    INDEX_TYPES = ['INT32_t', 'INT64_t']
    ...

    GENERAL_CONTEXT = {
                    'type_list': ELEMENT_TYPES,
                    'index_list' : INDEX_TYPES,
                    'default_index_type' : DEFAULT_INDEX_TYPE,
                    'integer_list' : INTEGER_ELEMENT_TYPES,
                    'real_list' : REAL_ELEMENT_TYPES,
                    'complex_list' : COMPLEX_ELEMENT_TYPES,
                    ...
                  }

## File extensions

CySparse is written essentially in Cython. We can thus generate four types of files: `.pyx`, `.pxd`, `.pxi` and of course `.py` files. For each type of file, we have defined a corresponding extension for a template file: `.cpx`, `.cpd`, `.cpi` and `cpy`. We register this correspondance like so:

    # register extensions
    cygenja_engine.register_extension('.cpy', '.py')
    cygenja_engine.register_extension('.cpx', '.pyx')
    cygenja_engine.register_extension('.cpd', '.pxd')
    cygenja_engine.register_extension('.cpi', '.pxi')

Now, each time cygenja will encounter a template `.cpx` file, it will generate one or several corresponding `.pyx` files.

## Filters

Jinja2 filters are essentially functions that take a string as input and return a modified version of this string. Here is an example:

    def cysparse_type_to_numpy_c_type(cysparse_type):
        """
        Transform a :program:`CySparse` enum type into the corresponding
        :program:`NumPy` C-type.

        For instance:

            INT32_T -> npy_int32

        Args:
            cysparse_type:

        """
        return 'npy_' + str(cysparse_type.lower()[:-2])

We keep the same name for the function as the function name itself to register it (this is not mandatory):

    engine.register_filter('cysparse_type_to_numpy_c_type', cysparse_type_to_numpy_c_type)

Now you can use `cysparse_type_to_numpy_c_type()` in your Jinja2 template [2]:

    cnp.ndarray[cnp.@index|cysparse_type_to_numpy_c_type@, ndim=1] a_row =
        cnp.PyArray_SimpleNew( 1, dmat, cnp.@index|cysparse_type_to_numpy_enum_type@)

## Actions

Before we can register any cygenja actions, we need to define some callbacks. Here are a few examples:

    def single_generation():
        yield '', GENERAL_CONTEXT

    def generate_following_only_index():
        GENERAL_CONTEXT['type'] = None
        for index in INDEX_TYPES:
            GENERAL_CONTEXT['index'] = index

            yield '_%s' % index, GENERAL_CONTEXT

The first function, `single_generation`, only generates one file without changing its name (the extension will be changed though). The second function, `generate_following_only_index`, is more interesting. It generates one file for each index type. These files all have a suffix `_index` attached to their names (i.e. `_INT32_t`, `_INT64_t`) and the `GENERAL_CONTEXT` `dict` is changed every time with the corresponding entry `index` updated. Here is a more complex version where we generate files with respect to an index type but also an element type:

    def generate_following_index_and_element():
        for index in INDEX_TYPES:
            GENERAL_CONTEXT['index'] = index
            for type in ELEMENT_TYPES:
                GENERAL_CONTEXT['type'] = type
                yield '_%s_%s' % (index, type), GENERAL_CONTEXT

Because these functions are user-defined, you have total control and can generate any complicated combinations that you like.

Now we can use these callbacks and register them. For instance:

    engine.register_action('config', '*.*', single_generation)

This registers any template file (`'*.*'`) located in `cysparse/config` (linux version) with the user callback `single_generation`.

    engine.register_action('cysparse/sparse/sparse_utils/generic',
                           'generate_indices.*',
                           generate_following_only_index)

This time, we associate template files with the name `generate_indices` inside the subdirectory `cysparse/sparse/sparse_utils/generic` with the `generate_following_only_index` callback [3].

Here, we only associate template files with extension `.cpi` to the `generate_following_index_and_element` callback inside subdirectory `cysparse/sparse/csc_mat_matrices/csc_mat_kernel`:

    cygenja_engine.register_action('cysparse/sparse/csc_mat_matrices/csc_mat_kernel',
                                   '*.cpi',
                                   generate_following_index_and_element)

You are allowed to define multiple actions for one subdirectory:

    cygenja_engine.register_action('cysparse/sparse/sparse_utils/generic',
                                   'find.*',
                                   generate_following_index_and_element)

    cygenja_engine.register_action('cysparse/sparse/sparse_utils/generic',
                                   'generate_indices.*',
                                   generate_following_only_index)

Remember that if a template file can be associated with several actions, only the first action will be triggered.

## File generation

We are now ready to generate some files from some templates. There is only one method to call: `generate`. Its signature is:

    engine.generate(dir_pattern, file_pattern, action_ch='g', recursively=True, force=False)

where `dir_pattern` is a glob pattern used to match directories and `file_pattern` a fnmatch pattern taken from all matching directories. This combination allows you to refine your operations with a great flexibility. The `action_ch` argument can be `g` (generate files), `c` (clean or erase files) or `d` (dry run).

This is the beginning of the output cygenja generates when asked a dry run for all file generation:

    Process file 'config/setup.cpy' with function 'single_generation':
       -> config/setup.py
    Process file 'cysparse/sparse/ll_mat.cpx' with function 'single_generation':
       -> cysparse/sparse/ll_mat.pyx
    Process file 'cysparse/sparse/csc_mat_matrices/csc_mat.cpx' with
                                function 'generate_following_index_and_element':
       -> cysparse/sparse/csc_mat_matrices/csc_mat_INT32_t_INT32_t.pyx
       -> cysparse/sparse/csc_mat_matrices/csc_mat_INT32_t_INT64_t.pyx
       -> cysparse/sparse/csc_mat_matrices/csc_mat_INT32_t_FLOAT32_t.pyx
       -> cysparse/sparse/csc_mat_matrices/csc_mat_INT32_t_FLOAT64_t.pyx
       -> cysparse/sparse/csc_mat_matrices/csc_mat_INT32_t_FLOAT128_t.pyx
       -> cysparse/sparse/csc_mat_matrices/csc_mat_INT32_t_COMPLEX64_t.pyx
       -> cysparse/sparse/csc_mat_matrices/csc_mat_INT32_t_COMPLEX128_t.pyx
    ...

At the moment of writing, we have 23 registered actions that trigger 492 file generations.

Footnotes

[1] | Yes, we are well aware that this not what is expected from the code. `os.path.abspath(__file__)` will never only return `cysparse`.

[2] | CySparse‘s Jinja2 environment allows us to use variables names like so: `@my_variable@`.

[3] | Thus the real directory is `cysparse/cysparse/sparse/sparse_utils/generic`.