7. Tests — CubicWeb 3.13.2

### Navigation

- index
- modules |
- next |
- previous |
- CubicWeb 3.13.2 »
- Repository development »

# 7. Tests

## 7.1. Unit tests

The CubicWeb framework provides the `cubicweb.devtools.testlib.CubicWebTC` test base class .

Tests shall be put into the mycube/test directory. Additional test data shall go into mycube/test/data.

It is much advised to write tests concerning entities methods, actions, hooks and operations, security. The `CubicWebTC` base class has convenience methods to help test all of this.

In the realm of views, automatic tests check that views are valid XHTML. See Automatic views testing for details. Since 3.9, bases for web functional testing using windmill are set. See test cases in cubicweb/web/test/windmill and python wrapper in cubicweb/web/test_windmill/ if you want to use this in your own cube.

Most unit tests need a live database to work against. This is achieved by CubicWeb using automatically sqlite (bundled with Python, see http://docs.python.org/library/sqlite3.html) as a backend.

The database is stored in the mycube/test/tmpdb, mycube/test/tmpdb-template files. If it does not (yet) exists, it will be built automatically when the test suit starts.

Warning

Whenever the schema changes (new entities, attributes, relations) one must delete these two files. Changes concerned only with entity or relation type properties (constraints, cardinalities, permissions) and generally dealt with using the sync_schema_props_perms() fonction of the migration environment need not a database regeneration step.

### 7.1.1. Unit test by example

We start with an example extracted from the keyword cube (available from http://www.cubicweb.org/project/cubicweb-keyword).

    from cubicweb.devtools.testlib import CubicWebTC
    from cubicweb import ValidationError

    class ClassificationHooksTC(CubicWebTC):

        def setup_database(self):
            req = self.request()
            group_etype = req.find_one_entity('CWEType', name='CWGroup')
            c1 = req.create_entity('Classification', name=u'classif1',
                                   classifies=group_etype)
            user_etype = req.find_one_entity('CWEType', name='CWUser')
            c2 = req.create_entity('Classification', name=u'classif2',
                                   classifies=user_etype)
            self.kw1 = req.create_entity('Keyword', name=u'kwgroup', included_in=c1)
            self.kw2 = req.create_entity('Keyword', name=u'kwuser', included_in=c2)

        def test_cannot_create_cycles(self):
            # direct obvious cycle
            self.assertRaises(ValidationError, self.kw1.set_relations,
                              subkeyword_of=self.kw1)
            # testing indirect cycles
            kw3 = self.execute('INSERT Keyword SK: SK name "kwgroup2", SK included_in C, '
                               'SK subkeyword_of K WHERE C name "classif1", K eid %s'
                               % self.kw1.eid).get_entity(0,0)
            self.kw1.set_relations(subkeyword_of=kw3)
            self.assertRaises(ValidationError, self.commit)

The test class defines a `setup_database()` method which populates the database with initial data. Each test of the class runs with this pre-populated database. A commit is done automatically after the `setup_database()` call. You don’t have to call it explicitely.

The test case itself checks that an Operation does it job of preventing cycles amongst Keyword entities.

create_entity is a useful method, which easily allows to create an entity. You can link this entity to others entities, by specifying as argument, the relation name, and the entity to link, as value. In the above example, the Classification entity is linked to a CWEtype via the relation classifies. Conversely, if you are creating a CWEtype entity, you can link it to a Classification entity, by adding reverse_classifies as argument.

Note

`commit()` method is not called automatically in test_XXX methods. You have to call it explicitely if needed (notably to test operations). It is a good practice to call `clear_all_caches()` on entities after a commit to avoid request cache effects.

You can see an example of security tests in the Step 1: configuring security into the schema.

It is possible to have these tests run continuously using apycot.

#### 7.1.1.1. Managing connections or users

Since unit tests are done with the SQLITE backend and this does not support multiple connections at a time, you must be careful when simulating security, changing users.

By default, tests run with a user with admin privileges. This user/connection must never be closed.

Before a self.login, one has to release the connection pool in use with a self.commit, self.rollback or self.close.

The login method returns a connection object that can be used as a context manager:

    with self.login('user1') as user:
        req = user.req
        req.execute(...)

On exit of the context manager, either a commit or rollback is issued, which releases the connection.

When one is logged in as a normal user and wants to switch back to the admin user without committing, one has to use self.restore_connection().

Usage with restore_connection:

    # execute using default admin connection
    self.execute(...)
    # I want to login with another user, ensure to free admin connection pool
    # (could have used rollback but not close here
    # we should never close defaut admin connection)
    self.commit()
    cnx = self.login('user')
    # execute using user connection
    self.execute(...)
    # I want to login with another user or with admin user
    self.commit();  cnx.close()
    # restore admin connection, never use cnx = self.login('admin'), it will return
    # the default admin connection and one may be tempted to close it
    self.restore_connection()

Warning

Do not use the references kept to the entities created with a connection from another !

### 7.1.2. Email notifications tests

When running tests potentially generated e-mails are not really sent but is found in the list MAILBOX of module `cubicweb.devtools.testlib`.

You can test your notifications by analyzing the contents of this list, which contains objects with two attributes:

- recipients, the list of recipients
- msg, object email.Message

Let us look at simple example from the `blog` cube.

    from cubicweb.devtools.testlib import CubicWebTC, MAILBOX

    class BlogTestsCubicWebTC(CubicWebTC):
        """test blog specific behaviours"""

        def test_notifications(self):
            req = self.request()
            cubicweb_blog = req.create_entity('Blog', title=u'cubicweb',
                                description=u'cubicweb is beautiful')
            blog_entry_1 = req.create_entity('BlogEntry', title=u'hop',
                                             content=u'cubicweb hop')
            blog_entry_1.set_relations(entry_of=cubicweb_blog)
            blog_entry_2 = req.create_entity('BlogEntry', title=u'yes',
                                             content=u'cubicweb yes')
            blog_entry_2.set_relations(entry_of=cubicweb_blog)
            self.assertEqual(len(MAILBOX), 0)
            self.commit()
            self.assertEqual(len(MAILBOX), 2)
            mail = MAILBOX[0]
            self.assertEqual(mail.subject, '[data] hop')
            mail = MAILBOX[1]
            self.assertEqual(mail.subject, '[data] yes')

### 7.1.3. Visible actions tests

It is easy to write unit tests to test actions which are visible to user or to a category of users. Let’s take an example in the conference cube.

    class ConferenceActionsTC(CubicWebTC):

        def setup_database(self):
            self.conf = self.create_entity('Conference',
                                           title=u'my conf',
                                           url_id=u'conf',
                                           start_on=date(2010, 1, 27),
                                           end_on = date(2010, 1, 29),
                                           call_open=True,
                                           reverse_is_chair_at=chair,
                                           reverse_is_reviewer_at=reviewer)

        def test_admin(self):
            req = self.request()
            rset = req.find_entities('Conference')
            self.assertListEqual(self.pactions(req, rset),
                                  [('workflow', workflow.WorkflowActions),
                                   ('edit', confactions.ModifyAction),
                                   ('managepermission', actions.ManagePermissionsAction),
                                   ('addrelated', actions.AddRelatedActions),
                                   ('delete', actions.DeleteAction),
                                   ('generate_badge_action', badges.GenerateBadgeAction),
                                   ('addtalkinconf', confactions.AddTalkInConferenceAction)
                                   ])
            self.assertListEqual(self.action_submenu(req, rset, 'addrelated'),
                                  [(u'add Track in_conf Conference object',
                                    u'http://testing.fr/cubicweb/add/Track'
                                    u'?__linkto=in_conf%%3A%(conf)s%%3Asubject&'
                                    u'__redirectpath=conference%%2Fconf&'
                                    u'__redirectvid=' % {'conf': self.conf.eid}),
                                   ])

You just have to execute a rql query corresponding to the view you want to test, and to compare the result of `pactions()` with the list of actions that must be visible in the interface. This is a list of tuples. The first element is the action’s __regid__, the second the action’s class.

To test actions in submenu, you just have to test the result of `action_submenu()` method. The last parameter of the method is the action’s category. The result is a list of tuples. The first element is the action’s title, and the second element the action’s url.

## 7.2. Automatic views testing

This is done automatically with the `cubicweb.devtools.testlib.AutomaticWebTest` class. At cube creation time, the mycube/test/test_mycube.py file contains such a test. The code here has to be uncommented to be usable, without further modification.

The `auto_populate` method uses a smart algorithm to create pseudo-random data in the database, thus enabling the views to be invoked and tested.

Depending on the schema, hooks and operations constraints, it is not always possible for the automatic auto_populate to proceed.

It is possible of course to completely redefine auto_populate. A lighter solution is to give hints (fill some class attributes) about what entities and relations have to be skipped by the auto_populate mechanism. These are:

- no_auto_populate, may contain a list of entity types to skip
- ignored_relations, may contain a list of relation types to skip
- application_rql, may contain a list of rql expressions that auto_populate cannot guess by itself; these must yield resultsets against which views may be selected.

Warning

Take care to not let the imported AutomaticWebTest in your test module namespace, else both your subclass and this parent class will be run.

## 7.3. Cache heavy database setup

Some tests suite require a complex setup of the database that takes seconds (or event minutes) to complete. Doing the whole setup for all individual tests make the whole run very slow. The `CubicWebTC` class offer a simple way to prepare specific database once for multiple tests. The test_db_id class attribute of your `CubicWebTC` must be set a unique identifier and the `pre_setup_database()` class method build the cached content. As the `pre_setup_database()` method is not grantee to be called, you must not set any class attribut to be used during test there. Databases for each test_db_id are automatically created if not already in cache. Clearing the cache is up to the user. Cache files are found in the `data/database` subdirectory of your test directory.

Warning

Take care to always have the same `pre_setup_database()` function for all call with a given test_db_id otherwise you test will have unpredictable result given the first encountered one.

## 7.4. Testing on a real-life database

The `CubicWebTC` class uses the cubicweb.devtools.ApptestConfiguration configuration class to setup its testing environment (database driver, user password, application home, and so on). The cubicweb.devtools module also provides a RealDatabaseConfiguration class that will read a regular cubicweb sources file to fetch all this information but will also prevent the database to be initalized and reset between tests.

For a test class to use a specific configuration, you have to set the _config class attribute on the class as in:

    from cubicweb.devtools import RealDatabaseConfiguration
    from cubicweb.devtools.testlib import CubicWebTC

    class BlogRealDatabaseTC(CubicWebTC):
        _config = RealDatabaseConfiguration('blog',
                                            sourcefile='/path/to/realdb_sources')

        def test_blog_rss(self):
            req = self.request()
            rset = req.execute('Any B ORDERBY D DESC WHERE B is BlogEntry, '
                               'B created_by U, U login "logilab", B creation_date D')
            self.view('rss', rset)

## 7.5. Testing with other cubes

Sometimes a small component cannot be tested all by itself, so one needs to specify other cubes to be used as part of the the unit test suite. This is handled by the `bootstrap_cubes` file located under `mycube/test/data`. One example from the preview cube:

    card, file, preview

The format is:

- possibly several empy lines or lines starting with `#` (comment lines)
- one line containing a coma separated list of cube names.

It is also possible to add a `schema.py` file in `mycube/test/data`, which will be used by the testing framework, therefore making new entity types and relations available to the tests.

## 7.6. Literate programming

CubicWeb provides some literate programming capabilities. The cubicweb-ctl tool shell command accepts differents format files. If your file ends with .txt or .rst, the file will be parsed by `doctest.testfile` with CubicWeb Migration API enabled in it.

Create a scenario.txt file into test/ directory and fill with some content. Please refer the `doctest.testfile` documentation.

Then, you can run it directly by:

    $ cubicweb-ctl shell <cube_instance> test/scenario.txt

When your scenario file is ready, put it in a new test case to be able to run it automatically.

    from os.path import dirname, join
    from logilab.common.testlib import unittest_main
    from cubicweb.devtools.testlib import CubicWebTC

    class AcceptanceTC(CubicWebTC):

            def test_scenario(self):
                    self.assertDocTestFile(join(dirname(__file__), 'scenario.txt'))

    if __name__ == '__main__':
            unittest_main()

### 7.6.1. Skipping a scenario

If you want to set up initial conditions that you can’t put in your unit test case, you have to use a `KeyboardInterrupt` exception only because of the way `doctest` module will catch all the exceptions internally.

    >>> if condition_not_met:
    ...     raise KeyboardInterrupt('please, check your fixture.')

### 7.6.2. Passing paramaters

Using extra arguments to parametrize your scenario is possible by prepend them by double dashes.

Please refer to the cubicweb-ctl shell –help usage.

Important

Your scenario file must be utf-8 encoded.

## 7.7. Test APIS

### 7.7.1. Using Pytest

The pytest utility (shipping with logilab-common, which is a mandatory dependency of CubicWeb) extends the Python unittest functionality and is the preferred way to run the CubicWeb test suites. Bare unittests also work the usual way.

To use it, you may:

- just launch pytest in your cube to execute all tests (it will discover them automatically)
- launch pytest unittest_foo.py to execute one test file
- launch pytest unittest_foo.py bar to execute all test methods and all test cases whose name contain bar

Additionally, the -x option tells pytest to exit at the first error or failure. The -i option tells pytest to drop into pdb whenever an exception occurs in a test.

When the -x option has been used and the run stopped on a test, it is possible, after having fixed the test, to relaunch pytest with the -R option to tell it to start testing again from where it previously failed.

### 7.7.2. Using the TestCase base class

The base class of CubicWebTC is logilab.common.testlib.TestCase, which provides a lot of convenient assertion methods.

class `logilab.common.testlib.` `TestCase`(methodName='runTest')

  A unittest.TestCase extension with some additional methods.

  `assertDictEquals`(*args, **kwargs)

    compares two dicts

    If the two dict differ, the first difference is shown in the error message :param dict1: a Python Dictionary :param dict2: a Python Dictionary :param msg: custom message (String) in case of failure

  `assertDirEqual`(*args, **kwargs)
    compares two files using difflib
  `assertDirEquals`(*args, **kwargs)
    compares two files using difflib
  `assertFileEqual`(*args, **kwargs)
    compares two files using difflib
  `assertFileEquals`(*args, **kwargs)
    compares two files using difflib
  `assertFloatAlmostEquals`(*args, **kwargs)

    compares if two floats have a distance smaller than expected precision.

    Parameters: | obj – a Float other – another Float to be comparted to <obj> prec – a Float describing the precision relative – boolean switching to relative/absolute precision msg – a String for a custom message

  `assertIsInstance`(obj, klass, msg=None, strict=False)

    check if an object is an instance of a class

    Parameters: | obj – the Python Object to be checked klass – the target class msg – a String for a custom message strict – if True, check that the class of <obj> is <klass>; else check with ‘isinstance’

  `assertLineEqual`(*args, **kwargs)

    compare two strings and assert that the text lines of the strings are equal.

    Parameters: | string1 – a String string2 – a String msg – custom message (String) in case of failure striplines – Boolean to trigger line stripping before comparing

  `assertLinesEquals`(*args, **kwargs)

    compare two strings and assert that the text lines of the strings are equal.

    Parameters: | string1 – a String string2 – a String msg – custom message (String) in case of failure striplines – Boolean to trigger line stripping before comparing

  `assertListEquals`(*args, **kwargs)

    compares two lists

    If the two list differ, the first difference is shown in the error message

    Parameters: | list_1 – a Python List list_2 – a second Python List msg – custom message (String) in case of failure

  `assertNone`(*args, **kwargs)

    assert obj is None

    Parameter: | obj – Python Object to be tested

  `assertNotNone`(*args, **kwargs)
    assert obj is not None
  `assertRaises`(excClass, callableObj=None, *args, **kwargs)

    override default failUnlessRaises method to return the raised exception instance.

    Fail unless an exception of class excClass is thrown by callableObj when invoked with arguments args and keyword arguments kwargs. If a different type of exception is thrown, it will not be caught, and the test case will be deemed to have suffered an error, exactly as for an unexpected exception.

    CAUTION! There are subtle differences between Logilab and unittest2 - exc is not returned in standard version - context capabilities in standard version - try/except/else construction (minor)

    Parameters: | excClass – the Exception to be raised callableObj – a callable Object which should raise <excClass> args – a List of arguments for <callableObj> kwargs – a List of keyword arguments for <callableObj>

  `assertSetEquals`(*args, **kwargs)

    compares two sets and shows difference between both

    Don’t use it for iterables other than sets.

    Parameters: | got – the Set that we found expected – the second Set to be compared to the first one msg – custom message (String) in case of failure

  `assertStreamEqual`(*args, **kwargs)
    compare two streams (using difflib and readlines())
  `assertStreamEquals`(*args, **kwargs)
    compare two streams (using difflib and readlines())
  `assertTextEqual`(*args, **kwargs)

    compare two multiline strings (using difflib and splitlines())

    Parameters: | text1 – a Python BaseString text2 – a second Python Basestring junk – List of Caracters msg_prefix – String (message prefix) striplines – Boolean to trigger line stripping before comparing

  `assertTextEquals`(*args, **kwargs)

    compare two multiline strings (using difflib and splitlines())

    Parameters: | text1 – a Python BaseString text2 – a second Python Basestring junk – List of Caracters msg_prefix – String (message prefix) striplines – Boolean to trigger line stripping before comparing

  `assertUnordIterEqual`(*args, **kwargs)

    compares two iterable and shows difference between both

    Parameters: | got – the unordered Iterable that we found expected – the expected unordered Iterable msg – custom message (String) in case of failure

  `assertUnordIterEquals`(*args, **kwargs)

    compares two iterable and shows difference between both

    Parameters: | got – the unordered Iterable that we found expected – the expected unordered Iterable msg – custom message (String) in case of failure

  `assertUnorderedIterableEqual`(*args, **kwargs)

    compares two iterable and shows difference between both

    Parameters: | got – the unordered Iterable that we found expected – the expected unordered Iterable msg – custom message (String) in case of failure

  `assertUnorderedIterableEquals`(*args, **kwargs)

    compares two iterable and shows difference between both

    Parameters: | got – the unordered Iterable that we found expected – the expected unordered Iterable msg – custom message (String) in case of failure

  `assertXMLEqualsTuple`(*args, **kwargs)
    compare an ElementTree Element to a tuple formatted as follow: (tagname, [attrib[, children[, text[, tail]]]])
  `assertXMLStringWellFormed`(*args, **kwargs)

    asserts the XML string is well-formed (no DTD conformance check)

    Parameter: | context – number of context lines in standard message (show all data if negative). Only available with element tree

  `assertXMLWellFormed`(*args, **kwargs)

    asserts the XML stream is well-formed (no DTD conformance check)

    Parameter: | context – number of context lines in standard message (show all data if negative). Only available with element tree

  classmethod `datapath`(*fname)
    joins the object’s datadir and fname
  `defaultTestResult`()
    return a new instance of the defaultTestResult
  `failUnlessRaises`(excClass, callableObj=None, *args, **kwargs)

    override default failUnlessRaises method to return the raised exception instance.

    Fail unless an exception of class excClass is thrown by callableObj when invoked with arguments args and keyword arguments kwargs. If a different type of exception is thrown, it will not be caught, and the test case will be deemed to have suffered an error, exactly as for an unexpected exception.

    CAUTION! There are subtle differences between Logilab and unittest2 - exc is not returned in standard version - context capabilities in standard version - try/except/else construction (minor)

    Parameters: | excClass – the Exception to be raised callableObj – a callable Object which should raise <excClass> args – a List of arguments for <callableObj> kwargs – a List of keyword arguments for <callableObj>

  `innerSkip`(msg=None)
    mark a generative test as skipped for the <msg> reason
  `optval`(option, default=None)
    return the option value or default if the option is not define
  `pdbclass`
    alias of `Debugger`
  `set_description`(descr)
    sets the current test’s description. This can be useful for generative tests because it allows to specify a description per yield
  `shortDescription`()
    override default unittest shortDescription to handle correctly generative tests

### 7.7.3. CubicWebTC API

class `cubicweb.devtools.testlib.` `CubicWebTC`(methodName='runTest')

  abstract class for test using an apptest environment

  attributes:

  - vreg, the vregistry
  - schema, self.vreg.schema
  - config, cubicweb configuration
  - cnx, dbapi connection to the repository using an admin user
  - session, server side session associated to cnx
  - app, the cubicweb publisher (for web testing)
  - repo, the repository object
  - admlogin, login of the admin user
  - admpassword, password of the admin user
  - shell, create and use shell environment
  `adminsession`
    return current server side session (using default manager account)
  `app`
    return a cubicweb publisher
  `assertSentEmail`(subject, recipients=None, nb_msgs=None)

    test recipients in system mailbox for given email subject

    Parameters: | subject – email subject to find in mailbox recipients – list of email recipients nb_msgs – expected number of entries
    Returns: | list of matched emails

  classmethod `create_user`(req, login=None, groups=('users', ), password=None, email=None, commit=True, **kwargs)
    create and return a new user entity
  `ctrl_publish`(req, ctrl='edit')
    call the publish method of the edit controller
  `expect_redirect`(callback, req)
    call the given callback with req as argument, expecting to get a Redirect exception
  `expect_redirect_publish`(req, path='edit')
    call the publish method of the application publisher, expecting to get a Redirect exception
  classmethod `grant_permission`(session, entity, group, pname=None, plabel=None)
    insert a permission on an entity. Will have to commit the main connection to be considered
  classmethod `init_config`(config)

    configuration initialization hooks.

    You may only want to override here the configuraton logic.

    Otherwise, consider to use a different `ApptestConfiguration` defined in the configcls class attribute

  `list_actions_for`(rset)
    returns the list of actions that can be applied on rset
  `list_boxes_for`(rset)
    returns the list of boxes that can be applied on rset
  `list_startup_views`()
    returns the list of startup views
  `list_views_for`(rset)
    returns the list of views that can be applied on rset
  `login`(login, **kwargs)
    return a connection for the given login/password
  classmethod `pre_setup_database`(session, config)

    add your pre database setup code by overriding this method

    Do not forget to set the cls.test_db_id value to enable caching of the result.

  `remote_call`(fname, *args)
    remote json call simulation
  `req_from_url`(url)

    parses url and builds the corresponding CW-web request

    req.form will be setup using the url’s query string

  `request`(rollbackfirst=False, url=None, **kwargs)
    return a web ui request
  `requestcls`
    alias of `FakeRequest`
  `schema`
    return the application schema
  `session`
    return current server side session (using default manager account)
  `setup_database`()
    add your database setup code by overriding this method
  `shell`()
    return a shell session object
  `url_publish`(url)

    takes url, uses application’s app_resolver to find the appropriate controller, and publishes the result.

    This should pretty much correspond to what occurs in a real CW server except the apache-rewriter component is not called.

  `user`(req=None)
    return the application schema
  `view`(vid, rset=None, req=None, template='main-template', **kwargs)

    This method tests the view vid on rset using template

    If no error occurred while rendering the view, the HTML is analyzed and parsed.

    Returns: | an instance of cubicweb.devtools.htmlparser.PageInfo encapsulation the generated HTML

### Navigation

- index
- modules |
- next |
- previous |
- CubicWeb 3.13.2 »
- Repository development »