.. _test-plan:

==============
Test Plan Unit
==============

The test plan unit is a facility that describes a sequence of job definitions
that should be executed together.

Jobs definitions are _selected_ by either listing their identifier or a regular
expression that matches their identifier. Selected jobs are executed in the
sequence they appear in the list, unless they need to be reordered to satisfy
dependencies which always take priority.

Test plans can contain additional meta-data which can be used in a graphical
user interface. You can assign a translatable name and
description to each test plan. 

Test plans are also typical units so they can be defined with the familiar
RFC822-like syntax that is also used for job definitions. They can also be
multiple test plan definitions per file, just like with all the other units,
including job definitions.

Test Plan Fields
-----------------

The following fields can be used in a test plan. Note that **not all** fields
need to be used or even should be used. Please remember that Checkbox needs to
maintain backwards compatibility so some of the test plans it defines may have
non-typical constructs required to ensure proper behavior. You don't have to
copy such constructs when working on a new test plan from scratch

``id``:
    Each test plan needs to have a unique identifier. This is exactly the same
    as with other units that have an identifier (like job definitions
    and categories).

    This field is not used for display purposes but you may need to refer
    to it on command line so keeping it descriptive is useful

``name``:
    A human-readable name of the test plan. The name should be relatively short
    as it may be used to display a list of test plans to the test operator.

    Remember that the user or the test operator may not always be familiar with
    the scope of testing that you are focusing on. Also consider that multiple
    test providers may be always installed at the same time. The translated
    version of the name (and icon, see below) is the only thing that needs
    to allow the test operator to  pick the right test plan.

    Please use short and concrete names like:
     - "Storage Device Certification Tests"
     - "Ubuntu Core Application's Clock Acceptance Tests"
     - "Default Ubuntu Hardware Certification Tests".

    The field has a soft limit of eighty characters. It cannot have multiple
    lines. This field should be marked as translatable by prepending the
    underscore character (\_) in front. This field is mandatory.

``description``:
    A human-readable description of this test plan. Here you can include as
    many or few details as you'd like. Some applications may offer a way
    of viewing this data. In general it is recommended to include a description
    of what is being tested so that users can make an informed decision but
    please in mind that the ``name`` field alone must be sufficient to
    discriminate between distinct test plans so you don't have to duplicate
    that information in the description.

    If your tests will require any special set-up (procuring external hardware,
    setting some devices or software in special test mode) it is recommended
    to include this information here.

    The field has no size limit. It can contain newline characters. This field
    should be marked as translatable by prepending the underscore character
    (\_) in front. This field is optional.

``include``:
    A multi-line list of job identifiers or patterns matching such identifiers
    that should be included for execution.

    This is the most important field in any test plan. It basically decides
    on which job definitions are selected by (included by) the test plan.
    Separate entries need to be placed on separate lines. White space does not
    separate entries as the id field may (sic!) actually include spaces.

    You have two options for selecting tests:

     - You can simply list the identifier (either partial or fully qualified)
       of the job you want to include in the test plan directly. This is very
       common and most test plans used by Checkbox actually look like that.

     - You can use regular expressions to select many tests at the same time.
       This is the only way to select generated jobs (created by template
       units). Please remember that the dot character has a special meaning so
       unless you actually want to match *any character* escape the dot with
       the backslash character (\\).

    Regardless of if you use patterns or literal job identifiers you can use
    their fully qualified name (the one that includes the namespace they reside
    in) or an abbreviated form. The abbreviated form is applicable for job
    definitions that reside in the same namespace (but not necessarily the same
    provider) as the provider that is defining the test plan.

    Plainbox will catch incorrect references to unknown jobs so you should
    be relatively safe. Have a look at the examples section below for examples
    on how you can refer to jobs from other providers (you simply use their
    fully qualified name for that)

``mandatory_include``:
    A multi-line list of job identifiers or patterns matching such identifiers
    that should always be executed.

    This optional field can be used to specify the jobs that should always run.
    This is particularly useful for specifying jobs that gather vital
    info about the tested system, as it renders impossible to generate a report
    with no information about system under test.

    For example, session results meant to be sent to the Ubuntu certification
    website must include the special job: miscellanea/submission-resources

    Example:

        mandatory_include:
            miscellanea/submission-resources

    Note that mandatory jobs will always be run first (along with their
    dependent jobs)

``bootstrap_include``:
    A multi-line list of job identifiers that should be run first, before the
    main body of testing begins. The job that should be included in the
    bootstrapping sections are the ones generating or helping to generate other
    jobs.

    Example:

        bootstrap_include:
            graphics/generator_driver_version

    Note that each entry in the bootstrap_include section must be a valid job
    identifier and cannot be a regular expression pattern.
    Also note that only resource jobs are allowed in this section.

``exclude``:
    A multi-line list of job identifiers or patterns matching such identifiers
    that should be excluded from execution.

    This optional field can be used to prevent some jobs from being selected
    for execution. It follows the similarly named  ``-x`` command line option
    to the ``plainbox run`` command.

    This field may be used when a general (broad) selection is somehow made
    by the ``include`` field and it must be trimmed down (for example, to
    prevent a specific dangerous job from running). It has the same syntax
    as the ``include``.

    When a job is both included and excluded, exclusion always takes priority.

``category-overrides``:
    A multi-line list of category override statements.

    This optional field can be used to alter the natural job definition
    category association. Currently Plainbox allows each job definition to
    associate itself with at most one category (see plainbox-category-units(7)
    and plainbox-job-units(7) for details). This is sub-optimal as some tests
    can be easily assigned equally well to two categories at the same time.

    For that reason, it may be necessary, in a particular test plan, to
    override the natural category association with one that more correctly
    reflects the purpose of a specific job definition in the context of a
    specific test plan.

    For example let's consider a job definition that tests if a specific piece
    of hardware works correctly after a suspend-resume cycle. Let's assume that
    the job definition  has a natural association with the category describing
    such hardware devices. In one test plan, this test will be associated
    with the hardware-specific category (using the natural association). In
    a special suspend-resume test plan the same job definition can
    be associated with a special suspend-resume category.

    The actual rules as to when to use category overrides and how to assign
    a natural category to a specific test is not documented here. We believe
    that each project should come up with a workflow and semantics that best
    match its users.

    The syntax of this field is a list of statements defined on separate lines.
    Each override statement has the following form::

        apply CATEGORY-IDENTIFIER to JOB-DEFINITION-PATTERN

    Both 'apply' and 'to' are literal strings. CATEGORY-IDENTIFIER is
    the identifier of a category unit. The JOB-DEFINITION-PATTERN has the
    same syntax as the ``include`` field does. That is, it can be either
    a simple string or a regular expression that is being compared to
    identifiers of all the known job definitions. The pattern can be
    either partially or fully qualified. That is, it may or may not
    include the namespace component of the job definition identifier.

    Overrides are applied in order and the last applied override is the
    effective override in a given test plan. For example, given the
    following two overrides::

        apply cat-1 to .*
        apply cat-2 to foo

    The job definition with the partial identifier ``foo`` will be associated
    with the ``cat-2`` category.

.. _testplan_estimated_duration:

``estimated_duration``:
    An approximate time to execute this test plan, in seconds.

    Since plainbox version 0.24 this field can be expressed in two formats. The
    old format, a floating point number of seconds is somewhat difficult to
    read for larger values. To avoid mistakes test designers can use the second
    format with separate sections for number of hours, minutes and seconds. The
    format, as regular expression, is ``(\d+h)?[: ]*(\d+m?)[: ]*(\d+s)?``. The
    regular expression expresses an optional number of hours, followed by the
    ``h`` character, followed by any number of spaces or ``:`` characters,
    followed by an optional number of minutes, followed by the ``m`` character,
    again followed by any number of spaces or ``:`` characters, followed by the
    number of seconds, ultimately followed by the ``s`` character.

    The values can no longer be fractional (you cannot say ``2.5m`` you need to
    say ``2m 30s``). We feel that sub-second granularity does is too
    unpredictable to be useful so that will not be supported in the future.

    This field is optional. If it is missing it is automatically computed by
    the identical field that may be specified on particular job definitions.

    Since sometimes it is easier to think in terms of test plans (they are
    typically executed more often than a specific job definition) this estimate
    may be more accurate as it doesn't include the accumulated sum of
    mis-estimates from all of the job definitions selected by a particular test
    plan.

Examples
--------

A simple test plan that selects several jobs::

    id: foo-bar-and-froz
    _name: Tests Foo, Bar and Froz
    _description:
        This example test plan selects the following three jobs:
            - Foo
            - Bar
            - Froz
    include:
        foo
        bar
        froz

A test plan that uses jobs from another provider's namespace in addition
to some of its own definitions::

    id: extended-tests
    _name: Extended Storage Tests (By Corp Inc.)
    _description:
        This test plan runs an extended set of storage tests, customized
        by the Corp Inc. corporation. In addition to the standard Ubuntu
        set of storage tests, this test plan includes the following tests::

        - Multipath I/O Tests
        - Degraded Array Recovery Tests
    include:
        com.canonical.certification:disk/.*
        multipath-io
        degrade-array-recovery

A test plan that generates jobs using bootstrap_include section::

    unit: test plan
    id: test-plan-with-bootstrapping
    _name: Tests with a bootstrapping stage
    _description:
        This test plan uses bootstrapping_include field to generate additional
        jobs depending on the output of the generator job.
    include: .*
    bootstrap_include:
        generator

    unit: job
    id: generator
    plugin: resource
    _description: Job that generates Foo and Bar resources
    command:
     echo "my_resource: Foo"
     echo
     echo "my_resource: Bar"

    unit: template
    template-unit: job
    template-resource: generator
    plugin: shell
    estimated_duration: 1
    id: generated_job_{my_resource}
    command: echo {my_resource}
    _description: Job instantiated from template that echoes {my_resource}



A test plan that marks some jobs as mandatory::

    unit: test plan
    id: test-plan-with-mandatory-jobs
    _name: Test plan with mandatory jobs
    _description:
        This test plan runs some jobs regardless of user selection.
    include:
        Foo
    mandatory_include:
        Bar

    unit: job
    id: Foo
    _name: Foo job
    _description: Job that might be deselected by the user
    plugin: shell
    command: echo Foo job

    unit: job
    id: Bar
    _name: Bar job (mandatory)
    _description: Job that should *always* run
    plugin: shell
    command: echo Bar job
