:title: Project Testing Interface

.. _pti:

Project Testing Interface
=========================

The following sections describe an example PTI (Project Testing Interface)
implementation. The goal is to setup a consistent interface for driving tests
and other necessary tasks to successfully configure :ref:`project_gating` within
your organization.


Projects layout
---------------

A proper PTI needs at least two projects:

* org-config: a :term:`config-project` curated by administrators, and
* org-jobs: a :term:`untrusted-project` to hold common jobs.

The projects that are being tested or deployed are also
:term:`untrusted-project`, and for the purpose of this example we will use a
couple of integrated projects named:

* org-server
* org-client


org-config
~~~~~~~~~~

The config project needs careful scrutiny as it defines privileged Zuul
configurations that are shared by all your projects:

:ref:`pipeline` triggers and requirements let you define when a change is
tested and what are the conditions for merging code. Approval from core
members or special labels to indicate a change is good to go are pipelines
configuration.

The base job let you define how the test environment is validated before
the actual job is executed. The base job also defines how and where the job
artifacts are stored.

More importantly, a config-project may enforce a set of integration jobs to
be executed on behalf of the other projects. A regular (untrusted-project) can
only manage its own configuration, and as part of a PTI implementation, you
want to ensure your projects' changes undergo validation that are defined
globally by your organization.

Because the nature of those configuration settings are privileged,
config-projects changes are only effective when merged.


org-jobs
~~~~~~~~

Jobs definition content are not privileged Zuul settings and jobs can be
defined in a regular :term:`untrusted-project`.
As a matter of fact, it is recommended to define jobs outside of the config
project so that job updates can be tested before being merged.

In this example, we are using a dedicated org-jobs project.


Projects content
----------------

In this example PTI, the organization requirements are a consistent code style
and an integration test to validate org-client and org-server works according
to a reference implementation.

In the org-jobs project, we define a couple of jobs:

.. code-block:: yaml

  # org-jobs/zuul.yaml
  - job:
      name: org-codestyle
      parent: run-test-command
      vars:
        test_command: $code-style-tool $org-check-argument
        # e.g.: linters --max-column 120

  - job:
      name: org-integration-test
      run: integration-tests.yaml
      required-projects:
        - org-server
        - org-client

The integration-tests.yaml playbook needs to implement an integration test
that checks both the server and client code.


In the org-config project, we define a project template:

.. code-block:: yaml

  # org-config/zuul.d/pti.yaml
  - project-template:
      name: org-pti
      queue: integrated
      check:
        jobs:
          - org-codestyle
          - org-integration-test
      gate:
        jobs:
          - org-codestyle
          - org-integration-test


Finally, in the org-config project, we setup the PTI template on both projects:

.. code-block:: yaml

  # org-config/zuul.d/projects.yaml
  - project:
      name: org-server
      templates:
        - org-pti

  - project:
      name: org-client
      templates:
        - org-pti


Usage
-----

With the above layout, the organization projects use a consistent testing
interface.
The org-client or org-server does not need extra settings, all new
contribution shall pass the codestyle and integration-test as defined by
the organization admin.


Project tests
~~~~~~~~~~~~~

Projects may add extra jobs on top of the PTI.
For example, the org-client project can add a user interface test:

.. code-block:: yaml

  # org-client/.zuul.yaml
  - job:
      name: org-client-ui-validation

  - project:
      check:
        jobs:
          - org-client-ui-validation
      gate:
        jobs:
          - org-client-ui-validation

In this example, new org-client change will run the PTI's jobs as well as the
org-client-ui-validation job.


Updating PTI test
~~~~~~~~~~~~~~~~~

Once the PTI is in place, if a project needs adjustment,
it can proceed as follow:

First a change on org-jobs is proposed to modify a job. For example, update a
codestyle check using such commit:

.. code-block:: text

  # org-jobs/change-url

  Update codestyle to enforce CamelCase.

Then, without merging this proposal, it can be tested across the projects using
such commit:

.. code-block:: text

  # org-client/change-url

  Validate new codestyle.
  Depends-On: org-jobs/change-url

Lastly the org-jobs may be enriched with:

.. code-block:: text

  # org-jobs/change-url

  Update codestyle to enforce CamelCase.
  Needed-By: org-client/change-url


.. note:: Extra care is required when updating PTI jobs as they affects all
          the projects. Ideally, the org-jobs project would use a org-jobs-check
          to run PTI jobs change on every projects.


Cross project gating
--------------------

The org-pti template is using the "integrated" queue to ensure projects change
are gated by the zuul scheduler. Though, the jobs need extra care to properly
test projects as they are prepared by Zuul. For example, the
org-integration-test playbook need to ensure the client and server are installed
from the zuul src_root.

This is called sibling installation, and it is a critical piece to ensure cross
project gating.
