Tests for tlsmate
#################

Most of the tests are executed against TLS servers running on the local host,
i.e., internet connectivity is not required (but it is required to build and
setup the used TLS server libraries).

To use the test environment and generate additional test cases first the CAs
and the TLS libraries to test against must be setup.

CA environment
==============

Everything related to certificate authorities is located under the directory
``ca/``. The CA (or better: the CAs) has been setup using basic Openssl
commands.

To reset everything to an initial state the following command can be used (from
the toplevel ``tlsmate`` directory)::

    $ (cd ca && make clean)

.. warning::
    The command above will delete all certificates, the associated keys and all
   CA related states, like CRLs, etc. Generating new test cases will not be
   possible until the CA is newly setup from scratch. But executing existing
   test cases is still possible.

Building the CA from scratch is done via the following command::

    $ (cd ca && make)

This will generate two different sets of certificate chains, one primarily
based on RSA, and another one based on elliptic curves.

RSA based certificates:
    - a root CA using an RSA key
    - an intermediate CA using an RSA key signed by the root CA above
    - multiple server certificates signed by the CA above:
        - RSA server key certificate
        - DSA server key certificate
        - revoked RSA key certificate
    - a client certificate using an RSA key signed by the CA above

Elliptic curve based:
    - a root CA using an ECDSA key
    - an intermediate CA using an ECDSA key signed by the root CA above
    - multiple server certificates signed by the CA above:
        - ECDSA server key certificate
        - Ed25519 server key certificate
        - Ed488 server key certificate
    - a client certificate using an ECDSA key signed by the CA above

In addition, for the RSA based intermediate CA a CRL is generated.

Each certificate has a name (``<name>`` in the following), for instance
``server-rsa``. The directory structure under ``ca/`` is now as follows (all
files are in PEM format):

Private keys are stored in ``ca/private/<name>.key``.

Certificates are stored in ``ca/certs/<name>.crt``.

Chains are stored in ``ca/chains/<name>.chn``. Chains are only relevant for
server and client certificates. The semantic here differs: For server
certificates, the chain contains additional certificates which are delivered by
the server in a Certificate message. It is suffient to include here only the CA
certificate, as root certificates do not need to be exchanged.

For client certificates, the chain contains all certificates offered by the
client in a Certificate message, i.e., the client certificate must be included
here as well. That sounds odd, but server certificates are handled by the TLS
server library, while client certificates are handled by ``tlsmate``.

The file ``ca/certs/ca-certificates.pem`` represents the trust store.

CRLs are available in ``ca/crl/<name>.crl`` (binary format) and in
``ca/crl/<name>.pem`` (PEM format).

When generating the CAs with the ``make`` command above, the following
environment variables are taken into account:

- ``TLSMATE_CA_PORT``: the port number for downloading a CRL. Defaults to 44400.
  All CRLs are provided by the same web server, with its document root pointing
  to ``ca/crl``. The related address is http://crl.localhost:$TLSMATE_CA_PORT.
  Note, that this URL including the port number will be stored in the
  certificates.
- ``TLSMATE_CA_RSA_OCSP_PORT``: The port number of the OCSP responder for ca-rsa.
  Defaults to 44401.
- ``TLSMATE_CA_ECDSA_OCSP_PORT``: The port number of the OCSP responder for
  ca-ecdsa. Defaults to 44402. For both reponders the URL is
  http://ocsp.localhost:<port>. Note, that the responder URL will be stored in
  the certificates.

The environment variables are useful in case the default port numbers cannot be
used.

TLS library versions
====================

Currently the following libraries are used:

- Openssl, v1.0.1e (vulnerable to heartbleed)
- Openssl, v1.0.1g (vulnerable to CCS-injection)
- Openssl, v1.0.2u
- Openssl, v1.1.1k
- Openssl, v3.0.0-alpha16
- wolfssl, v3.12.0 (vulnerable to ROBOT attack)

They must be downloaded and built, but this should be rather easy::

    $ (cd tlslibraries && make)

The Makefile will take care of downloading the required version, and all
libraries are statically linked to simplify their usage lateron.

To get rid of all those libraries just do::

    $ (cd tlslibraries && make clean)

Test setup
==========

Let's take an example: the test case is stored in
``tests/tls12/test_tls12_Heartbeat.py``. Now the test case can be generated by
executing::

    $ python tests/tls12/test_tls12_Heartbeat.py

This does the following things:
    - it starts a mini-web server to serve CRL downloads on localhost
    - it starts OCSP responders on localhost
    - it starts a TLS server on localhost with the library version required for
      the test case and with parameters defined in the test case.
    - it waits a little bit until all those services are well up and running
    - it executes its main TLS scenario (which can be a single handshake or a
      complete TLS server scan) against the previously started TLS server
    - during the execution of the test case all external interfaces are
      monitored by the recorder module, e.g., all messages sent and received
      are stored. Well, that's just half the truth. For the full details
      refer to the code. :-)
    - after the test scenario is finished, the recorded messages (and other
      stuff as well) is serialized to the file
      ``tests/tls12/recordings/Heartbeat.yaml``.

Now this recorded test case can be played back without involving any servers.
It means this is the real execution of a test case. Just use::

    $ pytest tests/tls12/test_tls12_Heartbeat.py

This will now do the following:
    - the YAML file ``tests/tls12/recordings/Heartbeat.yaml`` is deserialized
      by the recorder module
    - the main test scenario is executed, and all external interfaces are
      served by the recorder: sent messages are compared to the previously
      recorded message in the recorder module. For messages awaited in the test
      scenario the recorder will provide the previously recorded message. This
      way a bit-by-bit clone of the recorded test scenario is executed. Any
      deviation from the recorded scenario will fail the test case.

Now these test recorder files can be generated by using a Makefile. For
example, to remove all recorder files for TLS1.2 just do::

    $ (cd tests/tls12 && make clean)

Regeneration of these files can be done by::

    $ (cd tests/tls12 && make)

Test execution
==============

To execute all test cases, simply run (from the top level ``tlsmate``
directory)::

    $ make test

This will run the tests against the current python version. To test against
other versions as well, just use::

    $ make test-all
