Tests
=====

The ``tests`` directory is mainly intended for people who want to
contribute to IVRE and want to make sure their changes do not break
IVRE.

Dependencies
------------

To run IVRE tests you will need `coverage.py
<http://nedbatchelder.com/code/coverage/>`_.

Test case
---------

The first thing is to find samples. You need both (recent) Nmap XML scan
result files (or Nmap JSON files, as generated by
``ivre scancli --json``) and PCAP files (dump while you browse, and
browse a lot, or sniff a busy open Wi-Fi network, if that's legal in
your country).

A good test case should have a lot of various data (sniff from different
places, scan different hosts with different Nmap options).

It is mandatory to have at least, for the Nmap test:

-  Two scanned (and up) hosts with different IP addresses
-  One host scanned with the script ``http-robots.txt`` reporting
   ``/cgi-bin`` in its output.
-  One host scanned with an anonymous FTP server.
-  One scan result with traceroute and at least one hop with a hostname.
-  One host scanned with a hostname ending with ".com".

For the passive test:

-  Two records with different IP addresses.
-  One SSL certificate.

First run
---------

From the ``tests`` directory, create the ``samples`` subdirectory and
place your samples there (the PCAP files must have the extension
``.pcap``, the Nmap XML result files must have the extension ``.xml``,
and the Nmap JSON results must have the extension ``.json``).

Then, run ``python ./tests.py`` (optionally replace ``python`` by the
alternative interpreter you want to use, e.g., ``python2.7``; note
that ``coverage.py`` must be installed for this interpreter).  The
first run will create a ``samples/results`` file with the expected
values for some results. The next runs will use those values to check
if something has been broken.

For this reason, it is important to:

-  Run the tests for the first time with a "known-working" version.
-  Remove the file ``samples/results`` whenever a sample file is added,
   modified or removed.

Improving the test case
-----------------------

If you want to make sure to have enough samples, you can:

-  Check the ``samples/results`` file for ``*_count`` entries with low
   values (particularly 0, of course) and find or create new samples
   that will improve those values.
-  Check the report generated by coverage.py under the ``htmlcov``
   directory, and check whether your current test case covers at least
   the code you want to change.

Failures
--------

Tests failure are not always an issue. Apart from a new bug, of course,
here are some reasons that can explain test failures:

-  You have added new samples and have not removed the
   ``samples/results`` file.
-  Your samples do not match the minimum requirements detailed above.
-  A new feature has been added to IVRE and the new results are actually
   better than the stored ones.

Travis CI
---------

Tests are run with several MongoDB and PostgreSQL versions, as well as
TinyDB, SQLite and Elasticsearch for each pull requests. The tests run
with Python 2.7 and 3.4 to 3.8.

The configuration is in the `.travis.yml` YAML file.
