Installation guidelines — IVRE documentation

# Installation guidelines

## Database

Depending on the backends you wan to use, install a database server. Please keep in mind that currently, MongoDB is currently the only supported backend for all the purposes. To learn more about the different purposes, read the Principles.

The database servers installation and setup is not covered here, and depends on your platform and needs. Please refer to the server documentation on how to install it. For MongoDB you can read the installation section of their documentation.

## Dependencies

### External programs

If you plan to run scans from a machine, install Nmap, Masscan, and/or Zmap / Zgrab / Zgrab2. If you want to integrate screenshots, install Tesseract, ImageMagick, FFmpeg and PhantomJS.

If you plan to analyze PCAP file on a machine, install, depending on your needs:

- Zeek (previously known as Bro, version 3 minimum).

- Argus.

- Nfdump.

### Python

To install IVRE, you’ll need Python 3.7 minimum, with the following modules:

- bottle.

- cryptography.

- pymongo version 3.7 minimum.

- tinydb, to use the experimental TinyDB backend (this does not require a database server).

- sqlalchemy and psycopg2 to use the experimental PostgreSQL backend.

- elasticsearch and elasticsearch-dsl to use the experimental Elasticsearch backend.

- PIL optional, to trim screenshots.

- pyOpenSSL version 16.1.0 minimum, optional, to parse X509 certificates (a fallback exists that calls `Popen()` the `openssl` binary and parses its output, but it is much slower and less reliable).

### Databases

IVRE’s reference backend service is MongoDB, version 3.6 minimum. It is highly suggested that you use the latest stable release (the performances tend to improve a lot).

The `passive`, `nmap` and `view` purposes have an experimental PostgreSQL backend that can be used in lieu of MongoDB.

The `view` purpose has an experimental Elasticsearch backend. It can be used to create views accessible to other Elasticsearch tools, such as Kibana (see IVRE with Kibana).

Please refer to the database servers (or your distribution) documentation on how to install and configure them.

### Web

For production services, it is recommended to install either Apache with the WSGI module, or Nginx with uWSGI.

IVRE can use Dokuwiki as its notepad, it is also recommended to install it.

Please refer to the servers (or your distribution) documentation on how to install and configure them.

Configuration file samples are provided in IVRE’s source repository, under `pkg/apache` and `pkg/nginx`. Also, the Docker creation files in `docker/web` and `docker/web-apache` can provide useful examples.

If you do not want (or cannot) to install a Web server, you can try IVRE’s integrated server, suited for tests or tiny installations. Just run `ivre httpd`!

## IVRE

The installation of IVRE itself can be done:

- On Kali, just install the package by running `apt update && apt install ivre`. You can also install `ivre-doc` if needed.

- On Fedora, you can use the Copr package; follow the instructions.

- On other RPM-based Linux distributions, you can easily build RPM packages (using the provided `pkg/buildrpm` script, or use the `setup.py` script with your own options).

- On Arch Linux, there are AUR packages that can be installed using yay for example. The packages are:

  > - `ivre`: the main package, which depends on `python-ivre`.

  > - `python-ivre` the Python library.

  > - `ivre-web`: the Web application.

  > - `ivre-docs`: the documentation.

  These packages are based on the latest stable version; they all have a `-git` version, based on the current development code from the Github repository. You can install for example `ivre-git` and `ivre-web-git` if you want to test the latest developments.

  All the packages are based on the same bases: ivre and ivre-git.

- On BlackArch Linux (an Arch Linux-based penetration testing distribution) IVRE is packaged (and installed in the Live ISO).

- Using pip: run `pip install ivre` (this will download and install for you the IVRE package and its Python dependencies from PyPI, the Python Package Index).

- From the source code, using the `setup.py` (classical `./setup.py build; sudo ./setup.py install`) script.

- Using Docker (in this case you do not need to follow the instructions in Configuration, as the Docker containers are already configured).

## Configuration

You can set configuration values in several files:

- system-wide: `ivre.conf` in the following directories: `/etc/`, `/etc/ivre`, `/usr/local/etc`, `/usr/local/etc/ivre`.

- user-specific: `~/.ivre.conf` (read after the system-wide configuration files, so highest priority).

- execution-specific: another configuration file can be specified using the `$IVRE_CONF` environment variable.

The configuration files are Python files. They may set, for example, the variable `DB` to use a different database than the default one.

See Configuration to learn more about the different configuration parameters.

## Initialization

Once IVRE has been properly configured, it’s time to initialize its databases.

For that, the command-line tools (namely `ivre ipinfo`, `ivre scancli`, `ivre view`, `ivre flowcli` and `ivre runscansagentdb`, respectively for information about IP addresses, passive information, active information and running scans through agents) have a `--init` option.

So you can run, with a user or from a host where the configuration has a write access to the database (add `< /dev/null` to skip the confirmation):

    $ yes | ivre ipinfo --init
    $ yes | ivre scancli --init
    $ yes | ivre view --init
    $ yes | ivre flowcli --init
    $ yes | sudo ivre runscansagentdb --init

## Getting IP data

To fetch the IP address data files (mainly from Maxmind) and parse them (required if you want to scan or list all IP addresses from a country or an AS), just run the following command (it takes a long time, usually more than 40 minutes on a decent server):

    $ sudo ivre ipdata --download

It is advised to run this command on a regular basis (e.g., weekly). If you use IVRE on several machines, you may want to run the command on one machine and create an `ivre-data` package containing the files under the `/usr/share/ivre/geoip` directory (or distribute those files somehow).

The URLs downloaded are stored in the configuration. By default, the following files are downloaded:

    $ python
    >>> from ivre.config import IPDATA_URLS
    >>> for fname, url in IPDATA_URLS.items():
    ...     print("%s: %s" % (fname, url))
    ...
    GeoLite2-City.tar.gz: https://ivre.rocks/data/geolite/GeoLite2-City.tar.gz
    GeoLite2-City-CSV.zip: https://ivre.rocks/data/geolite/GeoLite2-City-CSV.zip
    GeoLite2-Country.tar.gz: https://ivre.rocks/data/geolite/GeoLite2-Country.tar.gz
    GeoLite2-Country-CSV.zip: https://ivre.rocks/data/geolite/GeoLite2-Country-CSV.zip
    GeoLite2-ASN.tar.gz: https://ivre.rocks/data/geolite/GeoLite2-ASN.tar.gz
    GeoLite2-ASN-CSV.zip: https://ivre.rocks/data/geolite/GeoLite2-ASN-CSV.zip
    GeoLite2-dumps.tar.gz: https://ivre.rocks/data/geolite/GeoLite2-dumps.tar.gz
    iso3166.csv: https://dev.maxmind.com/static/csv/codes/iso3166.csv
    BGP.raw: https://thyme.apnic.net/current/data-raw-table

## Using Agents

If you do not plan to run active scans with remote agents (where IVRE will not be installed), you can skip this section.

The agent does not require IVRE to be installed. It is a script that needs to be adapted to each situation.

The agent is only needed when you cannot install IVRE on the machine used to scan or when you want to use several machines to run one scan.

It requires a POSIX environment, and the commands `screen`, `rsync` and `nmap` (of course). See the Agents documentation for more information about that.