# Turbinia Local and Manual Installation Instructions

## **Introduction**

Turbinia can be run on the [Google Cloud Platform](https://cloud.google.com), on
local machines, or in a hybrid mode. See the
"[how it works](how-it-works.md)"
documentation for more details on what the architecture looks like for each of
these installation types. This page covers the local installation as well as the
manual steps for hybrid and cloud installation. **If you are setting up a GCP
cloud or hybrid installation, it is highly recommended and much simpler to use
the
[terraform installation method](install.md)
to bootstrap those environments.**

Each section of this document indicates which installation types it applies to
(cloud, hybrid or local), so you only need to follow the relevant steps for your
installation type.

### **Prerequisites**

Turbinia requires all worker nodes to have equal access to all Evidence data.
For Google Cloud this means using Google Cloud Storage (GCS), but for local and
hybrid configurations the easiest setup is to have NFS or a SAN mounted on a
common path on each worker. All output should also be written to this common
directory so that when new Evidence is generated by a Task, that the other
worker nodes can access it for processing. Turbinia can also write output to GCS
even when running locally (see the `GCS_OUTPUT_PATH` variable in the config).

## **Installation**

To run Turbinia it's recommended that you have at least two machines or cloud
instances, one for the server and one or more for workers. In a small or
development setup, you can also run both the server and worker on a single
instance.

### **GCP Setup**

**This section is required for cloud and hybrid configurations.**

*   Create or select a Google Cloud Platform project in the
    [Google Cloud Console](https://console.cloud.google.com).
*   Determine which GCP zone and region that you wish to deploy Turbinia into.
    Note that one of the GCP dependencies is Cloud Functions, and that only
    works in certain regions, so you will need to deploy in one of
    [the supported regions](https://cloud.google.com/functions/docs/locations).
*   Enable
    [Cloud Functions](https://console.cloud.google.com/apis/library/cloudfunctions.googleapis.com).
*   Follow the
    [instructions](https://cloud.google.com/pubsub/docs/quickstart-console) to:
    *   Enable
        [Cloud Pub/Sub](https://console.cloud.google.com/apis/library/pubsub.googleapis.com)
    *   Create a new Pub/Sub topic and subscription (pull type with 600s
        timeout). These can use the same base name (the part after `topics/` and
        `subscription/` in the paths).
    *   Please take a note of the topic name for the configuration steps, as
        this is what you will set the `PUBSUB_TOPIC` config variable to.
*   Enable
    [Cloud Datastore](https://console.cloud.google.com/apis/api/datastore.googleapis.com)
    *   Go to Datastore in the cloud console
    *   Hit the `Create Entity` button
    *   Select the same region that you selected in the previous steps. No need
        to create any Entities after selecting your region

#### **Create a GCE Instance for the Server**

**This section is required for cloud configurations.**

*   Create a
    [new GCE instance](https://console.cloud.google.com/compute/instances) from
    a recent version of Debian or Ubuntu (currently 18.0.4 is recommended). Once
    the host is configured we'll later clone the disk to use for the workers.
    *   This should work on other Linux flavors, but these are untested, and the
        installation steps would need to be adapted.

#### **Create a Google Cloud Storage (GCS) Bucket**

**This section is required for cloud installations.**

*   [Create a new GCS bucket](https://cloud.google.com/storage/docs/creating-buckets)
    and take note of the bucket name as this will be referenced later by the
    `GCS_OUTPUT_PATH` variable.

### **Core Installation**

**This section is required for cloud, hybrid and local configurations.**

#### **Preparation**

All steps listed below must be completed on all servers/workers unless otherwise
noted. If you are installing in a cloud environment, you will only have a server
VM at this point, and we will later clone this instance to create the workers.

*   Install dependencies
    *   `sudo apt-get install python-dev build-essential python-setuptools
        python-pip python-virtualenv liblzma-dev git john`
*   Create a turbinia user with password-less sudo access
    *   `sudo adduser --disabled-password turbinia`
    *   `echo "turbinia ALL = (root) NOPASSWD:
         /bin/mount,/bin/umount,/sbin/losetup" | sudo tee -a /etc/sudoers.d/turbinia`
*   Prepare configuration directory:
    *   `sudo mkdir /etc/turbinia`
    *   `sudo chown turbinia /etc/turbinia`
*   Checkout git at release branch for config and other setup
    *   `git clone https://github.com/google/turbinia.git`
    *   `cd turbinia`
    *   `git branch -l | grep release`
    *   `git checkout <latest release branch from previous step>`

#### **Installation and Configuration**

*   Log in as turbinia
    *   `sudo su - turbinia`
*   Install Turbinia
    *   Note: You may want to install this into a virtual-environment with venv
        or pipenv to reduce potential dependency conflicts and isolate these
        packages into their own environment.
    *   `pip3 install turbinia` for the server
    *   `pip3 install turbinia[worker]` for the worker
    *   `pip3 install turbinia[dev]` if you want to run tests or get the
        development dependencies.
    *   If you are running a local installation:
        *   `pip3 install turbinia[local]`
*   Install Worker binary dependencies
    *   You can install Plaso from the
        [GIFT PPA](https://launchpad.net/~gift/+archive/ubuntu/stable), or
        [see here](https://plaso.readthedocs.io/en/latest/sources/user/Users-Guide.html#user-s-guide)
        for other packaged installations.
    *   There are a few other binary
        dependencies that are not packaged with Ubuntu or PyPi, so these
        will need to be installed manually:
        [bulk_extractor](https://github.com/simsong/bulk_extractor/wiki/Installing-bulk_extractor),
        [hindsight](https://github.com/obsidianforensics/hindsight) (this one
        is technically in PyPi, but since it's not Python3 yet it needs to be
        installed separately) and
        [Volatility](https://github.com/volatilityfoundation/volatility/wiki/Installation).
        Alternately you can disable the Jobs that have those dependencies.
        You can do this by setting the `DISABLED_JOBS` in the
        `/etc/turbinia/turbinia.conf` config after it is installed (see below).
        You can see the list of Jobs with `turbiniactl listjobs` after everything
        is set up.
*   Create and configure the Turbinia configuration file.
    *   `cp <git clone path>/turbinia/config/turbinia_config_tmpl.py
        /etc/turbinia/turbinia.conf`
    *   Alternately, you can either put the file in `/home/$USER/.turbiniarc` or
        in another directory and then point the `TURBINIA_CONFIG_PATH`
        environment variable to that directory.
    *   Edit the config file to match your local installation details.
        *   Match the `PUBSUB_TOPIC` variable in the configuration to the name
            of the topic you created in GCP.
        *   If you are running Turbinia locally, make sure to set
            `GCS_OUTPUT_PATH` to `None`.
        *   For local and hybrid installations:
            *   Set `SHARED_FILESYSTEM = True`
        *   For local installations:
            *   Set `STATE_MANAGER = 'Redis'`
            *   Set `TASK_MANAGER = 'Celery'`
            *   Configure the `CELERY*`, `KOMBU*` and `REDIS*` variables as
                appropriate for your config.
        *   Set the following
    *   Configure the `OUTPUT_DIR`, `TMP_DIR`, and `MOUNT_DIR_PREFIX` to match
        your local system. On the worker nodes, create the corresponding
        directories and make sure they are owned by the turbinia user.
*   Configure the init scripts to run Turbinia on start
    *   `cp <git clone path>/turbinia/tools/turbinia@.service
        /etc/systemd/system/turbinia@server` for the server
    *   `cp <git clone path>/turbinia/tools/turbinia@.service
        /etc/systemd/system/turbinia@psqworker` for a GCP worker
    *   `cp <git clone path>/turbinia/tools/turbinia@.service
        /etc/systemd/system/turbinia@celeryworker` for a local (non-cloud)
        installation.
    *   Follow the instructions at the top of the
        `turbinia/tools/turbinia@.service` file to enable these services.

### **GCP Installation**

**This section is required for cloud and hybrid configurations.**

*   Install
    [google-cloud-sdk](https://cloud.google.com/sdk/docs/quickstart-linux)
*   Create a
    [scoped service account](https://cloud.google.com/compute/docs/access/service-accounts)
    (this is the best option) with the following roles:
    *   `Cloud Datastore User`: Used by PSQ to store result data, and in the
        future by the Task Manager to store queryable task data
    *   `Pub/Sub Editor`: Used by clients to talk to Turbinia, and by the Task
        Manager to talk to workers
    *   `Storage Object Admin` and `Storage Legacy Bucket Reader`: Only required
        on the GCS bucket used by Turbinia, if any. See
        [GCP Turbinia](install.md)
        for details
    *   `Compute Instance Admin`: Used to list instances and to attach disks to
        instances
    *   `Service Account User`: Used when attaching disks
    *   `Cloud Functions Developer`: Used by turbiniactl to query task status
*   Create a new key for your service account, and save it on server/workers and
    then configure init scripts in `/etc/systemd/system/turbinia*` to point to
    it by setting the `GOOGLE_APPLICATION_CREDENTIALS` var similar to
    `ExecStartPre=+/bin/sh -c '/bin/systemctl set-environment
    GOOGLE_APPLICATION_CREDENTIALS="/home/turbinia/turbinia-service-account-creds.json"'`
*   Add the service account to the gcloud auth
    *   `gcloud auth list`
    *   `gcloud auth activate-service-account
        --key-file=$GOOGLE_APPLICATION_CREDENTIALS`
*   Alternately you can run Turbinia under your own credentials (NOT RECOMMENDED
    except for development environments)
    *   Run `gcloud auth login` (may require you to copy/paste url to browser)
    *   Run `gcloud auth application-default login`

#### **Create GCP workers**

**This section is required for cloud configurations.**

*   Stop the server instance that has been configured above.
*   Create a new image from the server VM's disk.
*   Create a new Instance Template using the newly created image.
*   Create a new Managed Instance Group from the newly created Instance
    Template.

#### **Deploy Cloud Functions**

**This section is required for cloud and hybrid configurations.**

*   `cd <git clone path>/tools/gcf_init && ./deploy_gcf.py`

## **Local Turbinia**

Running Turbinia locally using Docker. This setup does not require Google 
Cloud Platform.

See [here](https://github.com/google/turbinia/tree/master/docker/local/) for detailed instructions.
