Metadata-Version: 2.1
Name: pybeagle
Version: 1.0.1
Summary: Beagle is an incident response and digital forensics tool which transforms data sources and logs into graphs
Home-page: https://github.com/yampelo/beagle
Author: yampelo
License: MIT
Platform: UNKNOWN
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.6
Classifier: Topic :: Software Development :: Libraries
Classifier: Topic :: Security
Classifier: Development Status :: 5 - Production/Stable
Classifier: Environment :: Web Environment
Classifier: Operating System :: OS Independent
Requires-Python: >=3.6.0,<3.7
Description-Content-Type: text/markdown
Requires-Dist: acora (==2.0)
Requires-Dist: aff4-snappy (==0.5.1)
Requires-Dist: ansimarkup (==1.4.0)
Requires-Dist: arrow (==0.10.0)
Requires-Dist: artifacts (==20170909)
Requires-Dist: atomicwrites (==1.3.0)
Requires-Dist: attrs (==19.1.0)
Requires-Dist: backcall (==0.1.0)
Requires-Dist: better-exceptions-fork (==0.2.1.post6)
Requires-Dist: certifi (==2019.3.9)
Requires-Dist: chardet (==3.0.4)
Requires-Dist: click (==7.0)
Requires-Dist: colorama (==0.4.1)
Requires-Dist: coverage (==5.0a4)
Requires-Dist: decorator (==4.4.0)
Requires-Dist: expiringdict (==1.1.4)
Requires-Dist: filelock (==2.0.6)
Requires-Dist: flask-sqlalchemy (==2.3.2)
Requires-Dist: flask (==1.0.2)
Requires-Dist: future (==0.16.0)
Requires-Dist: graphistry[networkx] (==1.0a11)
Requires-Dist: grpcio (==1.19.0)
Requires-Dist: gunicorn (==19.9.0)
Requires-Dist: hexdump (==3.3)
Requires-Dist: html5lib (==1.0.1)
Requires-Dist: httplib2 (==0.9.2)
Requires-Dist: idna (==2.5)
Requires-Dist: intervaltree (==2.1.0)
Requires-Dist: ipaddr (==2.2.0)
Requires-Dist: ipython-genutils (==0.2.0)
Requires-Dist: ipython (==6.5.0)
Requires-Dist: isodate (==0.6.0)
Requires-Dist: itsdangerous (==1.1.0)
Requires-Dist: jedi (==0.13.3)
Requires-Dist: jinja2 (==2.10)
Requires-Dist: loguru (==0.2.5)
Requires-Dist: lxml (==4.3.2)
Requires-Dist: markupsafe (==1.1.1)
Requires-Dist: neo4j (==1.7.2)
Requires-Dist: neobolt (==1.7.4)
Requires-Dist: neotime (==1.7.4)
Requires-Dist: networkx (==2.2)
Requires-Dist: numpy (==1.16.2)
Requires-Dist: oauth2client (==3.0.0)
Requires-Dist: pandas (==0.24.2)
Requires-Dist: parsedatetime (==2.4)
Requires-Dist: parso (==0.3.4)
Requires-Dist: pathlib (==1.0.1)
Requires-Dist: pickleshare (==0.7.5)
Requires-Dist: pluggy (==0.9.0)
Requires-Dist: portpicker (==1.1.1)
Requires-Dist: prompt-toolkit (==1.0.15)
Requires-Dist: protobuf (==3.6.1)
Requires-Dist: psutil (==5.6.1)
Requires-Dist: ptyprocess (==0.6.0)
Requires-Dist: py (==1.8.0)
Requires-Dist: pyaff4 (==0.26.post6)
Requires-Dist: pyarrow (==0.12.1)
Requires-Dist: pyasn1-modules (==0.2.4)
Requires-Dist: pyasn1 (==0.4.5)
Requires-Dist: pyblake2 (==0.9.3)
Requires-Dist: pycryptodome (==3.4.7)
Requires-Dist: pydgraph (==1.0.3)
Requires-Dist: pyelftools (==0.24)
Requires-Dist: pygments (==2.3.1)
Requires-Dist: pyparsing (==2.1.5)
Requires-Dist: pytest-cov (==2.6.1)
Requires-Dist: pytest (==4.3.1)
Requires-Dist: python-dateutil (==2.6.1)
Requires-Dist: python-evtx (==0.6.1)
Requires-Dist: pytsk3 (==20170802)
Requires-Dist: pytz (==2017.3)
Requires-Dist: pyyaml (==3.12)
Requires-Dist: rdflib[sparql] (==4.2.2)
Requires-Dist: readline (==6.2.4.1)
Requires-Dist: rekall-agent (==1.7.1)
Requires-Dist: rekall-capstone (==3.0.5.post2)
Requires-Dist: rekall-core (==1.7.2rc1)
Requires-Dist: rekall-efilter (==1.6.0)
Requires-Dist: rekall-lib (==1.7.2rc1)
Requires-Dist: rekall-yara (==3.6.3.1)
Requires-Dist: rekall (==1.7.2rc1)
Requires-Dist: requests (==2.18.1)
Requires-Dist: rsa (==4.0)
Requires-Dist: simplegeneric (==0.8.1)
Requires-Dist: six (==1.12.0)
Requires-Dist: sortedcontainers (==1.5.7)
Requires-Dist: sparqlwrapper (==1.8.2)
Requires-Dist: sqlalchemy (==1.3.1)
Requires-Dist: sseclient (==0.0.18)
Requires-Dist: traitlets (==4.3.2)
Requires-Dist: urllib3 (==1.21.1)
Requires-Dist: wcwidth (==0.1.7)
Requires-Dist: webencodings (==0.5.1)
Requires-Dist: werkzeug (==0.15.1)
Requires-Dist: more-itertools (==6.0.0) ; python_version > "2.7"
Requires-Dist: pexpect (==4.6.0) ; sys_platform != "win32"
Requires-Dist: appnope (==0.1.0) ; sys_platform == "darwin"


# Beagle

[![Build Status](https://travis-ci.com/yampelo/beagle.svg?token=ecmfGYY44tZk69KamLJx&branch=master)](https://travis-ci.com/yampelo/beagle) [![Read The Docs](https://readthedocs.org/projects/beagle-graphs/badge/?version=latest)](https://beagle-graphs.readthedocs.io/en/latest/?badge=latest) [![Docker](https://img.shields.io/docker/build/yampelo/beagle.svg)](https://hub.docker.com/r/yampelo/beagle)

<!-- @import "[TOC]" {cmd="toc" depthFrom=2 depthTo=4 orderedList=true} -->

<!-- code_chunk_output -->

1. [About Beagle](#about-beagle)
2. [Installation](#installation)
    1. [Docker](#docker)
    2. [Python Package](#python-package)
    3. [Configuration](#configuration)
3. [Web Interface](#web-interface)
    1. [Uploading Data](#uploading-data)
    2. [Browsing Existing Graphs](#browsing-existing-graphs)
    3. [Graph Interface](#graph-interface)
        1. [Inspecting Nodes and Edges](#inspecting-nodes-and-edges)
        2. [Expanding Neighbours](#expanding-neighbours)
        3. [Hiding Nodes](#hiding-nodes)
        4. [Running Mutators](#running-mutators)
        5. [Toggling Node and Edge Types](#toggling-node-and-edge-types)
        6. [Undo/Redo Action and Reset](#undoredo-action-and-reset)
        7. [Graph Perspectives](#graph-perspectives)
4. [Python Library](#python-library)
5. [Documentation](#documentation)

<!-- /code_chunk_output -->

## About Beagle

Beagle is an incident response and digital forensics tool which transforms data sources and logs into graphs. Supported data sources include FireEye HX Triages, Windows EVTX files, SysMon logs and Raw Windows memory images. The resulting Graphs can be sent to graph databases such as Neo4J or DGraph, or they can be kept locally as Python `NetworkX` objects.

Beagle can be used directly as a python library, or through a provided web interface.

<center>
<img src ="docs/imgs/upload_to_graph.gif">
</center>

The library can be used either as a sequence of functional calls.

```python
>>> from beagle.backends import NetworkX
>>> from beagle.datasources import SysmonEVTX

>>> graph = SysmonEVTX("malicious.evtx")
    .to_transformer()
    .to_graph(backend=NetworkX)
>>> graph
<networkx.classes.multidigraph.MultiDiGraph at 0x12700ee10>
```

Or by strictly calling each intermediate step of the data source to graph process.

```python
>>> from beagle.backends import NetworkX
>>> from beagle.datasources import SysmonEVTX
>>> from beagle.transformers import SysmonTransformer

>>> datasource = SysmonEVTX("malicious.evtx")

# Transformers take a datasource, and transform each event
# into a tuple of one or more nodes.
>>> transformer = SysmonTransformer(datasource=datasource)
>>> nodes = transformer.run()

# Transformers output an array of nodes.
[
    (<SysMonProc> process_guid="{0ad3e319-0c16-59c8-0000-0010d47d0000}"),
    (<File> host="DESKTOP-2C3IQHO" full_path="C:\Windows\System32\services.exe"),
    ...
]

# Backends take the nodes, and transform them into graphs
>>> backend = NetworkX(nodes=nodes)
>>> G = backend.graph()
<networkx.classes.multidigraph.MultiDiGraph at 0x126b887f0>
```

Graphs are centered around the activity of individual **processes**, and are meant primarly to help analysts investigate activity on hosts, not between them.

## Installation

### Docker

Beagle is available as a docker file:

```bash
docker pull yampelo/beagle
mkdir -p data/beagle
docker run -v "data/beagle":"/data/beagle" -p 8000:8000 beagle
```

### Python Package

It is also available as library. Full API Documentation is available on [https://beagle-graphs.readthedocs.io](https://beagle-graphs.readthedocs.io)

```
pip install pybeagle
```

### Configuration

-   [Complete overview of each configuration entry](docs/configuration.md)

Any entry in the [configuration file](https://github.com/yampelo/beagle/blob/master/config/beagle_default.cfg) can be modified using enviroment variables that follow the following format; `BEAGLE__{SECTION}__{KEY}`. For example, in order to change the VirusTotal API Key used when using the docker image, you would use `-e` parameter and set the `BEAGLE__VIRUSTOTAL__API_KEY` variable:

```bash
docker run -v "data/beagle":"/data/beagle" -p 8080:8080 -e "BEAGLE__VIRUSTOTAL__API_KEY=$API_KEY" beagle
```

Enviroment variables and directories can be easily defined using docker compose

```docker
version: "3"

services:
    beagle:
        image: yampelo/beagle
        volumes:
            - /data/beagle:/data/beagle
        ports:
            - "8080:8000"
        environment:
            - BEAGLE__VIRUSTOTAL__API_KEY=$key$
```

## Web Interface

Beagle's docker image comes with a web interface that wraps around the process of both transforming data into graphs, as well as using them to investigate data.

### Uploading Data

<center>
<img src ="docs/imgs/web_upload_dropdown.gif">
</center>

The upload form wraps around the graph creation process, and automatically uses `NetworkX` as the backend. Depending on the parameters required by the data source, the form will either prompt for a file upload, or text input. For example:

-   VT API Sandbox Report asks for the hash to graph.
-   FireEye HX requires the HX triage.

Any graph created is stored locally in the folder defined under the `dir` key from the `storage` section in the configuration. This can be modified by setting the `BEAGLE__STORAGE__DIR` enviroment variable.

Optionally, a comment can be added to any graph to better help describe it.

Each data source will automatically extract metadata from the provided parameter. The metadata and comment are visible later on when viewing the existing graphs of the datasource.

### Browsing Existing Graphs

Clicking on a datasource on the sidebar renderes a table of all parsed graphs for that datasource.

<center>
<img src="docs/imgs/existing_graphs_page.png">
</center>

### Graph Interface

Viewing a graph in Beagle provides a web interface that allows analysts to quickly pivot around an incident.

The interface is split into two main parts, the left part which contains various perspectives of the graph (Graph, Tree, Table, etc), and the right part which allows you to filter nodes and edges by type, search for nodes, and expand a nodes properties. It also allows you to undo and redo operations you perform on the graph.

Any element in the graph that has a divider above it is collapasble:

<center>
<img src="docs/imgs/collapsable.gif">
</center>

#### Inspecting Nodes and Edges

Nodes in the graph display the first 15 characters of their a specific field. For example, for a process node, this will be the process name.

Edges simply show the edge type.

A **single** click on a node or edge will focus that node and display its information in the "Node Info" panel on the right sidebar.

###### Focusing on a Node

<center>
<img src="docs/imgs/node_select.gif">
</center>

###### Focusing on an Edge

<center>
<img src="docs/imgs/edge_select.gif">
</center>

#### Expanding Neighbours

A **double click** on a node will pull in any neighbouring nodes. A neighbouring node is any node connected to the clicked on node by an edge. If there are no neighbors to be pulled in, no change will be seen in the graph.

-   This is regardless of direction. That means that a parent process or a child process could be pulled in when double clicking on a node.
-   Beagle will only pull in **25** nodes at a time.

<center>
<img src="docs/imgs/double_click_neighbours.gif">
</center>

#### Hiding Nodes

A **long single click** on a node will hide it from the graph, as well as any edges that depend on it.

<center>
<img src="docs/imgs/hiding_node.gif">
</center>

#### Running Mutators

Right clicking on a node exposes a context menu that allows you to run [graph mutators](docs/mutators.md). Mutators are functions which take the graph state, and return a new state.

Two extremely useful mutators are:

1. Backtracking a node: Find the seqeunce of nodes and edges that led to the creation of this node.
    - Backtracking a process node will show its process tree.
2. Expanding all descendants: From the current node, show every node that has this node as an ancestor.
    - Expanding a process node will show every child process node it spawned, any file it may have touched, and pretty much every activity that happened as a result of this node.

###### Backtracking a node

Backtracking a node is extremely useful, and is similar to doing a root cause infection in log files.

<center>
<img src="docs/imgs/backtrack_node.gif">
</center>

###### Expanding Node Descendants

Expanding a node's descendants allows you to immediatly view everything that happened because of this node. This action reveals the subgraph rooted at the selecte node.

<center>
<img src="docs/imgs/expand_descendants_node.gif">
</center>

#### Toggling Node and Edge Types

Sometimes, a Node or Edge might not be relevant to the current incident, you can toggle edge and node types on and off. As soon as the type is toggled, the nodes or edges of that type are removed from the visible graph.

Toggling a node type off prevents that node type to be used when using mutators, or when pulling in neighbours.

<center>
<img src="docs/imgs/toggle_types.gif">
</center>

#### Undo/Redo Action and Reset

Any action in the graph is immediatly reversable! Using the undo/redo buttons you can revert any action you perform. The reset button sets the graph state to when it loaded, saving you a refresh.

<center>
<img src="docs/imgs/redo_undo.gif">
</center>

#### Graph Perspectives

As you change the graphs current state using the above action, you might also want to view the current set of visible node and edges in a different perspective. The tabs at the top of the graph screen allow you to transform the data into a variety of views:

-   Graph (Default perspective)
-   Tree
-   Table
-   Timeline
-   Markdown

Each of the perspectives supports focusing on nodes by clicking on them.

<center>
<img src="docs/imgs/perspectives.gif">
</center>

## Python Library

The graph generation process can be performed programatically using the python library. The graph generation process is made up of three steps:

1. `DataSource` classes parse and yield events one by one.
2. `Transformer` classes take those inputs, and transform them into various `Node` classes such as `Process`.
3. `Backend` classes take the array of nodes, place them into a graph structure, and send them to a desired location.

The Python package can be installed via pip:

```python
pip install pybeagle
```

Creating a graph requires chaining these together. This can be done functionally:

```python
from beagle.datasources import HXTriage

# By default, using the to_graph() class uses NetworkX
G = HXTriage('test.mans').to_transformer().to_graph()
<networkx.classes.multidigraph.MultiDiGraph at 0x12700ee10>
```

Using the functional calls, you can also define which Backend you wish to usem for example, to send data to DGraph

```python
from beagle.datasources import HXTriage
from beagle.backends import DGraph

# The data will be sent to the DGraph instance configured in the
# configuration file
backend = HXTriage('test.mans').to_transformer().to_graph(backend=DGraph)

```

You can also manually invoke each step in the above process, accessing the intermediary outputs

```python
>>> from beagle.backends import NetworkX
>>> from beagle.datasources import HXTriage
>>> from beagle.transformers import FireEyeHXTransformer

>>> datasource = HXTriage("test.mans")
>>> transformer = FireEyeHXTransformer(datasource=datasource)
>>> nodes = transformer.run()
>>> backend = NetworkX(nodes=nodes)
>>> G = backend.graph()
```

If you want to manually call each step, you will need to ensure that the `Transformer` class instance is compatible with the output of the provided `DataSource` class.

-   All Backends are compatible with all Transformers.

Each data source defines the list of transformers it is compatible with, and this can be accessed via the `.transformers` attribute:

```python
>>> from beagle.datasources import HXTriage
>>> HXTriage.transformers
[beagle.transformers.fireeye_hx_transformer.FireEyeHXTransformer]
```

## Documentation

-   [REST API Overview](docs/rest_api.md)
-   [Configuration](docs/configuration.md)
-   [Developement](docs/development.md)
-   [Design Logic](docs/design_overview.md)


