# Raiden Development Guide

Welcome! This guide serves as the guideline to contributing to the Raiden Network
codebase. It's here to help you understand what development practises we use here
and what are the requirements for a Pull Request to be opened against Raiden.

- [Contributing](#contributing)
    - [Creating an Issue](#creating-an-issue)
    - [Creating a Pull Request](#creating-a-pull-request)
	- [Pull Request Reviews](#pull-request-reviews)
- [Development environment setup](#development-environment-setup)
- [Development Guidelines](#development-guidelines)
    - [Coding Style](#coding-style)
    - [Test Suite](#test-suite)
    - [Workflow](#workflow)

## Contributing

There are two ways you can contribute to the development. You can either open
an Issue or if you have programming abilities open a Pull Request.

### Creating an Issue

If you experience a problem while using Raiden or want to request a feature
then you should open an issue against the repository. All issues should
contain:

**For Feature Requests:**

- A description of what you would like to see implemented.
- An explanation of why you believe this would make a good addition to Raiden.

**For Bugs:**

- A short description of the problem.
- Detailed description of your system, raiden version, geth version, solidity
  version e.t.c.
- What was the exact unexpected thing that occured.
- What you were expecting to happen instead.

#### Including sensitive data in your issue reports

In most cases we would need to see the debug logs and the database of your raiden node to be able to determine what happened and help debug the issue. If your issue occured on the mainnet and you would like to keep the data private you can encrypt the logs with a special pgp key we provide and send them to RaidenErrors@brainbot.li . A short guide on how to do this follows:

1. Check [this](https://linuxconfig.org/how-to-encrypt-and-decrypt-individual-files-with-gpg) guide for how to install gpg and for general info on how to encrypt and decrypt files.
2. Get the raiden pgp key from [github](https://github.com/raiden-network/raiden/blob/master/raiden_errors_key.gpg).
3. Import the key by: `$: gpg --import path/to/raiden_errors_key.gpg`
4. Find the relevant debug logs and the DB and put them in an archive: `$: tar -cvf debug_data.tar path/to/debug.log path/to/raiden.db`
5. Encrypt the archive with our public key: `$: gpg -e -u "Enter your name here" -r "RaidenErrors@brainbot.li" debug_data.tar`
6. You should now have a file called `debug_data.tar.gpg` in your current directory.
7. Take the above file and email it to RaidenErrors@brainbot.li

Last updated: 20th of December 2018

#### Privacy Notice - Sending sensitive data

This Privacy Notice informs you about the handling of personal data in case you chose to send us your notes, log-files and/or databases for error tracking and debugging through the process described above.

##### Introduction
Brainbot labs Establishment have appointed a **data protection officer (DPO)** specifically for handling these data and ensuring their privacy.

- The email account "RaidenErrors@brainbot.li" is only accessible by the DPO and his deputy.

**Data handling within the Raiden team**
   - The DPO or his deputy  provides the data to the respective developer responsible for analyzing the data.
   - After the analysis the DPO or his deputy will ensure that the data will be deleted from the computers used for the analysis as well as from any email server used by brainbot labs Establishment to forward the information.

**Personal data collected**
Due to the fact that the user chooses the content of the files and the databases sent to brainbot labs Establishment, the personal data contained lies in the responsibility of the user who is submitting the data. However, if the user chooses to send standard log-files generated by the Raiden client the personal data will typically comprise:

- Ethereum addresses
- Token addresses and balances
- Raiden transfer data

**Usage of data collected**
The data collected from the users according to the process described above will only be used for analyzing errors reported by the user and as support for fixing the respective bugs. After releasing the corresponding bug fix, the data will be deleted according to the process as described above.



### Creating a Pull Request

If you have some coding abilities and would like to contribute to the actual
codebase of Raiden then you can open a Pull Request(PR) against the repository.

All PRs should be:

- Self-contained.
- As short as possible and address a single issue or even a part of an issue.
  Consider breaking long PRs into smaller ones.

In order for a Pull Request to get merged into the main repository you should
have one approved review from one of the core developers of Raiden and also all
Continuous Integration tests should be passing and the CI build should be
green.

Additionally you need to sign the raiden project CLA (Contributor License
Agreement). Our CLA bot will help you with that after you created a pull
request. If you or your employer do not hold the whole copyright of the
authorship submitted we can not accept your contribution.

For frequent contributors with write access to the repository we have a set of labels to put on Pull Requests to signal to our colleagues what the current state of the PR is. These are:

- [Dev: Please Review](https://github.com/raiden-network/raiden/labels/dev%3A%20Please%20Review) to a Pull Request that is currently ready for a reviewer to have a look at.
- [Dev: Work in Progress](https://github.com/raiden-network/raiden/labels/dev%3A%20Work%20In%20Progress) to a Pull Request that is either not yet ready for review or is getting PR review suggestions applied by the author until it's ready for review again.

### Pull Request Reviews

It is the responsibility of the author to ask for at least one person to review their Pull Request. That person should know the area of the code being changed. If the chosen reviwer does not feel confident in the review, they can then ask for someone else to additionally look at the code.

All the developers in the team should perform Pull Request reviews. Make it a habit to check [this](https://github.com/raiden-network/raiden/pulls?q=is%3Apr+is%3Aopen+label%3A%22dev%3A+Please+Review%22) link often to help your fellow colleagues who have PRs open pending for review.

We have a lot of tools that automatically check the quality of the code (flake8, mypy, pylint). All these are automatically ran by the CI. Therefore fixes related to linting are not part of PR reviews. Additionally reviewers are encouraged to not be nitpicky about the suggested changes they ask from the PR author. If something is indeed nitpicky then the reviewer is encouraged to state it beforehand. Example:

> nitpick: I don't really think XYZ makes sense here. If possible it would be nice to have it changed to KLM

The author of the PR can then choose to implement the nitpicks or ignore them.

PR authors should make pull request reviews easier. Make them as small as possible and even if some code is touched it does not mean that it needs to be refactored. For example don't mix style/typing changes with a big PR.

When a reviewer starts a PR review he should write a comment in the PR stating he is doing so. For example:

> Reviewing this now

This is to keep track of who is reviewing a PR and to also know when a PR review is ongoing.

For complicated PRs that touch the core of the protocol at least 2 core developers are recommended to have a look and provide an opinion.

When performing a PR review of non trivial PRs it is recommended to clone the branch locally, explore the changes with your editor, run tests and experiment with the changes so that a better understanding of the code change can be achieved and good constructive feedback given back to the author.

## Development environment setup

### Nix

See [nix.rst](docs/nix.rst).

### Development external dependencies

These are the required external dependencies for development:

- Python >=3.7.
- Ethereum client (Geth recommended).
- Solidity compiler >=0.4.23.
- A matrix capable server (Synapse recommended).
- A compiler tool chain to compile the eliptic curve library.
- Git for version control.

### Get the code

Start by getting the source code

    git clone https://github.com/raiden-network/raiden.git
    cd raiden

#### Install system wide dependencies

##### Archlinux

All the required packages are available in community or extra:

    sudo pacman -Sy go-ethereum python python-pip python-virtualenv solidity \
    base-devel git

##### Debian/Ubuntu

To get the Geth client and solidity compiler add Ethereum's PPA repository:

    sudo add-apt-repository -y ppa:ethereum/ethereum
    sudo apt-get update

Then install the required packages:

    sudo apt-get install build-essential git libffi-dev libgmp-dev libssl-dev \
    libtool pkg-config python3 python3-pip python3-dev python3-virtualenv \
    ethereum solc git

### Create the virtual env

It's highly recommended to use a virtual environment for development. This will
isolate the python dependencies used by Raiden from your system:

    virtualenv env
    source ./env/bin/activate

#### Development dependencies

Install the development dependencies:

    pip install -r requirements-dev.txt -c constraints.txt

## Development Guidelines

### Testing

To run the tests use pytest

    pytest raiden

Tests are split in unit tests, fuzz tests (which are currently grouped under
unit tests) and integration tests. The first are faster to execute while
the latter test the whole system but are slower to run. To choose which type of
tests to run, just use the appropriate directory.

    pytest raiden/tests/<integration|unit>

For a detailed explanation of the different types of tests, see the
[test suite section](#test-suite) below.

### Testing on the CI

By default whenever you make a Pull Request the linter tests, smoketests, unit tests and all the integration tests will run. This is in essence our entire test suite and at the time of writing takes around 50 mins assuming travis has enough free jobs for all the parallel jobs to run without any waiting time.

This can take a lot of time and as such there are various labels you can add in the body of a commit in order to change which tests run on Travis. Following is a breakdown of those labels:

- `[skip tests]`: This will skip all tests and run nothing.
- `[ci nightly]`: This will run no tests but start a nightly release build.
- `[no ci integration]`: This will only run the linter, smoketest and unit tests. It is rather fast.
- `[ci integration-general]`: This will run linter, smoke, unit and only the integration tests under the `integration/` directory but no other subdirectories. This will include our slowest integration tests at the moment. At the time of writing they take around 38 mins.
- `[ci integration-transfer]`: This will run linter, smoke, unit and only the integration tests under the `integration/transfer` directory. At the time of writing these take around 20 mins.
- `[ci integration-long-running]`: This will run linter, smoke, unit and only the integration tests under the `integration/long-running` directory. At the time of writing these are empty so they take up only the time that is required for setting up a job. Around 2 mins.
- `[ci integration-contracts]`: This will run linter, smoke, unit and only the integration tests under the `integration/contracts` directory. At the time of writing they take around 6 mins.
- `[ci integration-api]`: This will run linter, smoke, unit and only the integration tests under the `integration/api` directory. At the time of writing they take around 17 mins.
- `[ci integration-cli]`: This will run linter, smoke, unit and only the integration tests under the `integration/cli` directory. At the time of writing they take around 5 mins.

### Commiting Rules

For an exhaustive guide read [this](http://chris.beams.io/posts/git-commit/)
guide. It's all really good advice. Some rules that you should always follow though are:

- A commit title not exceeding 50 characters
- A blank line after the title (optional if there is no description)
- A description of what the commit did (optional if the commit is really small)

Why are these rules important? All tools that consume git repos and show you
information treat the first 80 characters as a title. Even Github itself does
this. And the git history looks really nice and neat if these simple rules are
followed.

### Encoding

Addresses should follow "sandwich encoding" so that each point of entry does
its own encoding into binary but the core programmatic API accepts only binary.
Thus we setup the following rules:

- Programmatic API should only expect binary and break if it accepts anything
  else. It should do type checking on its input and provide meaningful error
  for hex encoded addresses or length mismatch.
- All other places from which we receive addresses need to do their own
  encoding from whatever the input encoding is to binary. Such places are: CLI,
  Rest-API e.t.c.
- All modules which generate output to the outside world should also encode
  from binary to whatever the expected output encoding of that module is. Such
  places are stdout, communication with the ethereum node e.t.c.

### Coding Style

In this section we are going to describe the coding rules for contributing to
the raiden repository. All code you write should strive to comply with these
rules.

#### General Style

In this section we are going to see style rules that should be followed across all languages.

**Line breaks after operators**

For long expressions where we break the expression at the operators the line
break should come **after** the operator and not before.

The following should be avoided:

```python
        participant1_amount = (
            participant1_state.deposit
            + participant2_state.transferred_amount
            - participant1_state.transferred_amount
        )
```

instead it should be:

```python
        participant1_amount = (
            participant1_state.deposit +
            participant2_state.transferred_amount -
            participant1_state.transferred_amount
        )
```

#### Python

Raiden is written in Python and we follow the official Python style guide
[PEP8](https://www.python.org/dev/peps/pep-0008/). It is highly recommended to
use the [linting tools](requirements-lint.txt) in order to automatically
determine any and all style violations. The customizable part of pylint is at
[.pylint.rc](.pylint.rc).

**Line Length**

Flake8 will warn you for 99 characters which is the hard limit on the max
length. Try not to go above it. We also have a soft limit on 80 characters but
that is not enforced and is there just to encourage short lines.

**Shadowing built-ins**

Shadowing built-in names is not allowed. Pylint will also warn you about it. If you want to use a built-in name then add a trailing underscore and not a leading one. Leading ones are reserved for private attributes. For example `type` -> `type_`.

**Function definitions formatting**

The line length must be [smaller than 100 characters](#python), so the
function definition must be split when this limit is reached. When
splitting the function arguments, each must go into its own line, followed by
the argument type and a comma, and indented with 8 spaces.

The following is not allowed:

```python

def function_with_many_args(argument1: Type1, argument2: Type2, argument3: Type3) -> ReturnType:
    pass
```

This must be used instead:

```python

def function_with_many_args(
        argument1: Type1,
        argument2: Type2,
        argument3: Type3,
) -> ReturnType:
    pass
```

**Docstrings**

For docstrings we follow [PEP 0257](https://www.python.org/dev/peps/pep-0257/#multi-line-docstrings).

A single line docstring should be like this:
```python
def a(
        b: B,
        c: C,
) -> D:
    """ Here be docs """
    pass
```

A multiline docstring should have a short title and then a body. So like this:

```python
def a(
        b: B,
        c: C,
) -> D:
    """ Function Title

    body comes
    here
    """
    pass
```

The closing quotes should be on their own line. If in doubt consult the PEP.

**Usage of single and double quotes**

All strings must use single quotes by default.

Bad:

```python
s = "foo"
```

Good:

```python
s = 'foo'
```

The only reason to use double quotes is to avoid escaping the single quote in a
string. So this is okay:

```python
s = "Augusto's computer is awesome"
```

**Naming Convention**

Use descriptive variable names and avoid short abbreviations.

The following is bad:

```python
mgr = Manager()
a = AccountBalanceHolder()
s = RaidenService()
```

While this is good:

```python
manager = Manager()
balance_holder = AccountBalanceHolder()
service = RaidenService()
```

We try to follow a consistent naming convention throughout the codebase to make
it easy for the reader of the code to understand what is going on. Thus we
introduce the following rules:

For addresses:

- `<name>_address_hex` for hex encoded addresses
- `<name>_address` for binary encoded addresses

Lists of objects:

- `<name>s`, e.g. `channels` for a list `Channel` object instances.

- to initialize an empty list use `list()` instead of `[]`. Note this is only
  for style consistency's sake and may change in the future as there [may
  be](https://stackoverflow.com/questions/5790860/and-vs-list-and-dict-which-is-better)
  a tiny change in performance.

Mappings/dicts:

If it is a simple one to one mapping

`<name>_to_<name>`, e.g. `tokenaddress_to_taskmanager`

If the mapped to object is a list then add an `s`

`<name>_to_<name>s`, e.g. `tokenaddress_to_taskmanagers = defaultdict(list())`

To initialize an empty dict use `dict()` instead of `{}`. Note this is only for
style consistency's sake and may change in the future as there [may
be](https://stackoverflow.com/questions/5790860/and-vs-list-and-dict-which-is-better)
a tiny change in performance.

Class attributes and functions:

All class members should be private by default and they should start with a
leading `_`. Whatever is part of the interface of the class should not have the
leading underscore. The public interface of the class is what we should be
testing in our tests and it should be the only way other parts of the code use
the class through.

Minimal Example:

```python

class Diary(object):

    def __init__(self, entries):
        self._entries = entries

    def entry(index):
        return _entries[index]

```

**NewTypes and type comparisons**

For often used types it makes sense to define new types using the
`typing.NewType` function.  New type names should be capitalized.

```python
Address = NewType('Address', bytes)
```

These type definitions can not be used for type comparisons. To make this
possible always define an associated alias, which must start with `T_`.

```python
T_Address = bytes
```

**typing.Optional convention**

For typing.Optional we follow the convention that if the argument has a default
value of `None` then we should omit the use of `typing.Optional[]`.

Good Example:

```python

def foo(a: int = None)
```

Bad Example:

```python

def foo(a: typing.Optional[int] = None)
```

**imports**

Classes must be imported in the global namespace, unless there are name
collisions and module imports can be used. Example:

```python
import a
from b import Normal


class Conflict:
    pass


def f():
    return a.Conflict(), Normal()
```

#### Solidity

For solidity we generally follow the style guide as shown in the [solidity
documentation](http://solidity.readthedocs.io/en/develop/style-guide.html) with
a few notable exceptions:

**Variable Names**

All variable name should be in snake case, just like in python. Function names
on the other hand should be mixedCase. MixedCase is essentially like CamelCase
but with the initial letter being a small letter. This helps us to easily
determine which function calls are smart contract calls in the python code
side.

```js
function iDoSomething(uint awesome_argument) {
    doSomethingElse();
}
```

**Documentation**

Code should be documented. For docstrings the [Google
conventions](https://sphinxcontrib-napoleon.readthedocs.io/en/latest/example_google.html)
are used.

### Test suite

The test suite is divided into unit tests, integration tests and fuzz tests.
In addition to that, there is the scenario runner tool for acceptance testing.
The parts of the test suite differ in scope:

#### Fuzz tests

By fuzz testing we mean model-based randomized testing using
[hypothesis.stateful](https://hypothesis.readthedocs.io/en/latest/stateful.html)
as our testing tool.
The fuzz tests have the smallest scope, they are used only for the core state
machine - nearly all tested operations consist in processing state changes
through the central transition function (`raiden.node.state_transition`).

#### Unit tests

Many unit tests also test the proper processing of one or several state
changes, but they may also test just details of the state machine or other
parts of the code such as the api. The central `RaidenService` class and
the interaction with the transport layer and the smart contracts are not
in the scope of the unit tests.

Unit and fuzz tests use a shared set of factories, to be found in the
`raiden.tests.unit.factories` module, to create any objects necessary for
testing. Integration tests have their own fixtures module,
`raiden.tests.integration.fixtures`, for the same purpose.

#### Integration tests

The scope of the integration tests is the whole client. All integration tests
require instantiating one or more `RaidenService` instances and providing
the transport layer (e. g. by firing up a local Matrix server), which takes
considerable time.

As a general rule, a test should only be made an integration test if the
tested actions touch the transport layer. An exception to this are the tests
related to the smart contracts/smart contract proxies found in
`raiden.tests.integration.contracts`.

### Workflow

When developing a feature, or a bug fix you should always start by writing a
**test** for it, or by modifying existing tests to test for your feature.
Once you see that test failing you should implement the feature and confirm
that all your new tests pass.

Your addition to the test suite should call into the innermost level possible
to test your feature/bugfix. In particular, integration tests should be avoided
in favor of unit tests whenever possible.

Afterwards you should open a Pull Request from your fork or feature branch
against master. You will be given feedback from the core developers of raiden
and you should try to incorporate that feedback into your branch. Once you do
so and all tests pass your feature/fix will be merged.

#### Updating documentation/spec

If a PR you are working on is updating something in the REST API, then make
sure to also update the respective
[documentation](https://github.com/raiden-network/raiden/blob/master/docs/rest_api.rst).
The documentation update can also be in a different PR but should be taken care
of and linked to the initial PR.

If a PR you are working on is making a protocol change then make sure to also
update the [specification](https://github.com/raiden-network/spec). The spec
update can also be introduced in a different PR but should be taken care of and
linked to the initial PR.

#### Contributing to other people's PRs

If you are a core developer of Raiden with write privileges to the repository
then you can add commits or rebase to master any Pull Request by other people.

Let us take [this](https://github.com/raiden-network/raiden/pull/221) PR as an
example. The contributor has everything ready and all is looking good apart
from a minor glitch. You can wait until he fixes it himself but you can always
help him by contributing to his branch's PR:

```
git remote add hackaugusto git@github.com:hackaugusto/raiden.git
git fetch hackaugusto
git checkout travis_build
```

Right now you are working on the contributor's Pull Request. **Make sure** to
coordinate to avoid any conflicts and always warn people beforehand if you are
to work on their branch. Once you are done:

```
git commit -m 'Add my contribution

The PR was missing something. I added it.'
git push hackaugusto travis_build
```

Congratulations, you have added to someone else's PR!

#### Integrating Pull Requests

When integrating a successful Pull Request into the codebase we have the option
of using either a "Rebase and Merge" or to "Create a Merge commit".
Unfortunately in Github the default option is to "Create a Merge commit". This
is not our preferred option as in this way we can't be sure that the result of
the merge will also have all tests passing, since there may be other patches
merged since the PR opened. But there are many PRs which we definitely know
won't have any conflicts and for which enforcing rebase would make no sense and
only waste our time. As such we provide the option to use both at our own
discretion. So the general guidelines are:

- If there are patches that have been merged to master since the PR was opened,
  on top of which our current PR may have different behaviour then use **Rebase
  and Merge**.
- If there are patches that have been merged to master since the PR was opened
  which touch documentation, infrastucture or completely unrelated parts of the
  code then you can freely use **Create a Merge Commit** and save the time of
  rebasing.
