<!--- Copyright 2022 eprbell --->

<!--- Licensed under the Apache License, Version 2.0 (the "License"); --->
<!--- you may not use this file except in compliance with the License. --->
<!--- You may obtain a copy of the License at --->

<!---     http://www.apache.org/licenses/LICENSE-2.0 --->

<!--- Unless required by applicable law or agreed to in writing, software --->
<!--- distributed under the License is distributed on an "AS IS" BASIS, --->
<!--- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. --->
<!--- See the License for the specific language governing permissions and --->
<!--- limitations under the License. --->

# DaLI Frequently Asked Questions (Developer)

## Table of Contents
* **[General Questions](#general-questions)**
  * [What are the Contribution Guidelines?](#what-are-the-contribution-guidelines)
  * [What are the Design Guidelines?](#what-are-the-design-guidelines)
  * [What is the Best Way to Get Started on DaLI Development?](#what-is-the-best-way-to-get-started-on-dali-development)
  * [How to Develop a DaLI Data Loader Plugin?](#how-to-develop-a-dali-data-loader-plugin)
  * [How to Fill the Unique Id Field?](#how-to-fill-the-unique-id-field)
  * [Should I Implement a CSV or a REST Data Loader Plugin?](#should-i-implement-a-csv-or-a-rest-data-loader-plugin)
  * [How to Develop a DaLI Pair Converter Plugin?](#how-to-develop-a-dali-pair-converter-plugin)
  * [How to Develop a DaLI Country Plugin?](#how-to-develop-a-dali-country-plugin)
  * [How Does DaLI Merge Transactions Between Different Exchanges/Wallets?](#how-does-dali-merge-transactions-between-different-exchangeswallets)
  * [Why the Strange Directory Structure with Src?](#why-the-strange-directory-structure-with-src)

## General Questions

## What are the Contribution Guidelines?
Read the [contribution guidelines](../CONTRIBUTING.md#contributing-to-the-repository) section of the documentation.

## What are the Design Guidelines?
Read the [design guidelines](../README.dev.md#design-guidelines) section of the documentation.

## What is the Best Way to Get Started on DaLI Development?
Read the [contribution guidelines](../CONTRIBUTING.md#contributing-to-the-repository) and the [developer documentation](../README.dev.md) (especially the Internal Design section). Then look for an unassigned [issue](https://github.com/eprbell/dali-rp2/issues) that is marked as `good first issue`, or ask the project maintainers.

## How to Develop a DaLI Data Loader Plugin?
Read the [Internal Design](../README.dev.md#internal-design) section of the Developer Documentation (and in particular the Plugin subsections).

## How to Fill the Unique Id Field?
Data loader plugins read data from CSV and/or REST sources and return them to DaLI as transaction instances. One of the most important jobs of a data loader plugin is to fill in the `unique_id` field. This field is used by the DaLI transaction resolver to match incomplete transaction and join them. E.g. a transfer from Coinbase to Trezor is initially represented as 2 transactions:
* an IntraTransaction from Coinbase to UNKNOWN, generated by the Coinbase plugin (which doesn't know that the destination is a Trezor);
* an IntraTransaction from UNKNOWN to Trezor, generated by the Trezor plugin (which doesn't know that the source is Coinbase).

These two transactions both share the same `unique_id` (which is typically the transaction hash and is filled in by the respective plugins): the transactions are fed to the transaction resolver, which uses `unique_id` to match them up and to join them into a single transaction without UNKNOWN fields.

This is why `unique_id` is so important in data loaders: if an incorrect value is used, the transaction resolver will be unable to match and join incomplete transactions.

Some data sources (especially exchange CSVs) have invalid or missing transaction identifier information: sometimes it's empty, other times (confusingly) it's an internal id of the exchange (which cannot be used by the transaction resolver) instead of the hash. In such cases (where transaction hash is not available) DaLI plugins must set the `unique_id` field to `Keyword.UNKNOWN`.

Some examples of this:
* BlockFI CSVs don't contain the hash or any other transaction id;
* Coinbase CSVs contain an internal transaction id, but it's not the hash.

There is one more nuanced detail to this topic. If the transaction is complete and DOESN'T need resolution (e.g. a BUY or a SELL transaction), then it's ok to use an internal exchange id as the `unique_id`: it will help document the transaction and can be helpful when looking it up in the logs.

Check the following for more details:
* [transaction resolver dev docs](https://github.com/eprbell/dali-rp2/blob/main/README.dev.md#the-transaction-resolver)
* [manual plugin docs](https://github.com/eprbell/dali-rp2/blob/main/docs/configuration_file.md#partial-transactions-and-transaction-resolution), which contain an example of how users would use the `unique_id` field in manual plugin CSV files to finish incomplete transactions for unsupported exchanges.

## Should I Implement a CSV or a REST Data Loader Plugin?
REST endpoints typically provide more complete data compared to CSV files, so they are preferred if available. Often CSV files have incorrect or missing [unique_id](#how-to-fill-the-unique-id-field), spot_price, etc. Here is how to decide whether to write a CSV or a REST plugin for a given service (wallet or exchange):
* given the choice, a REST plugin is always preferrable to a CSV one, if the endpoint is available;
* if a REST plugin for the service is already implemented and the CSV files for this service are incomplete, then there is no need for a CSV plugin;
* if a REST plugin for the service is already implemented and the CSV files for this service are complete (no missing / incorrect information), then it's OK to have a CSV plugin as an alternative option;
* if a REST plugin for the service is NOT yet implemented and CSV files for this service are available, then a CSV plugin is good to have regardless of whether CSV files are complete or not (it's better than nothing). This CSV plugin may be decommissioned in the future, if a REST plugin is implemented.

## How to Develop a DaLI Pair Converter Plugin?
Read the [Internal Design](../README.dev.md#internal-design) section of the Developer Documentation (and in particular the Plugin subsections).

## How to Develop a DaLI Country Plugin?
Read the [Internal Design](../README.dev.md#internal-design) section of the Developer Documentation (and in particular the Plugin subsections).

## How Does DaLI Merge Transactions Between Different Exchanges/Wallets?
Read about the [transaction_resolver](../src/dali/transaction_resolver.py) in the [Internal Design](../README.dev.md#the-transaction-resolver) section of the Developer Documentation.

## Why the Strange Directory Structure with Src?
Because DaLI is a [src](https://bskinn.github.io/My-How-Why-Pyproject-Src/)-[based](https://hynek.me/articles/testing-packaging/) [project](https://blog.ionelmc.ro/2014/05/25/python-packaging/).

## How to Use the Cache to Speed Up Development?
Use the `-c` command line option to enable the cache. This instructs DaLI to store transactions coming from cache-enabled plugins in the cache. The next time DaLI is run, transaction data is read directly from the cache instead of from the plugin native source. To make a plugin cache-enabled, just define its `cache_key()` method (for an example, look at the [Coinbase plugin](../src/dali/plugin/input/rest/coinbase.py)). Note that the code doesn't yet check for newer data: if it finds cached data, it reads it but doesn't check the native source for more recent entries, so currently the cache is only useful to speed up development, not for regular use. The cache is stored in the `.dali_cache/` directory: to reset the cache delete this directory.
