![test](https://github.com/davips/hdict/workflows/test/badge.svg)
[![codecov](https://codecov.io/gh/davips/hdict/branch/main/graph/badge.svg)](https://codecov.io/gh/davips/hdict)
<a href="https://pypi.org/project/hdict">
<img src="https://img.shields.io/pypi/v/hdict.svg?label=release&color=blue&style=flat-square" alt="pypi">
</a>
![Python version](https://img.shields.io/badge/python-3.10+-blue.svg)
[![license: GPL v3](https://img.shields.io/badge/License-GPLv3-blue.svg)](https://www.gnu.org/licenses/gpl-3.0)

<!--- [![DOI](https://zenodo.org/badge/DOI/10.5281/zenodo.5501845.svg)](https://doi.org/10.5281/zenodo.5501845) --->
[![arXiv](https://img.shields.io/badge/arXiv-2109.06028-b31b1b.svg?style=flat-square)](https://arxiv.org/abs/2109.06028)
[![API documentation](https://img.shields.io/badge/API-autogenerated-a030a0.svg)](https://davips.github.io/hdict)
[![Manual](https://img.shields.io/badge/manual-handcrafted-a030a0.svg)](https://hosh.page)
[![Downloads](https://static.pepy.tech/badge/hdict)](https://pepy.tech/project/hdict)
![PyPI - Downloads](https://img.shields.io/pypi/dm/hdict)

# hdict { A _unique_ data structure }
[Website](https://hosh.page) | 
[Latest Release](https://pypi.org/project/hdict) |
[Current Code](https://github.com/davips/hdict) |
[API Documentation](https://davips.github.io/hdict)

Please see the [website](https://hosh.page) for more detailed usage information.

## Overview
Shortly, `hdict` is a data structure useful for frictionless experiments, distributed data, among others.
It can be more formally defined as a hosh¹-based cacheable lazy `dict` with predictable/deterministic universally unique² identifiers.

This library is stable for the most common scenarios.
However, we will wait for more use in the wild before reaching the major version 1.
See the Section Versioning below for more information.

¹ hosh: _Operable HaSH_

² unique: _probabilistically guaranteed_



### Context

  * [hosh](https://pypi.org/project/hosh): identification engine at the core of `hdict`
  * previous projects: [hoshmap](https://pypi.org/project/hoshmap), [idict](https://pypi.org/project/idict), [ldict](https://pypi.org/project/ldict), [garoupa](https://pypi.org/project/garoupa)

### The concept

`hdict` is like an ordinary `dict` with `str` keys. 
Each entry, called _field_, and the `hdict` itself, are identified by 40-digit hashes (see [hosh](https://pypi.org/project/hosh)).
A `hdict` object (say `d`) provides two extra entries: `_id` (hdict identifier) and `_ids` (field identifiers),
accessible through square brackets or through the shortcuts `d.id` and `d.ids`.

**`hdict` creation**
<details>
<p>

```python3
from hdict import hdict

# From named arguments.
d = hdict(x=5, y=7, z=10)

# From a dict object.
d = hdict({"x": 5, "y": 7, "z": 10})

# From an empty 'hdict' object.
d = hdict() >> {"x": 5} >> {"y": 7, "z": 10}

# All three options have the same result.
d.show(colored=False)
"""
{
    x: 5,
    y: 7,
    z: 10,
    _id: BN-3Q3Z.2Q.9nsbIYnOI75HT7xhgjvF6wErwBPTn,
    _ids: {
        x: ecvgo-CBPi7wRWIxNzuo1HgHQCbdvR058xi6zmr2,
        y: eJCW9jGsdZTD6-AD9opKwjPIOWZ4R.T0CG2kdyzf,
        z: u-Yykj2nDtKaUMGzfqScX5Y14qC7eqJrO7lXrJ1m
    }
}
"""
```

```python3

from hosh import setup

# For better integration within the documentation, we change the color theme.
setup(dark_theme=False)

d.show(colored=False)

"""
{
    x: 5,
    y: 7,
    z: 10,
    _id: BN-3Q3Z.2Q.9nsbIYnOI75HT7xhgjvF6wErwBPTn,
    _ids: {
        x: ecvgo-CBPi7wRWIxNzuo1HgHQCbdvR058xi6zmr2,
        y: eJCW9jGsdZTD6-AD9opKwjPIOWZ4R.T0CG2kdyzf,
        z: u-Yykj2nDtKaUMGzfqScX5Y14qC7eqJrO7lXrJ1m
    }
}
"""
```


</p>
</details>

A field contains a value or a function application.
A field pointing to an application is only evaluated on demand, i.e., lazily.

<!-- ------------------------------------------------------------------------ 
Value objects can have custom identifiers as well, if provided whithin the entry `ids`. 
Otherwise, identifiers for functions and values will be calculated through blake3 hashing of their content.
For functions, the bytecode is used as content.
For this reason, such functions should be simple, with minimal external dependencies or
with their import statements inside the function body.
This decreases the odds of using two functions with identical local code (and, therefore, identical identifiers)
performing different calculations.

transformation steps done through the operator `>>`, which symbolizes the ordering of the steps.
* **value insertion** - represented by dict-like objects
* **function application** - represented by ordinary Python functions

Functions, `hdict`s, and values have a deterministic UUID
(called _hosh_ - **o**perable **h**a**sh**). 
Identifiers (hoshes) for `hdict`s and values are predictable through the
magic available [here](https://pypi.org/project/garoupa).
An `hdict` is completely defined by its key-value pairs so that
it can be converted from/to a built-in `dict`.


------------------------------------------------------------------------  -->

Please refer to our [website](https://hosh.page) for more examples.


## Installation
### ...as a standalone lib
```bash
# Set up a virtualenv. 
python3 -m venv venv
source venv/bin/activate

# Install from PyPI...
pip install --upgrade pip
pip install -U hdict

# use the flag 'full' for extra functionality (recommended if working with persistence or some special values like DataFrames)
pip install -U hdict[full]

# ...or, install from updated source code.
pip install git+https://github.com/davips/hdict
```

### ...from source
```bash
git clone https://github.com/davips/hdict
cd hdict
poetry install

# use the flag 'full' for extra functionality (recommended if working with persistence or some special values like DataFrames)  
poetry install -E full
```

## Examples

Please refer to our [website](https://hosh.page) for examples.



## Development
### Licensing
The initial license choosen is GPL. Please contact the developer for other licensing needs.

### Versioning
While the version scheme has a meaningful calendar component (`minor=yymmdd`), it is still compatible with semantic versioning.
For instance, the version `0.230215.1` means `major=0`, `minor=230215`, `micro/patch=1`. Notes:
 * While `major=0`, some compatibility-breaking changes may occur.
 * From `major=1` onwards, compatibility-breaking changes increment it, and update the minor version to reflect the release date.
 * New (non breaking) features update only the minor version to reflect the release date.
 * Bug fixes (including compatibility-breaking ones) increment only the micro version.

### Contributing
#### Collaboration
We have ongoing research applying this tool to machine learning and clinical academic experiments.
Although, the scope of application is broad as it encompasses software development in general.
Feel free to contact us if you are interested in the project/concept or have a suggestion/interesting problem to be solved.

#### Donation
Currently there are no established forms of donation.
Expenses:
  * Programming hours
  * Support
  * Custom features
  * Domain name maintenance yearly costs

### Acknowledgement
This work was based on a previous research supported by Fapesp under supervision of
Prof. André C. P. L. F. de Carvalho at CEPID-CeMEAI (Grants 2013/07375-0 – 2019/01735-0)
until 2021-03-31.

We would like to thank the providers of free tools that make this project feasible:
  * github for hosting the code repository freely
  * Oracle for hosting the website freely
  * Developers of Mint, XFCE, and GNU/Linux for the operating system
  * JetBrains for Intellij IDEA Community Edition
  * Developers of Python packages listed in our project.toml
