# The Genovation python toolbox

This python tool-box is made to support Genovation associates in their day-to-day work.

# INSTALL

## Pre-requisites

* Windows PowerShell (Terminal on MS Windows)
  * Console to run `genov` tool box, and its commands
  * How-to from Microsoft: https://learn.microsoft.com/en-us/powershell/scripting/install/installing-powershell-on-windows

* Python:
  * `genov` is tested with Python 3.13
  * As of EoY 2024, Python 3.13.0 is available at: https://www.python.org/downloads/
  * Note: starting with Python 3.4, `pip` is by default included

Once installed, you can check in your terminal:
```console
foo@bar:~$ python3 --version
           >> Python 3.13.0
foo@bar:~$ pip3 --version
           >> pip 24.3.1 from /Library/Frameworks/Python.framework/Versions/3.13/lib/python3.13/site-packages/pip (python 3.13)
```

Notes:
* On your system, you might have to replace the `python3` command with `python`
* To check where your python is installed: `which python3`: `/Library/Frameworks/Python.framework/Versions/3.13/bin/python3`

## Install Genov

* Type in your Terminal: `pip3 install genov`

## Versions

* 0.0.1, as of 29-Nov-2024: framework initialized
* 0.0.2, as of 3-Dec-2024: get issues from jira, and persist in an Excel spreadsheet
* 0.0.4, as of 2-Jan-2025: framework enriched with test coverage, lint, typer documentation and sphinx documentation
* 0.0.5, as of 18-Mar-2025: DpDatePredictability, along with typer commands (CAUTION: poor performance...)
* 0.0.6, as of 22-Mar-2026: DpDataPredictability fully reengineered

# Contribute

## Dependencies

### To run

| Dependencies     | Description                                                                                                            |
|:-----------------|:-----------------------------------------------------------------------------------------------------------------------|
| `com-enovation`  | The seed toolbox that we use to initialize this toolbox. To decommission as commands are being re-instantiated here... |
| `typer`          | Library for building CLI applications, based on Click                                                                  |
| `tomlkit`        | Library for manipulating configuration file                                                                            |
| `pydash`         | Library for manipulating dictionaries with path...                                                                     |
| `jsonschema`     | Library for validating json schema                                                                                     |
| `jsonata-python` | Library for exploring dictionaries and json                                                                            |
| `openpyxl`       | Library for reading and writing excel files                                                                            |

### To contribute

| Dependencies               | Description                                                                                               |
|:---------------------------|:----------------------------------------------------------------------------------------------------------|
| `hatch`                    | Extensible, standards compliant build backend.                                                            |
| `pytest`                   | To run tests (natively imported by hatch).                                                                |
| `pytest-cov`               | To measure test coverage (natively imported by hatch).                                                    |
| `sphinx`                   | To document project                                                                                       |
| `sphinx-autodoc-typehints` | To use Python 3 annotations for documenting acceptable argument types and return value types of functions |
| `sphinx_rtd_theme`         | The 'read the docs' theme                                                                                 |

## Steps to publish changes/ enhancements to genov

### Code formatting and linting

* Run `hatch fmt`
  * Ensure you have no error
  * Message of the kind should be returned: `Found 2 errors (2 fixed, 0 remaining)`
* Notes:
  * Linting analyzes how the code runs and detects errors
  * Formatting restructures how code appears.

### Test coverage

* Run `GENOV_TEST_LEVEL=FULL hatch test --cover --strict`
  * Ensure you have no error, and that you effectively have 80% coverage
* Notes:
  * Target is to have 80% of the code covered with tests
  * No publication allowed in case an average of less than 80%
  * Environment variable `GENOV_TEST_LEVEL` can take value `FULL` or `DEFAULT`, and is then reflected in test/conftest.py --> e_test_level.

### Documentation

#### Readme.md

* Run `hatch run docs:md`
  * Message of the kind should be returned: `Docs saved to: GENOV.md`
* Notes: the command line above executes the following steps
  * The typer command to generate docs for our package that we access through our README.md
  * Requires an environment where the distribution is installed
    * `rm -rf dist;hatch build;pip3 install dist/genov-*.tar.gz`
  * Run the typer command to document:
    * Step 1, we call the command to produce the document: `typer genov.genov utils docs --output GENOV.md.raw --name genov`
    * Step 2, we reformat quotes, so they are properly handled: `awk '{gsub(/&quot;/, ""); print}' GENOV.md.raw > GENOV.md;`
  * Modify the `README.md` file to append the typer documentation:
    * Step 1, we rename the file: `mv README.md README.md.previous`
    * Step 2, we discard the tail of the readme file (pattern modified to `Otomated documentation below`so the script is not confused...): `awk '!(f>3);/Otomated documentation below/{f=1}; {if (f) {f=f+1}};' README.md.previous > README.md.tmp`
    * Step 3, we concatenate the latest documentation: `cat README.md.tmp GENOV.md > README.md`
    * Step 4, we cleanse the tmp version: `rm README.md.tmp`
  * Final command is: `rm -rf dist;hatch build;pip3 install dist/genov-*.tar.gz;typer genov.genov utils docs --output GENOV.md.raw --name genov;awk '{{gsub(/&quot;/, \"\\\"\"); print}}' GENOV.md.raw > GENOV.md;mv README.md README.md.previous; awk '!(f>3);/Otomated documentation below/{{f=1}}; {{if (f) {{f=f+1}}}};' README.md.previous > README.md.tmp;cat README.md.tmp GENOV.md > README.md;rm README.md.tmp`

#### Sphinx documentation

* Run `hatch run docs:sphinx`
  * Message of the kind should be returned: `The HTML pages are in [./docs/]_build/html.`
* Note:
  * To auto-document the entire application, we use sphinx
  * The command is `make html` from the sub-directory `./docs`
  * To document class constants, use the following
    ```console
    #: The description of my constant goes there, after "#:"
    MY_CONSTANT: str = "Value for my constant"
    ```

### Publish

* Update the `version`
  * In `pyproject.toml`
  * In `README.md` (along with a light description of the updates)
* Run `rm -rf dist/*;hatch build;hatch publish -r test` (To build and publish in test)
  * Directory `dist` is generated (with distribution files)
  * Message of the kind should be returned: `https://test.pypi.org/project/genov/[...latest version...]/`
* Run `hatch publish` (Once verified in test, to publish in production)
  * Message of the kind should be returned: `https://pypi.org/project/genov/[...latest version...]/`
* Notes:
  * Python equivalent command to publish: `python3 -m twine upload --repository pypi dist/*`
  * Package viewable at [pypi](https://pypi.org/project/genov)

## Pycharm configuration

* Unit test configuration, from menu `Run > Edit Configurations...`
  * `Configuration > Target > Script path: ~/PycharmProjects/com_enovation.murex/tests`
  * `Configuration > Working directory: ~/PycharmProjects/com_enovation.murex/`
  * `Configuration > Add content roots to PYTHONPATH: checked`
  * `Configuration > Add source roots to PYTHONPATH: checked`
  * `Configuration > Additional Arguments: --assert=plain`: to get assert messages when executing tests

## Python Cheat sheet

* To upgrade python:
  * The version used as the interpreter in PyCharm project, with simlinks under PycharmProjects/genov/venv/bin: `brew upgrade python`
    * Stored under `/opt/homebrew/bin/python3`
  * The version used by MacOsX: re-download/ re-install from python.org
    * Stored under `cd /usr/local/bin` --> `/Library/Frameworks/Python.framework/Versions/3.13/bin`
  * To install python: `brew install python`
    * See `https://brew.sh/` to install brew`

* To get emojis that can be printed by `rich.print()`: run `python -m rich.emoji` in console

* Check we have the latest versions:
  * pip: `python -m pip install --upgrade pip`
  * build to generate the distribution: `python -m pip install --upgrade build`

* Update packages using pip
  * Check all packages are fine: `pip check`
  * List all packages outdated: `pip list --outdated`
  * Update all packages outdated:
    * On Mac: `pip --disable-pip-version-check list --outdated --format=json | python -c "import json, sys; print('\n'.join([x['name'] for x in json.load(sys.stdin)]))" | xargs -n1 pip install -U`
    * On Windows: `pip freeze | %{$_.split('==')[0]} | %{pip install --upgrade $_}`
* A simple example package. You can use [Github-flavored Markdown](https://guides.github.com/features/mastering-markdown/) to write your content.

---
WARNING: do not modify below, as automatically override by script `hatch run docs:md!!!

Automated documentation below
---

# `genov`

Genov tool box, the application with all the commands you need in your day-to-day work at Genovation.

Use the VERBOSE parameter to set the level of logs you need, and let you guide by the HELP.

**Usage**:

```console
$ genov [OPTIONS] COMMAND1 [ARGS]... [COMMAND2 [ARGS]...]...
```

**Options**:

* `-v, --verbose / --no-verbose`: Level of logging verbosity: INFO (--verbose), WARNING (default) or ERROR (--no-verbose).  [default: (WARNING)]
* `--install-completion`: Install completion for the current shell.
* `--show-completion`: Show completion for the current shell, to copy it or customize the installation.
* `--help`: Show this message and exit.

**Commands**:

* `welcome`: The command welcomes to the Genovation...
* `jr-gt-iss`: The command fetches issues from a Jira...
* `df-to-stdout`: The command prints into console the...
* `df-to-xlsx`: The command persists the DataFrame aliased...
* `df-load-xlsx`: The command loads FILE from the file...
* `dict-to-stdout`: The command prints into console the...
* `dict-to-json`: The command persists the dictionary...
* `cfg-to-stdout`: The command prints into console the...
* `dp`: The command instantiates an instance of...
* `dp-asof`
* `dp-compute`: The command computes the Date...
* `dp-decompose`: The command decomposes the Date...
* `dp-slide`: The command slides the Date Predictability...
* `dp-describe`: The command describes the Date...

## `genov welcome`

The command welcomes to the Genovation toolbox!
This command will greet NAME, and return welcome message.
It experiments the rich library that can be used across other commands.

**Usage**:

```console
$ genov welcome [OPTIONS] NAME
```

**Arguments**:

* `NAME`: [required]

**Options**:

* `--help`: Show this message and exit.

## `genov jr-gt-iss`

The command fetches issues from a Jira PROJECT within a given SITE using a predefined TOKEN belonging to an ACCOUNT.
The result is transformed through a MAP, returned depending on AS_DF as a DataFrame or a dict, and stored in
context as ALIAS.

To connect to the underlying Atlassian Rest API, the following parameters are required from the configuration file
under ~/.genov/genov.toml to retrieve the following parameters:

- SITE: a site contains instances of Atlassian products under an organization. An organization has one or more
  sites, each site can only have one instance of each product
  -&gt; path in configuration file: the site from `atlassian.sites` where label is SITE
- PROJECT: a project is a collection of issues (stories, bugs, tasks, etc.), used to represent the development
  work for a product, project, or service in Jira
  -&gt; path in configuration file: the project from `atlassian.sites.jira.projects` where label is PROJECT
- Resource: the REST resources available in Jira Cloud
  -&gt; path in configuration file: the resource from `atlassian.sites.jira.resources` where label is
  &#x27;GET /rest/api/3/search/jql&#x27;
- MAP: used to transform complex and proprietary responses from Atlassian Rest APIs in data we can process,
  such as DataFrame, Dictionaries, etc.
  -&gt; path in configuration file: the map from `atlassian.sites.jira.resources.maps` where label is MAP
- ACCOUNT: an Atlassian account is an online Atlassian identity that exists independently of the Atlassian
  products that is used. An account is required to log in to any Atlassian products, such as Jira or Confluence. An
  Atlassian account is like a Google account. When you log into your Gmail account, you can also log in to YouTube,
  Google Docs, etc.
  -&gt; path in configuration file: the account from `atlassian.accounts` where label is ACCOUNT
- TOKEN: API tokens are used to authenticate users when making calls to Atlassian product APIs
  -&gt; path in configuration file: the token from `atlassian.accounts` where label is TOKEN.

Illustration of the expected configuration in the ~/.genov/genov.toml file

.. Code-block:: text

    {
        "atlassian": {
            "accounts": [
                {
                    "label": "pro",
                    "email": "jsg@genovation.associates",
                    "tokens": [
                        {
                            "label": "GSculpt",
                            "token": "whatever"
                        }
                    ]
                }
            ],
            "sites": [
                {
                    "label": "genovation",
                    "jira": {
                        "projects": [
                            {
                                "label": "GENOVATION",
                                "key": "GENO"
                            }
                        ],
                    "resources": [
                        {
                            "label": "GET /rest/api/3/search/jql",
                            "url": "https://genovation.atlassian.net/rest/api/3/search",
                            "maps": [
                                {
                                    "label": "flatList",
                                    "map": [
                                        {
                                            "dict.path": "id",
                                            "df.column": "id",
                                            "df.type": "int"
                                        }
                                    ]
                                }
                            ]
                        }
                    ]
                }
            ]
        }
    }

**Usage**:

```console
$ genov jr-gt-iss [OPTIONS]
```

**Options**:

* `-s, --site site`: Atlassian site to consider. If not set, defaulted to first configured site.  [default: (If not provided, defaulted with the first site configured in the configuration file)]
* `-p, --project project`: Atlassian jira project to consider. If not set, defaulted to first configured project.  [default: (If not provided, defaulted with the first project configured in the configuration file)]
* `-m, --map map`: Map to transform Atlassian responses. If not set, defaulted to first configured map.  [default: (If not provided, defaulted with the first map configured in the configuration file)]
* `--account account`: Atlassian account to consider. If not set, defaulted to first configured account.  [default: (If not provided, defaulted with the first account configured in the configuration file)]
* `-t, --token token`: The Atlassian token to connect to the REST APIs.  [default: (If not provided, defaulted with the first token configured in the configuration file)]
* `--as-df / --as-dict`: Either get the result as a DataFrame or as a dict.  [default: (As DataFrame)]
* `-a, --alias alias`: The alias for the issues stored in context.  [default: (issues)]
* `--help`: Show this message and exit.

## `genov df-to-stdout`

The command prints into console the dataframe instance that is stored in context as ALIAS.

**Usage**:

```console
$ genov df-to-stdout [OPTIONS] alias
```

**Arguments**:

* `alias`: The alias for the dataframe stored in context to be printed.  [required]

**Options**:

* `--help`: Show this message and exit.

## `genov df-to-xlsx`

The command persists the DataFrame aliased ALIAS in the file system as FILE.

**Usage**:

```console
$ genov df-to-xlsx [OPTIONS] alias file
```

**Arguments**:

* `alias`: The alias for the dataframe stored in context to be printed.  [required]
* `file`: The file to export the dataframe.  [required]

**Options**:

* `--help`: Show this message and exit.

## `genov df-load-xlsx`

The command loads FILE from the file system in a DataFrame that is staged in context under alias DF ALIAS.
In case only a subset of columns is to be loaded, these can be listed under COLUMNS, so useless columns are
discarded.
KWARGS can be used to provide any technical parameters to the underlying function "pandas.read_excel".

**Usage**:

```console
$ genov df-load-xlsx [OPTIONS] file
```

**Arguments**:

* `file`: The MS Excel file to import as a dataframe.  [required]

**Options**:

* `-o, --df_output Df Alias`: Alias for the loaded Date Frame to be persisted in context.  [default: (If not provided, defaulted to &#x27;_df&#x27;.)]
* `-c, --columns columns`: Columns labels to load, others will be discarded.  [default: (By default, all columns are loaded.)]
* `-k, --kwargs kwargs`: Parameters for the underlying function pandas.read_excel.  [default: (By default, no parameter provided.)]
* `--help`: Show this message and exit.

## `genov dict-to-stdout`

The command prints into console the dictionary instance that is stored in context as ALIAS.

**Usage**:

```console
$ genov dict-to-stdout [OPTIONS] alias
```

**Arguments**:

* `alias`: The alias for the json stored in context to be printed.  [required]

**Options**:

* `--help`: Show this message and exit.

## `genov dict-to-json`

The command persists the dictionary instance that is stored in context as ALIAS in the file system as FILE.

**Usage**:

```console
$ genov dict-to-json [OPTIONS] alias file
```

**Arguments**:

* `alias`: The alias for the JSON stored in context to be printed.  [required]
* `file`: The file to export the dict.  [required]

**Options**:

* `--help`: Show this message and exit.

## `genov cfg-to-stdout`

The command prints into console the content of the configuration file stored under ~/.genov/genov.toml.

**Usage**:

```console
$ genov cfg-to-stdout [OPTIONS]
```

**Options**:

* `--help`: Show this message and exit.

## `genov dp`

The command instantiates an instance of Date Predictability that is then staged in context under DP ALIAS:
- It is initialized with the backlog of work items retrieved from context as a DataFrame aliased DF ALIAS.
- The columns&#x27; labels from the work items DataFrame are configured: KEY, LABEL, MEASURE, STATUS.

**Usage**:

```console
$ genov dp [OPTIONS] df_alias
```

**Arguments**:

* `df_alias`: The alias for the dataframe stored in context, used to compute predictability upon.  [required]

**Options**:

* `-a, --as dp_alias`: Alias for the Date Predictability instance that is persisted in context. The Date Predictability instance is later used to execute commands such as COMPUTE, DECOMPOSE or SLIDE.  [default: (If not provided, defaulted to _dp)]
* `-k, --key key`: Column in the dataframe that contains the KEYs.  [default: (If not provided, defaulted to &#x27;key&#x27;.)]
* `-l, --label label`: Column in the dataframe that contains the LABELs.  [default: (If not provided, defaulted to &#x27;label&#x27;.)]
* `-m, --measure measure`: Column(s) in the dataframe that contain (s) the MEASUREs, either SPOT (e.g. eta) or RANGE (e.g. &#x27;("from", "to")&#x27;, surrounded with simple quotes).  [default: (If not provided, defaulted to &#x27;measure&#x27;, aka SPOT.)]
* `-s, --status status`: Column in the dataframe that contains the STATUSes.  [default: (If not provided, defaulted to &#x27;status&#x27;.)]
* `--help`: Show this message and exit.

## `genov dp-asof`

**Usage**:

```console
$ genov dp-asof [OPTIONS] int_asof
```

**Arguments**:

* `int_asof`: The LABEL to truncate posterior observation.  [required]

**Options**:

* `-i, --dp_instance Dp Alias`: The Date Predictability instance to use.  [default: (If not provided, defaulted to &#x27;_dp&#x27;.)]
* `--help`: Show this message and exit.

## `genov dp-compute`

The command computes the Date Predictability by key, from the Date Predictability instance DP ALIAS that is
retrieved in context.
Eventually, computed Date Predictability is persisted in context as a DataFrame aliased DF ALIAS.

**Usage**:

```console
$ genov dp-compute [OPTIONS]
```

**Options**:

* `-i, --dp_instance dp_alias`: The Date Predictability instance to use.  [default: (If not provided, defaulted to &#x27;_dp&#x27;.)]
* `-o, --df_output Df Alias`: Alias for the computed Date Frame persisted in context.  [default: (If not provided, defaulted to &#x27;_df&#x27;.)]
* `--help`: Show this message and exit.

## `genov dp-decompose`

The command decomposes the Date Predictability for a given KEY, from the Date Predictability instance DP ALIAS
that is retrieved in context.
Eventually, decomposed Date Predictability is persisted in context as a DataFrame aliased DF ALIAS.

**Usage**:

```console
$ genov dp-decompose [OPTIONS]
```

**Options**:

* `--str-key TEXT`
* `-i, --dp_instance Dp Alias`: The Date Predictability instance to use.  [default: (If not provided, defaulted to &#x27;_dp&#x27;.)]
* `-o, --df_output Df Alias`: Alias for the computed Date Frame persisted in context.  [default: (If not provided, defaulted to &#x27;_df&#x27;.)]
* `--help`: Show this message and exit.

## `genov dp-slide`

The command slides the Date Predictability for a given KEY, from the Date Predictability instance DP ALIAS
that is retrieved in context.
Eventually, slid Date Predictability is persisted in context as a DataFrame aliased DF ALIAS.

**Usage**:

```console
$ genov dp-slide [OPTIONS]
```

**Options**:

* `--str-key TEXT`
* `-i, --dp_instance dp_lias`: The Date Predictability instance to use.  [default: (If not provided, defaulted to &#x27;_dp&#x27;.)]
* `-o, --df_output df_alias`: Alias for the computed Date Frame persisted in context.  [default: (If not provided, defaulted to &#x27;_df&#x27;.)]
* `--help`: Show this message and exit.

## `genov dp-describe`

The command describes the Date Predictability data, from the Date Predictability instance DP ALIAS
that is retrieved in context.
Eventually, Date Predictability statistics are persisted in context as a dict aliased DICT_ALIAS.

**Usage**:

```console
$ genov dp-describe [OPTIONS]
```

**Options**:

* `-i, --dp_instance Dp Alias`: The Date Predictability instance to use.  [default: (If not provided, defaulted to &#x27;_dp&#x27;.)]
* `-o, --dict_output dict_alias`: Alias for the computed statistics persisted in context.  [default: (If not provided, defaulted to &#x27;_dict&#x27;.)]
* `--help`: Show this message and exit.
