README.rst

# ka_uts_dfr

## Overview

Dataframe (Pandas, Polars) Utilities

## Installation

Package `ka_uts_dfr` can be installed from PyPI or Anaconda.

To install with `pip`:

    $ python -m pip install ka_uts_dfr

To install with `conda`:

    $ conda install -c conda-forge ka_uts_dfr

This requires that the `readme` extra is installed:

    $ python -m pip install ka_uts_dfr[readme]

## Package Modules

### Classification

The Modules of Package `ka_uts_dfr` could be classified into the following module classes:

1. Modules for pandas dataframe
2. Modules for polars dataframe

## Modules for Pandas Dataframe

> Modules for Pandas Dataframe
> Name | Type
> --- | ---
> pddf.py | Pandas Dataframe

### pddf.py

The Module `pddf.py` contains a single static classes `PdDf`.

#### pddf.py Class: PdDf

The static Class `PdDf` is used to manage Pandas Dataframes; it contains the subsequent methods.

##### PdDf Methods

> Methods of static class PdDf
> Name | Description
> --- | ---
> sh_d_aod | show dictionary of array of dictionaries.
> sh_d_pddf | show dictionary of pandas dataframes.
> pivot_table | create pandas dataframe pivot table. The pivot rules are defined by a pivot dictionary.
> filter | Filter pandas dataframe. The filteris defined by filter dictionary
> set_ix_drop_col_filter | set index and drop column filter
> format-leading_zeros | format pandas dataframe columns with leading zeros
> format-as-date | format pandas dataframe columns as date

##### PdDf Method: sh_d_aod

###### Parameter

> Parameter of PdDf method sh_d_aod
> Name | Type | Description
> --- | --- | ---
> df | TyPdDf | Pandas Datafame
> key | str | Keyword arguments

###### Return Value

> Return Value of PdDf method sh_d_aod
> Name | Type | Description
> --- | --- | ---
> d_aod | TyDoAoD | dictionary of array of dictionaries

##### PdDf Method: sh_d_pddf

###### Parameter

> Parameter of PdDf method sh_d_pddf
> Name | Type | Description
> --- | --- | ---
> cls | class | current class
> df | TyPdDf | Pandas Datafame
> key | str | keyword arguments

###### Return Value

> Return Value of PdDf method sh_d_pddf
> Name | Type | Description
> --- | --- | ---
> d_df | TyDoPdDf | dictionary of pandas dataframes

##### PdDf Method: pivot_table

###### Parameter

> Parameter of PdDf method pivot_table
> Name | Type | Description
> --- | --- | ---
> cls | class | current class
> df | TyPdDf | pandas datafame
> d_pv | TyDic | pivot table definition dictionary

###### Return Value

> Return Value of PdDf method pivot_table
> Name | Type | Description
> --- | --- | ---
> dfpv | TyPdDf | pandas dataframe pivot table

##### PdDf Method: filter

###### Parameter

> Parameter of PdDf method filter
> Name | Type | Description
> --- | --- | ---
> cls | class | current class
> df | TyPdDf | pandas datafame
> d_filter | TyDic | filter definition dictionary
> relation | TyStr | filter relation

###### Return Value

> Return Value of PdDf method filter
> Name | Type | Description
> --- | --- | ---
> df_new | TyPdDf | filtered pandas datafame

##### PdDf Method: set_ix_drop_col_filter

###### Parameter

> Parameter of PdDf method set_ix_drop_col_filter
> Name | Type | Description
> --- | --- | ---
> cls | class | current class
> df | TyPdDf | pandas datafame
> d_filter | TyDic | filter definition dictionary
> relation | str | filter relation

###### Return Value

> Return Value of PdDf method set_ix_drop_col_filter
> Name | Type | Description
> --- | --- | ---
> df_new | TyPdDf | filtered pandas datafame

##### PdDf Module: format_leading_zeros

###### Parameter

> Parameter of PdDf method format_leading_zeros
> Name | Type | Description
> --- | --- | ---
> cls | class | current class
> df | TyPdDf | pandas datafame
> d_filter | TyDic | filter definition dictionary
> relation | str | filter relation

###### Return Value

> Return Value of PdDf method format_leading_zeros
> Name | Type | Description
> --- | --- | ---
> df_new | TyPdDf | filtered pandas datafame

##### PdDf Method: format_as_date

###### Parameter

> Parameter of PdDf method format_as_date
> Name | Type | Description
> --- | --- | ---
> cls | class | current class
> df | TyPdDf | pandas datafame
> d_filter | TyDic | filter definition dictionary
> relation | str | filter relation

###### Return Value

> Return Values of PdDf methodR ormat_as_date
> Name | Type | Description
> --- | --- | ---
> df_new | TyPdDf | filtered pandas datafame

## Modules for Polars Dataframe

> Modules for Polars Dataframe
> Module | Classes
> --- | ---
> Name|Type | Name | Type | Description
> pldf | Polars Dataframe | PdDf | Static | Manage Polars Dataframes

### pldf.py

The Module `pldf` contains a single static class `PLDF`.

#### PlDf

The static Class `PlDf` contains the subsequent methods.

##### PlDf Methods

> pldf Methods
> Name | Description
> --- | ---
> filter | Filter polars dataframe using the given statement.
> pivot | Create polars dataframe pivot table. The pivot rules are defined by the given pivot dictionary.
> pivot_filter | Filter polars dataframe using the given statement and create polars dataframe pivot table from filtered dataframe. The pivot rules are defined by the given pivot dictionary.
> to_aod | create pandas dataframe pivot table. The pivot rules are defined by pivot dictionary
> to_doa | create pandas dataframe pivot table. The pivot rules are defined by pivot dictionary

##### PlDf Method: filter

###### Parameter

> Parameter of PlDf method filter
> Name | Type | Description
> --- | --- | ---
> cls | class | current class
> df | TyPdDf | polars datafame
> stmt | TyStmt | filter statement

###### Return Value

> Return Value of PlDf method filter
> Name | Type | Description
> --- | --- | ---
> df_new | TyPlDf | filtered polars datafame

##### PlDf Method: pivot

###### Parameter

> Parameter of P.Df method pivot
> Name | Type | Description
> --- | --- | ---
> cls | class | current class
> df | TyPlDf | polars datafame
> d_pv | TyDic | pivot table definition dictionary

###### Return Value

> Return value of PdDf method pivot
> Name | Type | Description
> --- | --- | ---
> dfpv | TyPlDf | polars dataframe pivot table

##### PlDf Method: pivot_filter

###### Parameter

> Parameter of PdDf method pivot_filter
> Name | Type | Description
> --- | --- | ---
> cls | class | current class
> df | TyPlDf | polars datafame
> d_pv | TyDic | pivot table definition dictionary
> stmt | TyStmt | filter statement

###### Return Value

> Return value of PlDf method pivot_gilter
> Name | Type | Description
> --- | --- | ---
> dfpv | TyPlDf | polars dataframe pivot table

##### PlDf Method: to_aod

###### Parameter

> Parameter of PdDf method to_aod
> Name | Type | Description
> --- | --- | ---
> df | TyPlDf | polars datafame

###### Return Value

> Return value of PlDf method to_aod
> Name | Type | Description
> --- | --- | ---
> aod | TyAoD | Array of Dictionaries

##### PlDf Method: to_doa

###### Parameter

> Parameter of PdDf method to_doa
> Name | Type | Description
> --- | --- | ---
> df | TyPlDf | polars datafame

###### Return Value

> Return value of PlDf method to_doa
> Name | Type | Description
> --- | --- | ---
> doa | TyDoA | Dictionary of Arrays

# Appendix

## Package Logging

### Description

The Standard or user specifig logging is carried out by the log.py module of the logging package ka_uts_log using the standard- or user-configuration files in the logging package configuration directory:

- <logging package directory>/cfg/ka_std_log.yml,
- <logging package directory>/cfg/ka_usr_log.yml.

The Logging configuration of the logging package could be overriden by yaml files with the same names in the application package- or application data-configuration directories:

- <application package directory>/cfg
- <application data directory>/cfg.

### Log message types

Logging defines log file path names for the following log message types: .

1. debug
2. info
3. warning
4. error
5. critical

#### Log types and Log directories

Single or multiple Application log directories can be used for each message type:

> Log types and directoriesg
> Log type | Log directory
> --- | ---
> long | short | multiple | single
> debug | dbqs | dbqs | logs
> info | infs | infs | logs
> warning | wrns | wrns | logs
> error | errs | errs | logs
> critical | crts | crts | logs

#### Application parameter for logging

> Application parameter used in log naming
> Name | Decription | Values | Example
> --- | --- | --- | ---
> dir_dat | Application data directory |  | /otev/data
> tenant | Application tenant name |  | UMH
> package | Application package name |  | otev_xls_srr
> cmd | Application command |  | evupreg
> pid | Process ID |  | 681025
> log_ts_type | Timestamp type used in logging files|ts, dt | ts, dt' | ts
> log_sw_single_dir | Enable single log directory or multiple log directories | True, False | True

#### Log files naming

##### Naming Conventions

> Naming conventions for logging file paths
> Type | Directory | File
> --- | --- | ---
> debug | /<dir_dat>/<tenant>/RUN/<package>/<cmd>/<Log directory> | <Log type>_<ts>_<pid>.log
> info | /<dir_dat>/<tenant>/RUN/<package>/<cmd>/<Log directory> | <Log type>_<ts>_<pid>.log
> warning | /<dir_dat>/<tenant>/RUN/<package>/<cmd>/<Log directory> | <Log type>_<ts>_<pid>.log
> error | /<dir_dat>/<tenant>/RUN/<package>/<cmd>/<Log directory> | <Log type>_<ts>_<pid>.log
> critical | /<dir_dat>/<tenant>/RUN/<package>/<cmd>/<Log directory> | <Log type>_<ts>_<pid>.log

##### Naming Examples

> Naming examples for logging file paths
> Type | Directory | File
> --- | --- | ---
> debug | /data/otev/umh/RUN/otev_xls_srr/evupreg/logs | debs_1737118199_9470.log
> info | /data/otev/umh/RUN/otev_xls_srr/evupreg/logs | infs_1737118199_9470.log
> warning | /data/otev/umh/RUN/otev_xls_srr/evupreg/logs | wrns_1737118199_9470.log
> error | /data/otev/umh/RUN/otev_xls_srr/evupreg/logs | errs_1737118199_9470.log
> critical | /data/otev/umh/RUN/otev_xls_srr/evupreg/logs | crts_1737118199_9470.log

## Python Terminology

### Python Packages

#### Overview

> Python Packages Overview
> Name | Definition
> --- | ---
> Python package | Python packages are directories that contains the special module `__init__.py` and other modules, packages files or directories.
> Python sub-package | Python sub-packages are python packages which are contained in another pyhon package.
> Python package sub-directory | directory contained in a python package.
> Python package special sub-directory | Python package sub-directories with a special meaning like data or cfg

#### Examples

> Python Package sub-directory-Examples
> Name | Description
> --- | ---
> bin | Directory for package scripts.
> cfg | Directory for package configuration files.
> data | Directory for package data files.
> service | Directory for systemd service scripts.

### Python package files

#### Overview

> Python package overview files
> Name | Definition
> --- | ---
> Python package files | Files within a python package.
> Special python package files | Package files which are not modules and used as python and used as python marker files like `__init__.py`.
> Python package module | Files with suffix `.py`; they could be empty or contain python code; other modules can be imported into a module.
> Special python package module | Modules like `__init__.py` or `main.py` with special names and functionality.

#### Examples

> Python package examples files
> Name | Type | Description
> --- | --- | ---
> py.typed | Type checking marker file | The `py.typed` file is a marker file used in Python packages to indicate that the package supports type checking. This is a part of the PEP 561 standard, which provides a standardized way to package and distribute type information in Python.
> __init__.py | Package directory marker file | The dunder (double underscore) module `__init__.py` is used to execute initialisation code or mark the directory it contains as a package. The Module enforces explicit imports and thus clear namespace use and call them with the dot notation.
> __main__.py | entry point for the package | The dunder module `__main__.py` serves as an entry point for the package. The module is executed when the package is called by the interpreter with the command python -m <package name>.
> __version__.py | Version file | The dunder module `__version__.py` consist of assignment statements used in Versioning.

### Python methods

#### Overview

> Python methods overview
> Name | Description
> --- | ---
> Python method | Python functions defined in python modules.
> Special python method | Python functions with special names and functionalities.
> Python class | Classes defined in python modules.
> Python class method | Python methods defined in python classes

#### Examples

> Python methods examples
> Name | Type | Description
> --- | --- | ---
> __init__ | class object constructor method | The special method `__init__` is called when an instance (object) of a class is created; instance attributes can be defined and initalized in the method.

# Table of Contents

Table of Content

- ka_uts_dfr
  - Overview
  - Installation
  - Package Modules
    - Classification
  - Modules for Pandas Dataframe
    - pddf.py
      - pddf.py Class: PdDf
        - PdDf Methods
        - PdDf Method: sh_d_aod
          - Parameter
          - Return Value
        - PdDf Method: sh_d_pddf
          - Parameter
          - Return Value
        - PdDf Method: pivot_table
          - Parameter
          - Return Value
        - PdDf Method: filter
          - Parameter
          - Return Value
        - PdDf Method: set_ix_drop_col_filter
          - Parameter
          - Return Value
        - PdDf Module: format_leading_zeros
          - Parameter
          - Return Value
        - PdDf Method: format_as_date
          - Parameter
          - Return Value
  - Modules for Polars Dataframe
    - pldf.py
      - PlDf
        - PlDf Methods
        - PlDf Method: filter
          - Parameter
          - Return Value
        - PlDf Method: pivot
          - Parameter
          - Return Value
        - PlDf Method: pivot_filter
          - Parameter
          - Return Value
        - PlDf Method: to_aod
          - Parameter
          - Return Value
        - PlDf Method: to_doa
          - Parameter
          - Return Value
- Appendix
  - Package Logging
    - Description
    - Log message types
      - Log types and Log directories
      - Application parameter for logging
      - Log files naming
        - Naming Conventions
        - Naming Examples
  - Python Terminology
    - Python Packages
      - Overview
      - Examples
    - Python package files
      - Overview
      - Examples
    - Python methods
      - Overview
      - Examples
- Table of Contents