# ka_uts_dfr

# Overview

Dataframe 'Utilities'

# Installation

Package `ka_uts_dfr` can be installed from PyPI or Anaconda.

To install with `pip`:

    $ python -m pip install ka_uts_dfr

To install with `conda`:

    $ conda install -c conda-forge ka_uts_dfr

This requires that the `readme` extra is installed:

    $ python -m pip install ka_uts_dfr[readme]

# Package Modules

## Classification

The Modules of Package `ka_uts_dfr` could be classified into the following module classes:

1. Modules for pandas dataframe
2. Modules for polars dataframe

# Modules for Pandas Dataframe

> Modules for Pandas Dataframe
> Name | Type
> --- | ---
> pddf.py | Pandas Dataframe

## pddf.py

The Module `pddf.py` contains a single static classes `PdDf`.

### pddf.py Class: PdDf

The static Class `PdDf` is used to manage Pandas Dataframes; it contains the subsequent methods.

#### PdDf Methods

> Methods of static class PdDf
> Name | Description
> --- | ---
> sh_d_aod | show dictionary of array of dictionaries.
> sh_d_pddf | show dictionary of pandas dataframes.
> pivot_table | create pandas dataframe pivot table. The pivot rules are defined by a pivot dictionary.
> filter | Filter pandas dataframe. The filteris defined by filter dictionary
> set_ix_drop_col_filter | set index and drop column filter
> format-leading_zeros | format pandas dataframe columns with leading zeros
> format-as-date | format pandas dataframe columns as date

#### PdDf Method: sh_d_aod

##### Parameter

> Parameter of PdDf method sh_d_aod
> Name | Type | Description
> --- | --- | ---
> df | TyPdDf | Pandas Datafame
> key | str | Keyword arguments

##### Return Value

> Return Value of PdDf method sh_d_aod
> Name | Type | Description
> --- | --- | ---
> d_aod | TyDoAoD | dictionary of array of dictionaries

#### PdDf Method: sh_d_pddf

##### Parameter

> Parameter of PdDf method sh_d_pddf
> Name | Type | Description
> --- | --- | ---
> cls | class | current class
> df | TyPdDf | Pandas Datafame
> key | str | keyword arguments

##### Return Value

> Return Value of PdDf method sh_d_pddf
> Name | Type | Description
> --- | --- | ---
> d_df | TyDoPdDf | dictionary of pandas dataframes

#### PdDf Method: pivot_table

##### Parameter

> Parameter of PdDf method pivot_table
> Name | Type | Description
> --- | --- | ---
> cls | class | current class
> df | TyPdDf | pandas datafame
> d_pv | TyDic | pivot table definition dictionary

##### Return Value

> Return Value of PdDf method pivot_table
> Name | Type | Description
> --- | --- | ---
> dfpv | TyPdDf | pandas dataframe pivot table

#### PdDf Method: filter

##### Parameter

> Parameter of PdDf method filter
> Name | Type | Description
> --- | --- | ---
> cls | class | current class
> df | TyPdDf | pandas datafame
> d_filter | TyDic | filter definition dictionary
> relation | TyStr | filter relation

##### Return Value

> Return Value of PdDf method filter
> Name | Type | Description
> --- | --- | ---
> df_new | TyPdDf | filtered pandas datafame

#### PdDf Method: set_ix_drop_col_filter

##### Parameter

> Parameter of PdDf method set_ix_drop_col_filter
> Name | Type | Description
> --- | --- | ---
> cls | class | current class
> df | TyPdDf | pandas datafame
> d_filter | TyDic | filter definition dictionary
> relation | str | filter relation

##### Return Value

> Return Value of PdDf method set_ix_drop_col_filter
> Name | Type | Description
> --- | --- | ---
> df_new | TyPdDf | filtered pandas datafame

#### PdDf Module: format_leading_zeros

##### Parameter

> Parameter of PdDf method format_leading_zeros
> Name | Type | Description
> --- | --- | ---
> cls | class | current class
> df | TyPdDf | pandas datafame
> d_filter | TyDic | filter definition dictionary
> relation | str | filter relation

##### Return Value

> Return Value of PdDf method format_leading_zeros
> Name | Type | Description
> --- | --- | ---
> df_new | TyPdDf | filtered pandas datafame

#### PdDf Method: format_as_date

##### Parameter

> Parameter of PdDf method format_as_date
> Name | Type | Description
> --- | --- | ---
> cls | class | current class
> df | TyPdDf | pandas datafame
> d_filter | TyDic | filter definition dictionary
> relation | str | filter relation

##### Return Value

> Return Values of PdDf methodR ormat_as_date
> Name | Type | Description
> --- | --- | ---
> df_new | TyPdDf | filtered pandas datafame

# Modules for Polars Dataframe

> Modules for Polars Dataframe
> Module | Classes
> --- | ---
> Name|Type | Name | Type | Description
> pldf | Polars Dataframe | PdDf | Static | Manage Polars Dataframes

## pldf.py

The Module `pldf` contains a single static class `PLDF`.

### PlDf

The static Class `PlDf` contains the subsequent methods.

#### PlDf Methods

> pldf Methods
> Name | Description
> --- | ---
> filter | Filter polars dataframe using the given statement.
> pivot | Create polars dataframe pivot table. The pivot rules are defined by the given pivot dictionary.
> pivot_filter | Filter polars dataframe using the given statement and create polars dataframe pivot table from filtered dataframe. The pivot rules are defined by the given pivot dictionary.
> to_aod | create pandas dataframe pivot table. The pivot rules are defined by pivot dictionary
> to_doa | create pandas dataframe pivot table. The pivot rules are defined by pivot dictionary

#### PlDf Method: filter

##### Parameter

> Parameter of PlDf method filter
> Name | Type | Description
> --- | --- | ---
> cls | class | current class
> df | TyPdDf | polars datafame
> stmt | TyStmt | filter statement

##### Return Value

> Return Value of PlDf method filter
> Name | Type | Description
> --- | --- | ---
> df_new | TyPlDf | filtered polars datafame

#### PlDf Method: pivot

##### Parameter

> Parameter of P.Df method pivot
> Name | Type | Description
> --- | --- | ---
> cls | class | current class
> df | TyPlDf | polars datafame
> d_pv | TyDic | pivot table definition dictionary

##### Return Value

> Return value of PdDf method pivot
> Name | Type | Description
> --- | --- | ---
> dfpv | TyPlDf | polars dataframe pivot table

#### PlDf Method: pivot_filter

##### Parameter

> Parameter of PdDf method pivot_filter
> Name | Type | Description
> --- | --- | ---
> cls | class | current class
> df | TyPlDf | polars datafame
> d_pv | TyDic | pivot table definition dictionary
> stmt | TyStmt | filter statement

##### Return Value

> Return value of PlDf method pivot_gilter
> Name | Type | Description
> --- | --- | ---
> dfpv | TyPlDf | polars dataframe pivot table

#### PlDf Method: to_aod

##### Parameter

> Parameter of PdDf method to_aod
> Name | Type | Description
> --- | --- | ---
> df | TyPlDf | polars datafame

##### Return Value

> Return value of PlDf method to_aod
> Name | Type | Description
> --- | --- | ---
> aod | TyAoD | Array of Dictionaries

#### PlDf Method: to_doa

##### Parameter

> Parameter of PdDf method to_doa
> Name | Type | Description
> --- | --- | ---
> df | TyPlDf | polars datafame

##### Return Value

> Return value of PlDf method to_doa
> Name | Type | Description
> --- | --- | ---
> doa | TyDoA | Dictionary of Arrays

# Appendix

## Package Logging

### Description

The Standard or user specifig logging is carried out by the log.py module of the logging package ka_uts_log using the configuration files ka_std_log.yml or ka_usr_log.yml in the configuration directory cfg of the logging package ka_uts_log. The Logging configuration of the logging package could be overriden by yaml files with the same names in the configuration directory cfg of the application packages.

### Log message types

Logging defines log file path names for the following log message types: .

1. debug
2. info
3. warning
4. error
5. critical

#### Application parameter for logging

> Application parameter used in log naming
> Name | Decription | Values | Example
> --- | --- | --- | ---
> dir_dat | Application data directory |  | /otev/data
> tenant | Application tenant name |  | UMH
> package | Application package name |  | otev_xls_srr
> cmd | Application command |  | evupreg
> pid | Process ID |  | æevupreg
> log_ts_type | Timestamp type used in logging files|ts, dt | ts, dt | ts
> log_sw_single_dir | Enable single log directory or multiple log directories | True, False | True

#### Log type and Log directories

Single or multiple Application log directories can be used for each message type:

> Log types and directoriesg
> Log type | Log directory
> --- | ---
> long | short | multiple | single
> debug | dbqs | dbqs | logs
> info | infs | infs | logs
> warning | wrns | wrns | logs
> error | errs | errs | logs
> critical | crts | crts | logs

#### Log files naming

##### Conventions

> Naming conventions for logging file paths
> Type | Directory | File
> --- | --- | ---
> debug | /<dir_dat>/<tenant>/RUN/<package>/<cmd>/<Log directory> | <Log type>_<ts>_<pid>.log
> info | /<dir_dat>/<tenant>/RUN/<package>/<cmd>/<Log directory> | <Log type>_<ts>_<pid>.log
> warning | /<dir_dat>/<tenant>/RUN/<package>/<cmd>/<Log directory> | <Log type>_<ts>_<pid>.log
> error | /<dir_dat>/<tenant>/RUN/<package>/<cmd>/<Log directory> | <Log type>_<ts>_<pid>.log
> critical | /<dir_dat>/<tenant>/RUN/<package>/<cmd>/<Log directory> | <Log type>_<ts>_<pid>.log

##### Examples

> Naming examples for logging file paths
> Type | Directory | File
> --- | --- | ---
> debug | /data/otev/umh/RUN/otev_xls_srr/evupreg/logs | debs_1737118199_9470.log
> info | /data/otev/umh/RUN/otev_xls_srr/evupreg/logs | infs_1737118199_9470.log
> warning | /data/otev/umh/RUN/otev_xls_srr/evupreg/logs | wrns_1737118199_9470.log
> error | /data/otev/umh/RUN/otev_xls_srr/evupreg/logs | errs_1737118199_9470.log
> critical | /data/otev/umh/RUN/otev_xls_srr/evupreg/logs | crts_1737118199_9470.log

## Python Terminology

### Python package

#### Overview

> Python package
> Name | Definition
> --- | ---
> Python package | Python packages are directories that contains the special module `__init__.py` and other modules, packages files or directories.
> Python sub-package | Python sub-packages are python packages which are contained in another pyhon package.

### Python package sub-directories

#### Overview

> Python package sub-directories
> Name | Definition
> --- | ---
> Python package sub-directory | Python packages sub-directories are directories contained in python packages.
> Special Python package sub-directory | Special Python package sub-directories are python package sub-directories with with a special meaning

### Special python package sub-directories

#### Overview

> Special python sun-directories
> Name | Description
> --- | ---
> data | Directory for package data files.
> cfg | Directory for package configuration files.

### Python package files

#### Overview

> Python package files
> Name | Definition
> --- | ---
> Python package files | Python packages are files within a python package.
> Special python package files | Special python package files are package files which are not modules and used as python marker files like `__init__.py`
> Python package module | Python modules are files with suffix `.py`; they could be empty or contain python code; other modules can be imported into a module.
> Special python package module | Special python modules like `__init__.py` or `main.py` are python modules with special names and functionality.

#### Special python package files

Overview °°°°°°°°

> Special python package files
> Name | Type | Description
> --- | --- | ---
> py.typed | Type checking marker file | The `py.typed` file is a marker file used in Python packages to indicate that the package supports type checking. This is a part of the PEP 561 standard, which provides a standardized way to package and distribute type information in Python.

#### Special python package modules

Overview °°°°°°°°

> Special Python package modules
> Name | Type | Description
> --- | --- | ---
> __init__.py | Package directory marker file | The dunder (double underscore) module `__init__.py` is used to execute initialisation code or mark the directory it contains as a package. The Module enforces explicit imports and thus clear namespace use and call them with the dot notation.
> __main__.py | entry point for the package | The dunder module `__main__.py` serves as an entry point for the package. The module is executed when the package is called by the interpreter with the command python -m <package name>.
> __version__.py | Version file | The dunder module `__version__.py` consist of assignment statements used in Versioning.

### Python elements

Overview °°°°°°°°

> Python elements
> Python method | Python methods are python functions defined in python modules.
> Special python method | Special python methods are python functions with special names and functionalities.
> Python class | Python classes are defined in python modules.
> Python class method | Python class methods are python methods defined python classes.

#### Special python methods

Overview °°°°°°°°

> Special python methods
> Name | Type | Description
> --- | --- | ---
> __init__ | class object constructor method | The special method `__init__` is called when an instance (object) of a class is created; instance attributes can be defined and initalized in the method.

## Table of Contents

Table of Content

- Overview
- Installation
- Package Modules
  - Classification
- Modules for Pandas Dataframe
  - pddf.py
    - pddf.py Class: PdDf
      - PdDf Methods
      - PdDf Method: sh_d_aod
        - Parameter
        - Return Value
      - PdDf Method: sh_d_pddf
        - Parameter
        - Return Value
      - PdDf Method: pivot_table
        - Parameter
        - Return Value
      - PdDf Method: filter
        - Parameter
        - Return Value
      - PdDf Method: set_ix_drop_col_filter
        - Parameter
        - Return Value
      - PdDf Module: format_leading_zeros
        - Parameter
        - Return Value
      - PdDf Method: format_as_date
        - Parameter
        - Return Value
- Modules for Polars Dataframe
  - pldf.py
    - PlDf
      - PlDf Methods
      - PlDf Method: filter
        - Parameter
        - Return Value
      - PlDf Method: pivot
        - Parameter
        - Return Value
      - PlDf Method: pivot_filter
        - Parameter
        - Return Value
      - PlDf Method: to_aod
        - Parameter
        - Return Value
      - PlDf Method: to_doa
        - Parameter
        - Return Value
- Appendix
  - Package Logging
    - Description
    - Log message types
      - Application parameter for logging
      - Log type and Log directories
      - Log files naming
        - Conventions
        - Examples
  - Python Terminology
    - Python package
      - Overview
    - Python package sub-directories
      - Overview
    - Special python package sub-directories
      - Overview
    - Python package files
      - Overview
      - Special python package files
      - Special python package modules
    - Python elements
      - Special python methods
  - Table of Contents