# ka_uts_arr

# Overview

Array Utilities

# Installation

Package `ka_uts_array` can be installed from PyPI or Anaconda.

To install with `pip`:

    $ python -m pip install ka_uts_arr

To install with `conda`:

    $ conda install -c conda-forge ka_uts_arr

# Package logging

(c.f.: Appendix: Package Logging)

# Package files

## Classification

The Files of Package `ka_uts_arr` could be classified into the follwing file types (c.f.: Appendix: Python Terminology):

1. Special files
   1. py.typed
2. Special modules
   1. __init__.py
   2. __version__.py
3. Modules
   1. Modules for dictionaries
      1. dic.py
   2. Modules for dictionaries of arrays
      1. doaod.py
      2. doa.py
   3. Modules for dictionaries of callables
      1. doc.py*
   4. Modules for dictionaries of dataframes
      1. dopddf.py

## Package Modules

### Overview

The Modules of Package `ka_uts_arr` could be classified into the follwing module types:

1. Modules for arrays
2. Modules for arrays of arrays
3. Modules for arrays of basic objects

# Modules for arrays

The Module type `Modules for arrays` contains only the module `arr.py`.

## Module: arr.py

The Module `arr.py` contains only the static class `Arr`.

### Class: Arr

The Class `Arr` contains the following methods:

#### Arr Methods

> Arr methods
> Name | Short description
> --- | ---
> append | Append item to the array
> append_unique | Append item to the array if the item is not in the array.
> apply_function | Apply function with the keyword arguments to all non empty array elements.
> apply_replace | Replace source by target to all array elements.
> apply_str | Apply function str to all non empty array elements.
> encode | Join array elements with blank separator and encode result string.
> ex_intersection | Intersection of first array with second array.
> extend | Extend first array with second array.
> get_key_value | Get next array item value without line feed for the given index or the given default value if the item value is identical to the given value without line feeds.
> get_item | Extend array of dicts. with non empty dict.
> get_text | Extend array of dicts. with non empty dict.
> get_text_spli | Extend array of dicts. with non empty dict.
> intersection | Extend array of dicts. with non empty dict.
> is_empty | Extend array of dicts. with non empty dict.
> is_not_empty | Extend array of dicts. with non empty dict.
> join_not_none | Extend array of dicts. with non empty dict.
> length | Extend array of dicts. with non empty dict.
> merge | Extend array of dicts. with non empty dict.
> sh_dic_from_keys_values | Extend array of dicts. with non empty dict.
> sh_dic_zip | Join elements of array of dicts.
> sh_item | Show True if an element exists in the array
> sh_item_if | Show True if an element exists in the array
> sh_item_lower | Show True if an element exists in the array
> sh_item_str | Show True if an element exists in the array
> sh_item0 | Deduplicate array of dicts.
> sh_item0_if | Deduplicate array of dicts.
> sh_subarray | Deduplicate array of dicts.
> to_dic | Show arr. of arrays created from arr. of dict. by using any key- and all value-arrays
> yield_items | Convert array of dictionaries to array of arrays controlled by key- and value-switch.

# Modules for array of arrays

The Module type `Modules for array of arrays` contains only the module `aoa.py`.

## Module: aoa.py

The Module `aoa.py` contains only the static class `AoA`.

### aoa.py Class: AoA

The static Class `AoA` contains the subsequent methods.

#### AoA Methods

> AoA Methods
> Name | Short description
> --- | ---
> concatinate | Concatinate all arrays of array of arrays
> csv_writerows | Write array of arrays to csv file
> nvl | Replace empty array of arrays
> to_aod | Convert array of arrays to array of dictionaries using an array of keys
> to_arr_from_2cols | Convert array of arrays to array using a 2-dimensional index array
> to_doa_from_2cols | Convert array of arrays to dictionary of arrays using a 2-dimensionl index array
> to_dic_from_2cols | Convert array of arrays to dictionary by using a 2-dimensional index array

#### AoA Method: concatinate

##### Description

Concatinate all arrays of array of arrays.

##### Parameter

> Parameter of: AoA Method: concatinate
> Name | Type | Default | Description
> --- | --- | --- | ---
> aoa | TyAoA |  | Array of arrays

##### Return Value

> Return Value of: AoA Method: concatinate
> Name | Type | Description
> --- | --- | ---
> arr_new | TyArr | new array

#### AoA Method: csv_writerows

##### Description

Write Array of Arrays to Csv file defined by the path string using the function "writerows" of module "csv".

##### Parameter

> Parameter of: AoA Method: csv_writerows
> Name | Type | Description
> --- | --- | ---
> aoa | TyAoA | Array of arrays
> path | TyPath | Path string
> kwargs | TyDic | Keyword aruments

##### Return Value

> Return Value of: AoA Method: csv_writerows
> Name | Type | Description
> --- | --- | ---
> | None |

#### AoA Method: nvl

##### Description

Return the empty array if the Array of Arrays is None.

##### Parameter

> Parameter of: AoA Method nvl
> Name | Type | Default | Description
> --- | --- | --- | ---
> aoa | TyAoA |  | Array of arrays

##### Return Value

> Return Value of: AoA Method: nvl
> Name | Type | Description
> --- | --- | ---
> aoa_new | TyAoA | new Array of arrays

#### AoA Method: to_aod

##### Description

Convert array of arrays to array of Dictionaries.

##### Parameter

> Parameter of: AoA Method: to_aod
> Name | Type | Default | Description
> --- | --- | --- | ---
> aoa | TyAoA |  | Array of arrays
> keys | TyArr |  | Array of keys

##### Return Value

> Return Value of: AoA Method: to_aod
> Name | Type | Description
> --- | --- | ---
> aod | TyAoD | array of dictionaries

#### AoA Method: to_arr_from_2cols

##### Description

Convert Array of Arrays to unique array with distinct elements by selecting 2 columns of each Array as elements of the new array using a 2-dimensional index-array.

##### Parameter

> Parameter of: AoA Method: to_arr_from_2cols
> Name | Type | Default | Description
> --- | --- | --- | ---
> aoa | TyAoA |  | Array of arrays
> a_ix | TyAoI |  | Array of integer

##### Return Value

> Return Value of: AoA Method: to_arr_from_2cols
> Name | Type | Description
> --- | --- | ---
> arr | TyArr | Array

#### AoA Method: to_doa_from_2cols

##### Description

Convert array of arrays to dictionary of unique arrays (array with distinct elements)

1. Select 2 columns of each array as key-, value-candidates of the new dictionary using a 2-dimensional index-array.
2. If the new key exists then the new value extends the key value as unique array,

# otherwise
  the new value is assigned as unique array to the key.

##### Parameter

> Parameter of: AoA Method: to_doa_from_2cols
> Name | Type | Default | Description
> --- | --- | --- | ---
> aoa | TyAoA |  | Array of arrays
> a_ix | TyAoI |  | Array of integer

##### Return Value

> Return Value of: AoA Method: to_doa_from_2cols
> Name | Type | Description
> --- | --- | ---
> doa | TyDoA | Dictionry of arrays

#### AoA Method: to_dic_from_2cols

##### Description

Convert array of arrays to dictionary by selecting 2 columns of each array as key-, value-candidates of the new dictionary if the key is not none using a 2-dimensional index-array.

##### Parameter

> Parameter of: AoA Method: to_dic_from_2cols
> Name | Type | Default | Description
> --- | --- | --- | ---
> aoa | TyAoA |  | Array of arrays
> a_ix | TyAoI |  | Array of integer

##### Return Value

> Return Value of: AoA Method: to_dic_from_2col*
> Name | Type | Description
> --- | --- | ---
> dic | TyDic | Dictionary

# Modules for array of objects

> Modules-for-array-of-dictionaries
> Name | Description
> --- | ---
> aoo.py | Array of objects

## Module: aoo.py

The Module `aoo.py` contains the single static class `AoO`;

### aoo.py Class: AoO

#### Methods

> AoO Methods
> Name | short Description
> --- | ---
> to_unique | Concatinate array arrays

#### AoO Method: to_unique

Deduplicate array of objects

##### Parameter

> Parameter of: AoO Method: to_unique
> Name | Type | Description
> --- | --- | ---
> aoo | TyAoO | Array of objects

##### Return Value

> Return Value of: AoO Method: to_unique
> Name | Type | Description
> --- | --- | ---
> aoo_new | TyAoO | New array of objects

## Module: aopath.py

The Module `aopath.py` contains only thestatic class `AoPath`;

### Class: AoPath

#### Methods

> AoPath methods
> Name | short Description
> --- | ---
> join | Join array of paths using the os separator
> sh_a_path | Show array of paths for path template.
> sh_a_path_by_tmpl | Convert array of path template keys and kwargs to array of paths.
> sh_path_tmpl | Convert array of path templates to path template.
> yield_path_kwargs | yield path for path-array and kwargs.
> yield_path_kwargs_new | yield path from dictionary- and path-array and modified kwargs by dictionary item
> yield_path_item_kwargs | yield path from path-array, item from array and kwargs.
> yield_path_item_kwargs_new | yield path from path-array, item from array and modified kwargs by dictionary item.

#### AoPath Method: join

1. Convert array of paths (1.argument) by striping the leading or trailing os separator.
2. join the converted array of paths.

##### Parameter

> Parameter of: AoPath Method: join
> Name | Type | Default | Description
> --- | --- | --- | ---
> aopath | TyAoPath |  | array of paths

##### Return Value

> Return Value of: AoPath Method: join
> Name | Type | Description
> --- | --- | ---
> path | TyPath | Path

#### AoPath Method: sh_a_path

Convert path template to array of paths using glob function of module glob.py.

##### Parameter

> Parameter of: AoPath Method: sh_a_path
> Name | Type | Default | Description
> --- | --- | --- | ---
> path | TyPath |  | Path

##### Return Value

> Return Value of: AoPath Method: sh_a_path
> Name | Type | Description
> --- | --- | ---
> a_path | TyAoPath | Array of paths

#### AoPath Method: sh_a_path_by_tmpl

1. Select array of path templates from keyword arguments (1.arguments) using the parameter
   - array of path template keys (1.argument);
2. join the array of path templates with the os separator
3. convert the created final path template to an array of paths.

##### Parameter

> Parameter of: AoPath Method: sh_a_path_by_tmpl
> Name | Type | Default | Description
> --- | --- | --- | ---
> a_path_tmpl_key | TyAoPath |  | array of path template keys
> kwargs | TyDic |  | keyword arguments

##### Return Value

> Return Value of: AoPath Method: sh_a_path_by_tmpl
> Name | Type | Default | Description
> --- | --- | --- | ---
> a_path | TyAoPath |  | Path

#### AoPath Method: yield_path_kwargs

1. Create array of paths by executing the function sh_a_path_by_tmpl with the parameter:
   - array of path template keys (2.argument).
2. Loop over array of paths to yield:
   1. yield path, kwargs (3. argument)

##### Parameter

> Parameter of: AoPath Method: yield_path_kwargs
> Name | Type | Default | Description
> --- | --- | --- | ---
> cls | Tyclass |  | current class
> a_path_tmpl_key | TyAoPath |  | array of path template keys
> kwargs | TyDic |  | keyword arguments

##### Return Value

> Return Value of: AoPath Method: yield_path_kwargs
> Name | Type | Description
> --- | --- | ---
> (path, kwargs) | TyAoPath | Path

#### AoPath Method: yield_path_kwargs_new

##### Synopsis

sh_a_path_by_tmpl(a_path_tmpl_key, kwargs)

##### Description

1. Create array of directories by executing the function sh_a_path_by_tmpl with the arguments:
   - array of directory template keys (2.argument).
2. Loop over array of directories to:
   1. create kwargs_new by executing ths given function sh_kwargs_new (4. argument) with the arguments:
      - directory, given kwargs (5. argument)
   2. create array of paths by executing the function sh_a_oath_by_tmpl with the arguments:
      - given array of path template keys (3. argument), kwargs_new
3. Loop over array of paths within the outer loop to:
   1. yield path, kwargs_new

##### Parameter

> Parameter of: AoPath Method: yield_path_kwargs_new
> Name | Type | Default | Description
> --- | --- | --- | ---
> cls | Tyclass |  | Current class
> a_dir_tmpl_key | TyAoPath |  | Array of path template keys
> a_path_tmpl_key | TyAoPath |  | Array of path template keys
> sh_kwargs_new | TyAoPath |  | Show new keyword arguments function
> kwargs | TyDic |  | Keyword arguments

##### Return Value

> Return Value of: AoPath Method: yield_path_kwargs_new
> Name | Type | Description
> --- | --- | ---
> (path, kwargs_new) | TyAoPath | Path, new keyword arguments

#### AoPath Method: yield_path_item_kwargs

1. Create array of paths by executing the function sh_a_path_by_tmpl with the arguments:
   - array of path template keys (2.argument).
2. Create array of items by selecting the value in the directory kwargs (4. argument) for the kwargs key (3. argument)
3. Loop over array of path and array of items to:
   1. yield path, item, kwargs (4. argument)

##### Parameter

> Parameter of: AoPath Method: yield_path_item_kwargs
> Name | Type | Default | Description
> --- | --- | --- | ---
> cls | Tyclass |  | current class
> a_path_tmpl_key | TyAoPath |  | array of path template keys
> a_arr_key | TyAoPath |  | array of path template keys
> kwargs | TyDic |  | keyword arguments

##### Return Value

> Return Value of: AoPath Method: yield_path_item_kwargs
> Name | Type | Description
> --- | --- | ---
> (path, item, kwargs) | TyAoPath | Path, Item, keyword arguments

#### Method: AoPath.yield_path_item_kwargs_new

1. Create array of directories by executing the function sh_a_path_by_tmpl with the parameter:
   - a_dir_tmpl_key (2.argument).
2. Create array of items by selecting the value in the directory kwargs (4. argument) for the key arr_key (3. argument)
3. Loop over the array of directories to:
   1. create kwargs_new by executing ths function sh_kwargs_new (4. argument) with the arguments:
      - directory, given kwargs (5. argument)
   2. create array of paths by executing the function sh_a_oath_by_tmpl with the arguments:
      - given array of path template keys (3. argument), kwargs_new
   3. Loop over array of path and array of items within the outer loop to:
      1. yield path, item, kwargs_new

##### Parameter

> Parameter of: AoPath Method: yield_path_item_kwargs_new
> Name | Type | Default | Description
> --- | --- | --- | ---
> cls | Tyclass |  | current class
> a_dir_tmpl_key | TyAoPath |  | array of path template keys
> a_path_tmpl_key | TyAoPath |  | array of path template keys
> sh_kwargs_new | TyAoPath |  | show new keyword arguments function
> kwargs | TyDic |  | keyword arguments

##### Return Value

> Return Value of: AoPath Method: yield_path_item_kwargs_new
> Name | Type | Description
> --- | --- | ---
> (path, item, kwargs_new) | TyAoPath | Path, Item, new keyword arguments

# Module: aos.py

## Classes

The Module `aos.py` contains the single static class `AoS`;

### Class: AoS

#### AoS Methods

> AoS Methods
> Name | short Description
> --- | ---
> nvl | Replace empty array of strings
> sh_a_date | Convert array of strings to array of dates
> to_lower | Convert array of strings to array of lowered strings.
> to_unique | Deduplicate array of arrays
> to_unique_lower | Convert array of strings to deduplicted array of lowered strings.
> to_unique_lower_invariant | Convert array of arrays to array of arrays

#### Method: AoS.to_unique

##### Parameter

> Parameter of: AoS Method: to_unique
> Name | Type | Default | Description
> --- | --- | --- | ---
> aoo | TyAoO |  | array of objects

##### Return Value

> Return Value of: AoS Method: to_unique
> Name | Type | Description
> --- | --- | ---
> aoo_new | TyAoO | new array of objects

# Appendix

## Package Logging

### Description

The Standard or user specifig logging is carried out by the log.py module of the logging package ka_uts_log using the configuration files ka_std_log.yml or ka_usr_log.yml in the configuration directory cfg of the logging package ka_uts_log. The Logging configuration of the logging package could be overriden by yaml files with the same names in the configuration directory cfg of the application packages.

### Log message types

Logging defines log file path names for the following log message types: .

1. debug
2. info
3. warning
4. error
5. critical

#### Application parameter for logging

> Application parameter used in log naming
> Name | Decription | Values | Example
> --- | --- | --- | ---
> dir_dat | Application data directory |  | /otev/data
> tenant | Application tenant name |  | UMH
> package | Application package name |  | otev_xls_srr
> cmd | Application command |  | evupreg
> pid | Process ID |  | æevupreg
> log_ts_type | Timestamp type used in logging files|ts, dt | ts, dt | ts
> log_sw_single_dir | Enable single log directory or multiple log directories | True, False | True

#### Log type and Log directories

Single or multiple Application log directories can be used for each message type:

> Log types and directoriesg
> Log type | Log directory
> --- | ---
> long | short | multiple | single
> debug | dbqs | dbqs | logs
> info | infs | infs | logs
> warning | wrns | wrns | logs
> error | errs | errs | logs
> critical | crts | crts | logs

#### Log files naming

##### Naming Conventions

> Naming conventions for logging file paths
> Type | Directory | File
> --- | --- | ---
> debug | /<dir_dat>/<tenant>/RUN/<package>/<cmd>/<Log directory> | <Log type>_<ts>_<pid>.log
> info | /<dir_dat>/<tenant>/RUN/<package>/<cmd>/<Log directory> | <Log type>_<ts>_<pid>.log
> warning | /<dir_dat>/<tenant>/RUN/<package>/<cmd>/<Log directory> | <Log type>_<ts>_<pid>.log
> error | /<dir_dat>/<tenant>/RUN/<package>/<cmd>/<Log directory> | <Log type>_<ts>_<pid>.log
> critical | /<dir_dat>/<tenant>/RUN/<package>/<cmd>/<Log directory> | <Log type>_<ts>_<pid>.log

##### Naming Examples

> Naming examples for logging file paths
> Type | Directory | File
> --- | --- | ---
> debug | /data/otev/umh/RUN/otev_xls_srr/evupreg/logs | debs_1737118199_9470.log
> info | /data/otev/umh/RUN/otev_xls_srr/evupreg/logs | infs_1737118199_9470.log
> warning | /data/otev/umh/RUN/otev_xls_srr/evupreg/logs | wrns_1737118199_9470.log
> error | /data/otev/umh/RUN/otev_xls_srr/evupreg/logs | errs_1737118199_9470.log
> critical | /data/otev/umh/RUN/otev_xls_srr/evupreg/logs | crts_1737118199_9470.log

## Python Terminology

### Python packages

> Python packages
> Name | Definition
> --- | ---
> Python package | Python packages are directories that contains the special module `__init__.py` and other modules, packages files or directories.
> Python sub-package | Python sub-packages are python packages which are contained in another pyhon package.

#### Python package Sub-directories

> Python packages Sub-directories
> Name | Definition
> --- | ---
> Python package sub-directory | Sub-directories are directories contained in python packages.
> Special Python package sub-directory | Python package sub-directories with a special meaning.

##### Special python package Sub-directories

> Special python Sub-directories
> Name | Description
> --- | ---
> bin | Directory for package scripts.
> cfg | Directory for package configuration files.
> data | Directory for package data files.
> service | Directory for systemd service scripts.

#### Python package files

> Python package files
> Name | Definition
> --- | ---
> Python package files | Files within a python package.
> Special python package files | Package files which are not modules and used as python and used as python marker files like `__init__.py`.
> Python package module | Files with suffix `.py`; they could be empty or contain python code; other modules can be imported into a module.
> Special python package module | Modules like `__init__.py` or `main.py` with special names and functionality.

##### Special python package files

> Special python package files
> Name | Type | Description
> --- | --- | ---
> py.typed | Type checking marker file | The `py.typed` file is a marker file used in Python packages to indicate that the package supports type checking. This is a part of the PEP 561 standard, which provides a standardized way to package and distribute type information in Python.

##### Special python package modules

> Special Python package modules
> Name | Type | Description
> --- | --- | ---
> __init__.py | Package directory marker file | The dunder (double underscore) module `__init__.py` is used to execute initialisation code or mark the directory it contains as a package. The Module enforces explicit imports and thus clear namespace use and call them with the dot notation.
> __main__.py | entry point for the package | The dunder module `__main__.py` serves as an entry point for the package. The module is executed when the package is called by the interpreter with the command python -m <package name>.
> __version__.py | Version file | The dunder module `__version__.py` consist of assignment statements used in Versioning.

### Python elements

> Python elements
> Name | Description
> --- | ---
> Python method | Python functions defined in python modules.
> Special python method | Python functions with special names and functionalities.
> Python class | Classes defined in python modules.
> Python class method | Python methods defined in python classes

#### Special python methods

> Special python methods
> Name | Type | Description
> --- | --- | ---
> __init__ | class object constructor method | The special method `__init__` is called when an instance (object) of a class is created; instance attributes can be defined and initalized in the method.

## Table of Contents

Table of Content

- Overview
- Installation
- Package logging
- Package files
  - Classification
  - Package Modules
    - Overview
- Modules for arrays
  - Module: arr.py
    - Class: Arr
      - Arr Methods
- Modules for array of arrays
  - Module: aoa.py
    - aoa.py Class: AoA
      - AoA Methods
      - AoA Method: concatinate
        - Description
        - Parameter
        - Return Value
      - AoA Method: csv_writerows
        - Description
        - Parameter
        - Return Value
      - AoA Method: nvl
        - Description
        - Parameter
        - Return Value
      - AoA Method: to_aod
        - Description
        - Parameter
        - Return Value
      - AoA Method: to_arr_from_2cols
        - Description
        - Parameter
        - Return Value
      - AoA Method: to_doa_from_2cols
        - Description
        - Parameter
        - Return Value
      - AoA Method: to_dic_from_2cols
        - Description
        - Parameter
        - Return Value
- Modules for array of objects
  - Module: aoo.py
    - aoo.py Class: AoO
      - Methods
      - AoO Method: to_unique
        - Parameter
        - Return Value
  - Module: aopath.py
    - Class: AoPath
      - Methods
      - AoPath Method: join
        - Parameter
        - Return Value
      - AoPath Method: sh_a_path
        - Parameter
        - Return Value
      - AoPath Method: sh_a_path_by_tmpl
        - Parameter
        - Return Value
      - AoPath Method: yield_path_kwargs
        - Parameter
        - Return Value
      - AoPath Method: yield_path_kwargs_new
        - Synopsis
        - Description
        - Parameter
        - Return Value
      - AoPath Method: yield_path_item_kwargs
        - Parameter
        - Return Value
      - Method: AoPath.yield_path_item_kwargs_new
        - Parameter
        - Return Value
- Module: aos.py
  - Classes
    - Class: AoS
      - AoS Methods
      - Method: AoS.to_unique
        - Parameter
        - Return Value
- Appendix
  - Package Logging
    - Description
    - Log message types
      - Application parameter for logging
      - Log type and Log directories
      - Log files naming
        - Naming Conventions
        - Naming Examples
  - Python Terminology
    - Python packages
      - Python package Sub-directories
        - Special python package Sub-directories
      - Python package files
        - Special python package files
        - Special python package modules
    - Python elements
      - Special python methods
  - Table of Contents