Metadata-Version: 2.1
Name: gaohn-common-utils
Version: 0.0.111
Summary: A small utility package
Author-email: Gao Hongnan <hongnangao@gmail.com>
Project-URL: Homepage, https://github.com/gao-hongnan/common-utils
Project-URL: Bug Tracker, https://github.com/gao-hongnan/common-utils/issues
Classifier: Programming Language :: Python :: 3
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Requires-Python: >=3.7
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: awscli
Requires-Dist: db-dtypes <2.0,>=1.1
Requires-Dist: google-cloud-bigquery <4.0,>=3.10
Requires-Dist: google-cloud-storage <3.0,>=2.8
Requires-Dist: ipython
Requires-Dist: numpy <2.0,>=1.24
Requires-Dist: pandas <2.0,>=1.5
Requires-Dist: prettytable <4.0,>=3.7
Requires-Dist: pydantic <3.0,>=2.0
Requires-Dist: python-dotenv <2.0,>=1.0
Requires-Dist: pytz <2024,>=2023
Requires-Dist: requests <3.0,>=2.28
Requires-Dist: rich <14.0,>=13.3
Requires-Dist: tabulate <1.0,>=0.9
Requires-Dist: torch <3.0,>=2.0
Requires-Dist: torchaudio <3.0,>=2.0
Requires-Dist: torchvision <1.0,>=0.15
Provides-Extra: dev
Requires-Dist: bandit <2.0,>=1.7 ; extra == 'dev'
Requires-Dist: black <24.0,>=23.3 ; extra == 'dev'
Requires-Dist: colorama <1.0,>=0.4 ; extra == 'dev'
Requires-Dist: coverage <8.0,>=7.0 ; extra == 'dev'
Requires-Dist: isort <6.0,>=5.12 ; extra == 'dev'
Requires-Dist: mypy <2.0,>=1.3 ; extra == 'dev'
Requires-Dist: pylint <3.0,>=2.17 ; extra == 'dev'
Requires-Dist: pytest <7.0,>=6.2 ; extra == 'dev'
Requires-Dist: pytest-cov <5.0,>=4.0 ; extra == 'dev'
Requires-Dist: toml <1.0,>=0.10 ; extra == 'dev'

# Common Utils

[![Continuous Integration](https://github.com/gao-hongnan/common-utils/actions/workflows/continuous_integration.yaml/badge.svg)](https://github.com/gao-hongnan/common-utils/actions/workflows/continuous_integration.yaml)

## Continuous Integration

### Virtual Environment

First, make a virtual environment with `make_venv.sh`:

```bash
curl -s -o make_venv.sh \
  https://raw.githubusercontent.com/gao-hongnan/common-utils/main/scripts/devops/make_venv.sh && \
bash make_venv.sh venv --pyproject --dev && \
source venv/bin/activate && \
rm make_venv.sh
```

### Continue on error vs If Always

See
[here](https://stackoverflow.com/questions/58858429/how-to-run-a-github-actions-step-even-if-the-previous-step-fails-while-still-f/58859404#58859404).

### Run Bandit Security Check

```bash
bash ./scripts/devops/ci/ci_security_bandit.sh \
  --severity-level=low \
  --format=json \
  --output=bandit_results.json \
  common_utils
```

### Run Linter Check

```bash
bash ./scripts/devops/ci/ci_linter_pylint.sh \
  --rcfile=pyproject.toml \
  --fail-under=10 \
  --score=yes \
  --output-format=json:pylint_results.json,colorized \
  common_utils
```

### Run Formatter Black Check

```bash
bash ./scripts/devops/ci/ci_formatter_black.sh \
  --check \
  --diff \
  --color \
  --verbose \
  common_utils
```

### Run Formatter Isort Check

```bash
bash ./scripts/devops/ci/ci_formatter_isort.sh \
  --check \
  --diff \
  --color \
  --verbose \
  common_utils
```

## Run MyPy Type Check

```bash
bash ./scripts/devops/ci/ci_typing_mypy.sh \
  --config-file=pyproject.toml \
  common_utils \
  | tee mypy_results.log
```

### Run Unit Test

## Run Integration Test

### Run System Test

### Run Acceptance Test

See [madewithml](https://madewithml.com/courses/mlops/testing/).

### Run Data Test (Great Expectations)

### Run Markdown Lint

```bash
npm install -g markdownlint-cli && \
touch .markdownlint.json && \
```

```bash
npm install --save-dev --save-exact prettier
```

```bash
# prettier
function pr() {
  if [ -z "$1" ]; then
    echo "Error: TARGET_DIR is mandatory."
    return 1
  fi

  TARGET_DIR="$1"
  prettier "$TARGET_DIR" --write \
    --prose-wrap always \
    --print-width 80 \
    --tab-width 4 \
    --use-tabs true
}
```

```bash
pr <TARGET_MARKDOWN_FILE>
markdownlint --fix <TARGET_MARKDOWN_FILE>
```
