{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1. Package versioning"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Package versioning within paa is done based on semantic versioning.\n",
    "\n",
    "`major.minor.patch`\n",
    "\n",
    "By default, patch is updated, but the minor and major could also be update based on, for example, commit messages or manually from the log file. \n",
    "\n",
    "Package auto assembler does try to pull latest version from package storage, but in case of failure uses version logs stored in `.paa/tracking`.\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "initialize VersionHandler"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "pv = VersionHandler(\n",
    "    # required\n",
    "    versions_filepath = '../tests/package_auto_assembler/other/lsts_package_versions.yml',\n",
    "    log_filepath = '../tests/package_auto_assembler/other/version_logs.csv',\n",
    "    # optional\n",
    "    default_version = \"0.0.1\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "add new package"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "pv.add_package(\n",
    "    package_name = \"new_package\",\n",
    "    # optional\n",
    "    version = \"0.0.1\"\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "update package version"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "There are no known versions of 'another_new_package', 0.0.1 will be used!\n"
     ]
    }
   ],
   "source": [
    "pv.increment_patch(\n",
    "    package_name = \"new_package\"\n",
    ")\n",
    "## for not tracked package\n",
    "pv.increment_patch(\n",
    "    package_name = \"another_new_package\",\n",
    "    # optional\n",
    "    default_version = \"0.0.1\"\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "display current versions and logs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'another_new_package': '0.0.1', 'new_package': '0.0.2'}"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pv.get_versions(\n",
    "    # optional\n",
    "    versions_filepath = '../tests/package_auto_assembler/other/lsts_package_versions.yml'\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'0.0.2'"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pv.get_version(\n",
    "    package_name='new_package'\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Timestamp</th>\n",
       "      <th>Package</th>\n",
       "      <th>Version</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2024-07-29 03:26:39</td>\n",
       "      <td>new_package</td>\n",
       "      <td>0.0.1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2024-07-29 03:26:40</td>\n",
       "      <td>new_package</td>\n",
       "      <td>0.0.2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2024-07-29 03:26:40</td>\n",
       "      <td>another_new_package</td>\n",
       "      <td>0.0.1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "             Timestamp              Package Version\n",
       "0  2024-07-29 03:26:39          new_package   0.0.1\n",
       "1  2024-07-29 03:26:40          new_package   0.0.2\n",
       "2  2024-07-29 03:26:40  another_new_package   0.0.1"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pv.get_logs(\n",
    "    # optional\n",
    "    log_filepath = '../tests/package_auto_assembler/other/version_logs.csv'\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "flush versions and logs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "pv.flush_versions()\n",
    "pv.flush_logs()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "get latest available version with pip"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'0.3.1'"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pv.get_latest_pip_version(package_name = 'package-auto-assembler')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2. Import mapping"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Install and import names of dependencies may vary. The mapping files maps import names to install names so that requirements extraction from `.py` files is possible. Some of the mapping are packaged and would not need to provided, but in case a dependency used within new package was not inluded, it is possible to augment default mapping through `.paa/package_mapping.json`.\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "initialize ImportMappingHandler"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "im = ImportMappingHandler(\n",
    "    # required\n",
    "    mapping_filepath = \"../env_spec/package_mapping.json\"\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "load package mappings"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'PIL': 'Pillow',\n",
       " 'bs4': 'beautifulsoup4',\n",
       " 'fitz': 'PyMuPDF',\n",
       " 'attr': 'attrs',\n",
       " 'dotenv': 'python-dotenv',\n",
       " 'googleapiclient': 'google-api-python-client',\n",
       " 'google_auth_oauthlib': 'google-auth-oauthlib',\n",
       " 'sentence_transformers': 'sentence-transformers',\n",
       " 'flask': 'Flask',\n",
       " 'stdlib_list': 'stdlib-list',\n",
       " 'sklearn': 'scikit-learn',\n",
       " 'yaml': 'pyyaml',\n",
       " 'package_auto_assembler': 'package-auto-assembler',\n",
       " 'git': 'gitpython'}"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "im.load_package_mappings(\n",
    "    # optional\n",
    "    mapping_filepath = \"../env_spec/package_mapping.json\"\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3. Extracting and merging requirements"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Maintaining requirements is much simpler, when done automatically based on the `.py` files. \n",
    "\n",
    "The actual requirements files is still constructed. Standard libraries are not added, others are added with their versions, if specified. Local files are also used as dependencies, from which imports are extracted as well. \n",
    "\n",
    "For example:\n",
    "\n",
    "```python\n",
    "import os\n",
    "import pandas\n",
    "import attr #>=22.2.0\n",
    "from .components.local_dep import *\n",
    "```\n",
    "\n",
    "produces \n",
    "\n",
    "``` txt\n",
    "pandas\n",
    "attrs >=22.2.0\n",
    "yaml\n",
    "```\n",
    "\n",
    "as requirements file, where `yaml` is extracted from `local_dep.py` file.\n",
    "\n",
    "Checking dependecies for vulnerabilities is usefull and it is done with `pip audit` which is integrated into the paa package and is used by default.\n",
    "\n",
    "Optional requirements for `extras_require` could be probided the same way normal requirements are, but each like that contains an import like that should be commented out in a special way, starting with `#!`, for example:\n",
    "\n",
    "```python\n",
    "import os\n",
    "import pandas\n",
    "import attr #>=22.2.0\n",
    "#! import hnswlib #==0.8.0\n",
    "```\n",
    "\n",
    "produces\n",
    "\n",
    "``` txt\n",
    "pandas\n",
    "attrs >=22.2.0\n",
    "hnswlib==0.8.0; extra == \"hnswlib\"\n",
    "```\n",
    "\n",
    "Sometimes automatic translation of import names to install names via `package_mapping.json`, for packages where these names differ, may not be enough. A manual overwrite can be done with exlusion of some dependencies from automatic extraction pipeline with `#-` comment next to import and `#@` prefix before text that is intended to end up in an equivalent requirements file, for example:\n",
    "\n",
    "```python\n",
    "import os\n",
    "import pandas\n",
    "import attr #>=22.2.0\n",
    "import tensorflow #-\n",
    "#@ tensorflow-gpu\n",
    "```\n",
    "\n",
    "produces\n",
    "\n",
    "\n",
    "``` txt\n",
    "pandas\n",
    "attrs >=22.2.0\n",
    "tensorflow-gpu\n",
    "```\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "initialize RequirementsHandler"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "rh = RequirementsHandler(\n",
    "    # optional/required later\n",
    "    module_filepath = \"../tests/package_auto_assembler/other/example_module.py\",\n",
    "    package_mappings = {'PIL': 'Pillow',\n",
    "                        'bs4': 'beautifulsoup4',\n",
    "                        'fitz': 'PyMuPDF',\n",
    "                        'attr': 'attrs',\n",
    "                        'dotenv': 'python-dotenv',\n",
    "                        'googleapiclient': 'google-api-python-client',\n",
    "                        'sentence_transformers': 'sentence-transformers',\n",
    "                        'flask': 'Flask',\n",
    "                        'stdlib_list': 'stdlib-list',\n",
    "                        'sklearn': 'scikit-learn',\n",
    "                        'yaml': 'pyyaml'},\n",
    "    requirements_output_path = \"../tests/package_auto_assembler/other/\",\n",
    "    output_requirements_prefix = \"requirements_\",\n",
    "    custom_modules_filepath = \"../tests/package_auto_assembler/dependancies\",\n",
    "    python_version = '3.8',\n",
    "    add_header = True\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "list custom modules for a given directory"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['example_local_dependacy_1', 'example_local_dependacy_2']"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "rh.list_custom_modules(\n",
    "    # optional\n",
    "    custom_modules_filepath=\"../tests/package_auto_assembler/dependancies\"\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "check if module is a standard python library"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "False"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "rh.is_standard_library(\n",
    "    # required\n",
    "    module_name = 'example_local_dependacy_1',\n",
    "    # optional\n",
    "    python_version = '3.8'\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "rh.is_standard_library(\n",
    "    # required\n",
    "    module_name = 'logging',\n",
    "    # optional\n",
    "    python_version = '3.8'\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "extract requirements from the module file"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(['attrs>=22.2.0'],\n",
       " ['torch<=2.4.1', 'fastapi[all]', 'scikit-learn==1.5.1', 'numpy'])"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "rh.extract_requirements(\n",
    "    # optional\n",
    "    module_filepath = \"../tests/package_auto_assembler/other/example_module.py\",\n",
    "    custom_modules = ['example_local_dependacy_2', 'example_local_dependacy_1'],\n",
    "    package_mappings = {'PIL': 'Pillow',\n",
    "                        'bs4': 'beautifulsoup4',\n",
    "                        'fitz': 'PyMuPDF',\n",
    "                        'attr': 'attrs',\n",
    "                        'dotenv': 'python-dotenv',\n",
    "                        'googleapiclient': 'google-api-python-client',\n",
    "                        'sentence_transformers': 'sentence-transformers',\n",
    "                        'flask': 'Flask',\n",
    "                        'stdlib_list': 'stdlib-list',\n",
    "                        'sklearn': 'scikit-learn',\n",
    "                        'yaml': 'pyyaml'},\n",
    "    python_version = '3.8',\n",
    "    add_header=True\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['attrs>=22.2.0']"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "rh.requirements_list"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['torch<=2.4.1', 'fastapi[all]', 'scikit-learn==1.5.1', 'numpy']"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "rh.optional_requirements_list"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "audit dependencies"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "No known vulnerabilities found\n",
      "\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "rh.check_vulnerabilities(\n",
    "    # optional if ran extract_requirements() before\n",
    "    requirements_list = None,\n",
    "    raise_error = True\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[]"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "rh.vulnerabilities"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Found 1 known vulnerability in 1 package\n",
      "\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Name    Version ID                  Fix Versions\n",
      "------- ------- ------------------- ------------\n",
      "hnswlib 0.7.0   GHSA-xwc8-rf6m-xr86\n",
      "\n",
      "Error: Found vulnerabilities, resolve them or ignore check to move forwards!\n"
     ]
    }
   ],
   "source": [
    "try:\n",
    "    rh.check_vulnerabilities(\n",
    "        # optional if ran extract_requirements() before\n",
    "        requirements_list = ['attrs>=22.2.0', 'pandas', 'hnswlib==0.7.0'],\n",
    "        raise_error = True\n",
    "    )\n",
    "except Exception as e:\n",
    "    print(f\"Error: {e}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[{'name': 'hnswlib',\n",
       "  'version': '0.7.0',\n",
       "  'id': 'GHSA-xwc8-rf6m-xr86',\n",
       "  'fix_versions': None}]"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "rh.vulnerabilities"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "save requirements to a file"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "rh.write_requirements_file(\n",
    "    # optional/required later\n",
    "    module_name = 'example_module',\n",
    "    requirements = ['### example_module.py', 'attrs>=22.2.0'],\n",
    "    output_path = \"../tests/package_auto_assembler/other/\",\n",
    "    prefix = \"requirements_\"\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "read requirements"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['attrs>=22.2.0']"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "rh.read_requirements_file(\n",
    "    # required\n",
    "    requirements_filepath = \"../tests/package_auto_assembler/other/requirements_example_module.txt\"\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4. Preparing metadata"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since all of the necessary information for building a package needs to be contained within main component `.py` file, basic metadata is provided with the use of `__package_metadata__` dictionary object, defined within that `.py` file. It is also used as a trigger for package building within paa pipeline. \n",
    "\n",
    "Even though some general information shared between packages could be provided through general config, but package specific info should be provided through `__package_metadata__`. It should support most text fields from setup file, but for others the following fields are available:\n",
    "\n",
    "- `classifiers`: adds classifiers to the general ones from config, unless it's `Development Status :: ` then module level definition will overwrite the one from config\n",
    "- `extras_require`: a dictionary of optional package that wouldn't be installed during normal installation. The key could be used during installation and the value would be a list of dependencies.\n",
    "- `install_requires` : adds requirements to the list read from imports\n",
    "\n",
    "\\* Note that providing dependencies this way does not check them through pip-audit or translate them through package mapping\n",
    "\n",
    "---\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "initializing MetadataHandler"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "mh = MetadataHandler(\n",
    "    # optional/required later\n",
    "    module_filepath = \"../tests/package_auto_assembler/other/example_module.py\"\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "check if metadata is available"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "mh.is_metadata_available(\n",
    "    # optional\n",
    "    module_filepath = \"../tests/package_auto_assembler/other/example_module.py\"\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "extract metadata from module"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'author': 'Kyrylo Mordan',\n",
       " 'author_email': 'parachute.repo@gmail.com',\n",
       " 'version': '0.0.1',\n",
       " 'description': 'A mock handler for simulating a vector database.',\n",
       " 'keywords': ['python', 'vector database', 'similarity search']}"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "mh.get_package_metadata(\n",
    "    # optional\n",
    "    module_filepath = \"../tests/package_auto_assembler/other/example_module.py\"\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5. Merging local dependacies into single module"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Package auto assembler creates `single module packages`, meaning that once package is built all of the object are imported from a single place. The packaging tool does allow for `local dependecies` which are `.py` files imported from specified dependencies directory and its subfolders. Packaging structure may look like the following:\n",
    "\n",
    "```\n",
    "packaging repo/\n",
    "└src/\n",
    "  ├ <package names>.py\n",
    "  └ components\n",
    "    ├local_dependecy.py\n",
    "    └subdir_1\n",
    "      └local_dependency_2.py \n",
    "```\n",
    "\n",
    "During packaging process paa merges main module with its local dependies into a single file.\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "initializing LocalDependaciesHandler"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "ldh = LocalDependaciesHandler(\n",
    "    # required\n",
    "    main_module_filepath = \"../tests/package_auto_assembler/other/example_module.py\",\n",
    "    dependencies_dir = \"../tests/package_auto_assembler/dependancies/\",\n",
    "    # optional\n",
    "    save_filepath = \"./combined_example_module.py\"\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "combine main module with dependacies"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\"\"\"\n",
      "Mock Vector Db Handler\n",
      "\n",
      "This class is a mock handler for simulating a vector database, designed primarily for testing and development scenarios.\n",
      "It offers functionalities such as text embedding, hierarchical navigable small world (HNSW) search,\n",
      "and basic data management within a simulated environment resembling a vector database.\n",
      "\"\"\"\n",
      "\n",
      "import logging\n",
      "import json\n",
      "import time\n",
      "import attr #>=22.2.0\n",
      "import sklearn\n",
      "\n",
      "__design_choices__ = {}\n",
      "\n",
      "@attr.s\n",
      "class Shouter:\n",
      "\n",
      "    \"\"\"\n",
      "    A class for managing and displaying formatted log messages.\n",
      "\n",
      "    This class uses the logging module to create and manage a logger\n",
      "    for displaying formatted messages. It provides a method to output\n",
      "    various types of lines and headers, with customizable message and\n",
      "    line lengths.\n",
      "    \"\"\"\n",
      "\n",
      "    # Formatting settings\n",
      "    dotline_length = attr.ib(default=50)\n",
      "\n",
      "    # Logger settings\n",
      "    logger = attr.ib(default=None)\n",
      "    logger_name = attr.ib(default='Shouter')\n",
      "    loggerLvl = attr.ib(default=logging.DEBUG)\n",
      "    log\n"
     ]
    }
   ],
   "source": [
    "print(ldh.combine_modules(\n",
    "    # optional\n",
    "    main_module_filepath = \"../tests/package_auto_assembler/other/example_module.py\",\n",
    "    dependencies_dir = \"../tests/package_auto_assembler/dependancies/\",\n",
    "    add_empty_design_choices = False\n",
    ")[0:1000])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['example_local_dependacy_2', 'example_local_dependacy_1', 'dep_from_bundle_1']"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ldh.dependencies_names_list"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "save combined module"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "ldh.save_combined_modules(\n",
    "    # optional\n",
    "    combined_module = ldh.combine_modules(),\n",
    "    save_filepath = \"./combined_example_module.py\"\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 6. Prepare README"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Package description is based on `.ipynb` with same name as the `.py`. By default it is converted to markdown as is, but there is also an option to execute it.\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "import logging\n",
    "ldh = LongDocHandler(\n",
    "    # optional/required later\n",
    "    notebook_path = \"../tests/package_auto_assembler/other/example_module.ipynb\",\n",
    "    markdown_filepath = \"../example_module.md\",\n",
    "    timeout = 600,\n",
    "    kernel_name = 'python3',\n",
    "    # logger\n",
    "    loggerLvl = logging.DEBUG\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "convert notebook to md without executing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Converted ../tests/package_auto_assembler/example_module.ipynb to ../example_module.md\n"
     ]
    }
   ],
   "source": [
    "ldh.convert_notebook_to_md(\n",
    "    # optional\n",
    "    notebook_path = \"../tests/package_auto_assembler/other/example_module.ipynb\",\n",
    "    output_path = \"../example_module.md\"\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "convert notebook to md with executing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Converted and executed ../tests/package_auto_assembler/example_module.ipynb to ../example_module.md\n"
     ]
    }
   ],
   "source": [
    "ldh.convert_and_execute_notebook_to_md(\n",
    "    # optional\n",
    "    notebook_path = \"../tests/package_auto_assembler/other/example_module.ipynb\",\n",
    "    output_path = \"../example_module.md\",\n",
    "    timeout = 600,\n",
    "    kernel_name = 'python3'\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "return long description"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [],
   "source": [
    "long_description = ldh.return_long_description(\n",
    "    # optional\n",
    "    markdown_filepath = \"../example_module.md\"\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 7. Assembling setup directory"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Packages are created following rather simple sequence of steps. At some point of the process a temporary directory is created to store the following files:\n",
    "\n",
    "- `__init__.py` is a simple import from a single module\n",
    "- `<package name>.py` is a single module with all of the local dependecies\n",
    "- `cli.py` is optional packaged cli tool\n",
    "- `routes.py` is optional packaged file with fastapi routes\n",
    "- `streamlit.py` is optional packaged streamlit app\n",
    "- `setup.py` is a setup file for making a package\n",
    "- `README.md` is a package description file based on `.ipynb` file\n",
    "- `LICENSE` is optional license file\n",
    "- `MANIFEST.in` is a list of additional files to be included with the package\n",
    "- `mkdocs` is a folder with built mkdocs site based on optional `extra_docs` for the module, module docstring and `README.md`\n",
    "- `artifacts` contains optional files that would be packaged with the module\n",
    "- `tests` contains files needed to run tests with [`pytest`](https://docs.pytest.org/en/stable/)\n",
    "- `.paa.tracking` contains tracking files from `.paa` dir to make each release of the package independent of PPR that released it\n",
    "\n",
    "---\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "initializing SetupDirHandler"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [],
   "source": [
    "sdh = SetupDirHandler(\n",
    "    # required\n",
    "    module_filepath = \"../tests/package_auto_assembler/other/example_module.py\",\n",
    "    # optional/ required\n",
    "    module_name = \"example_module\",\n",
    "    metadata = {'author': 'Kyrylo Mordan',\n",
    "                'version': '0.0.1',\n",
    "                'description': 'Example module.',\n",
    "                'long_description' : long_description,\n",
    "                'keywords': ['python']},\n",
    "    license_path = \"../LICENSE\",\n",
    "    requirements = ['attrs>=22.2.0'],\n",
    "    classifiers = ['Development Status :: 3 - Alpha',\n",
    "                   'Intended Audience :: Developers',\n",
    "                   'Intended Audience :: Science/Research',\n",
    "                   'Programming Language :: Python :: 3',\n",
    "                   'Programming Language :: Python :: 3.9',\n",
    "                   'Programming Language :: Python :: 3.10',\n",
    "                   'Programming Language :: Python :: 3.11',\n",
    "                   'License :: OSI Approved :: MIT License',\n",
    "                   'Topic :: Scientific/Engineering'],\n",
    "    setup_directory = \"./example_setup_dir\"\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "create empty setup dir"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [],
   "source": [
    "sdh.flush_n_make_setup_dir(\n",
    "    # optional\n",
    "    setup_directory = \"./example_setup_dir\"\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "copy module to setup dir"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [],
   "source": [
    "sdh.copy_module_to_setup_dir(\n",
    "    # optional\n",
    "    module_filepath = \"./combined_example_module.py\",\n",
    "    setup_directory = \"./example_setup_dir\"\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "copy license to setup dir"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sdh.copy_module_to_setup_dir(\n",
    "    # optional\n",
    "    license_path = \"../LICENSE\",\n",
    "    setup_directory = \"./example_setup_dir\"\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "create init file"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [],
   "source": [
    "sdh.create_init_file(\n",
    "    # optional\n",
    "    module_name = \"example_module\",\n",
    "    setup_directory = \"./example_setup_dir\"\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "create setup file"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [],
   "source": [
    "sdh.write_setup_file(\n",
    "    # optional\n",
    "    module_name = \"example_module\",\n",
    "    metadata = {'author': 'Kyrylo Mordan',\n",
    "                'version': '0.0.1',\n",
    "                'description': 'Example Module',\n",
    "                'keywords': ['python']},\n",
    "    requirements = ['attrs>=22.2.0'],\n",
    "    classifiers = ['Development Status :: 3 - Alpha',\n",
    "                   'Intended Audience :: Developers',\n",
    "                   'Intended Audience :: Science/Research',\n",
    "                   'Programming Language :: Python :: 3',\n",
    "                   'Programming Language :: Python :: 3.9',\n",
    "                   'Programming Language :: Python :: 3.10',\n",
    "                   'Programming Language :: Python :: 3.11',\n",
    "                   'License :: OSI Approved :: MIT License',\n",
    "                   'Topic :: Scientific/Engineering'],\n",
    "    setup_directory = \"./example_setup_dir\"\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 8. Creating release notes from commit messages"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Package versioning could be enhanced with release notes. Since the tool is mainly meant for ci/cd, it takes advantage of commit messages to construct a release note for every version. \n",
    "\n",
    "Commit history is analysed from the last merge, if nothiong found then the next and the next, until at least one of `[<package name>]` labels are found within commit messages. They are bundled together to for a note, where each commit message or messages deliminated with `;` are turned in a list element. Previos notes are used to establish which part of commit history to use as a starting point.\n",
    "\n",
    "Commit messages could also be used to increment version by something other then a default patch. \n",
    "\n",
    "- `[<package name>][..+]` increments patch (default behavior)\n",
    "- `[<package name>][.+.]` increments minor\n",
    "- `[<package name>][+..]` increments major\n",
    "- `[<package name>][0.1.2]` forces specific version `0.1.2`\n",
    "\n",
    "\\* First release within new packaging repo may struggle to extract release note since commit messages are only analysed from merges in the commit history. \n",
    "\n",
    "---\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "No relevant commit messages found!\n",
      "..trying depth 2 !\n",
      "No relevant commit messages found!\n",
      "No messages to clean were provided\n"
     ]
    }
   ],
   "source": [
    "rnh = ReleaseNotesHandler(\n",
    "    # path to existing or new release notes file\n",
    "    filepath = '../tests/package_auto_assembler/other/release_notes.md',\n",
    "    # name of label in commit message [example_module] for filter\n",
    "    label_name = 'example_module',\n",
    "    # new version to be used in release notes\n",
    "    version = '0.0.1'\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "overwritting commit messages from example"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['fixing paa tests',\n",
       " 'fixing paa tests',\n",
       " 'fixing paa tests',\n",
       " '[package_auto_assembler] increasing default max search depth for commit history to 5',\n",
       " 'fixing mocker-db release notes',\n",
       " 'Update package version tracking files',\n",
       " 'Update README',\n",
       " 'Update requirements']"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# commit messages from last merge\n",
    "rnh.commit_messages"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "example_commit_messages = [\n",
    "    '[example_module] usage example for initial release notes; bugfixes for RNH',\n",
    "    '[BUGFIX] missing parameterframe usage example and reduntant png file',\n",
    "    '[example_module][0.1.2] initial release notes handler',\n",
    "    'Update README',\n",
    "    'Update requirements'\n",
    "]\n",
    "rnh.commit_messages = example_commit_messages"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "internal methods that run on intialiazation of ReleaseNotesHandler"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Example filtered_messaged:\n",
      "['[example_module] usage example for initial release notes; bugfixes for RNH', '[example_module][0.1.2] initial release notes handler']\n",
      "Example processed_messages:\n",
      "['usage example for initial release notes', 'bugfixes for RNH', 'initial release notes handler']\n"
     ]
    }
   ],
   "source": [
    "# get messages relevant only for label\n",
    "rnh._filter_commit_messages_by_package()\n",
    "print(\"Example filtered_messaged:\")\n",
    "print(rnh.filtered_messages)\n",
    "\n",
    "# clean messages\n",
    "rnh._clean_and_split_commit_messages()\n",
    "print(\"Example processed_messages:\")\n",
    "print(rnh.processed_messages)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "get version update from relevant messages"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Example version_update: 0.1.2\n"
     ]
    }
   ],
   "source": [
    "version_update = rnh.extract_latest_version()\n",
    "print(f\"Example version_update: {version_update}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "get latest version from relevant release notes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Example latest_version: 0.1.2\n"
     ]
    }
   ],
   "source": [
    "latest_version = rnh.extract_latest_version()\n",
    "print(f\"Example latest_version: {latest_version}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "augment existing release note with new entries or create new"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Example processed_note_entries:\n",
      "['# Release notes\\n', '\\n', '### 0.1.2\\n', '\\n', '    - usage example for initial release notes\\n', '\\n', '    - bugfixes for RNH\\n', '\\n', '    - initial release notes handler\\n', '\\n', '### 0.0.1\\n', '\\n', '    - initial version of example_module\\n']\n"
     ]
    }
   ],
   "source": [
    "# augment existing release note with new entries or create new\n",
    "rnh.create_release_note_entry(\n",
    "    # optional\n",
    "    existing_contents=rnh.existing_contents,\n",
    "    version=rnh.version,\n",
    "    new_messages=rnh.processed_messages\n",
    ")\n",
    "print(\"Example processed_note_entries:\")\n",
    "print(rnh.processed_note_entries)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "saving updated relese notes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['# Release notes\\n',\n",
       " '\\n',\n",
       " '### 0.1.2\\n',\n",
       " '\\n',\n",
       " '    - usage example for initial release notes\\n',\n",
       " '    - bugfixes for RNH\\n',\n",
       " '    - initial release notes handler\\n',\n",
       " '### 0.1.2\\n',\n",
       " '\\n',\n",
       " '    - usage example for initial release notes\\n',\n",
       " '\\n',\n",
       " '    - bugfixes for RNH\\n',\n",
       " '\\n',\n",
       " '    - initial release notes handler\\n',\n",
       " '\\n',\n",
       " '### 0.0.1\\n',\n",
       " '\\n',\n",
       " '    - initial version of example_module\\n']"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "rnh.existing_contents"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "rnh.save_release_notes()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['# Release notes\\n',\n",
       " '\\n',\n",
       " '### 0.1.2\\n',\n",
       " '\\n',\n",
       " '    - usage example for initial release notes\\n',\n",
       " '\\n',\n",
       " '    - bugfixes for RNH\\n',\n",
       " '\\n',\n",
       " '    - initial release notes handler\\n',\n",
       " '\\n',\n",
       " '### 0.0.1\\n',\n",
       " '\\n',\n",
       " '    - initial version of example_module\\n']"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# updated content\n",
    "rnh.get_release_notes_content()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 9. Analysing package dependencies"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Extracting info from installed dependencies can provide important insight into inner workings of a package and help avoid some of the licenses. \n",
    "\n",
    "Licenses are extracted from package metadata and normalized for analysis. Missing labels are marked with `-` and not recognized licenses with `unknown`.\n",
    "\n",
    "Information about unrecognized license labels could be provided through `.paa/package_licenses json` file that would contain install package name and corresponding license label.\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "da = DependenciesAnalyser(\n",
    "    # optional\n",
    "    package_name = 'mocker-db',\n",
    "    package_licenses_filepath = '../tests/package_auto_assembler/other/package_licenses.json',\n",
    "    allowed_licenses = ['mit', 'apache-2.0', 'lgpl-3.0', 'bsd-3-clause', 'bsd-2-clause', '-', 'mpl-2.0']\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "finding installed packages with a list of tags"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('comparisonframe', '0.0.0'),\n",
       " ('mocker-db', '0.0.1'),\n",
       " ('package-auto-assembler', '0.0.0'),\n",
       " ('proompter', '0.0.0')]"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "da.filter_packages_by_tags(tags=['aa-paa-tool'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "extracting some metadata from the installed package"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'keywords': ['aa-paa-tool'],\n",
       " 'version': '0.0.1',\n",
       " 'author': 'Kyrylo Mordan',\n",
       " 'author_email': 'parachute.repo@gmail.com',\n",
       " 'classifiers': ['Development Status :: 3 - Alpha',\n",
       "  'Intended Audience :: Developers',\n",
       "  'Intended Audience :: Science/Research',\n",
       "  'Programming Language :: Python :: 3',\n",
       "  'Programming Language :: Python :: 3.9',\n",
       "  'Programming Language :: Python :: 3.10',\n",
       "  'Programming Language :: Python :: 3.11',\n",
       "  'License :: OSI Approved :: MIT License',\n",
       "  'Topic :: Scientific/Engineering',\n",
       "  'PAA-Version :: 0.4.3',\n",
       "  'PAA-CLI :: False'],\n",
       " 'paa_version': '0.4.3',\n",
       " 'paa_cli': 'False',\n",
       " 'license_label': 'MIT'}"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "package_metadata = da.get_package_metadata(\n",
    "    package_name = 'mocker-db'\n",
    ")\n",
    "package_metadata"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "extracting package requirements"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['requests',\n",
       " 'attrs >=22.2.0',\n",
       " 'httpx',\n",
       " 'hnswlib ==0.8.0',\n",
       " 'gridlooper ==0.0.1',\n",
       " 'dill ==0.3.7',\n",
       " 'numpy ==1.26.0',\n",
       " \"sentence-transformers ; extra == 'sentence_transformers'\"]"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "package_requirements = da.get_package_requirements(\n",
    "    package_name = 'mocker-db'\n",
    ")\n",
    "package_requirements"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "extracting tree of dependencies"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'requests': {'charset-normalizer': [],\n",
       "  'idna': [],\n",
       "  'urllib3': [],\n",
       "  'certifi': []},\n",
       " 'attrs': {'importlib-metadata': {'zipp': [], 'typing-extensions': []}},\n",
       " 'httpx': {'anyio': {'idna': [],\n",
       "   'sniffio': [],\n",
       "   'exceptiongroup': [],\n",
       "   'typing-extensions': []},\n",
       "  'certifi': [],\n",
       "  'httpcore': {'certifi': [], 'h11': {'typing-extensions': []}},\n",
       "  'idna': [],\n",
       "  'sniffio': []},\n",
       " 'hnswlib': {'numpy': []},\n",
       " 'gridlooper': {'dill': [],\n",
       "  'attrs': {'importlib-metadata': {'zipp': [], 'typing-extensions': []}},\n",
       "  'tqdm': {'colorama': []}},\n",
       " 'dill': [],\n",
       " 'numpy': []}"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "extracted_dependencies_tree = da.extract_dependencies_tree(\n",
    "    package_name = 'mocker-db'\n",
    ")\n",
    "extracted_dependencies_tree"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "addding license labels to tree of dependencies"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'requests': 'apache-2.0',\n",
       " 'requests.charset-normalizer': '-',\n",
       " 'requests.idna': '-',\n",
       " 'requests.urllib3': '-',\n",
       " 'requests.certifi': 'mpl-2.0',\n",
       " 'attrs': '-',\n",
       " 'attrs.importlib-metadata': '-',\n",
       " 'attrs.importlib-metadata.zipp': '-',\n",
       " 'attrs.importlib-metadata.typing-extensions': '-',\n",
       " 'httpx': '-',\n",
       " 'httpx.anyio': 'mit',\n",
       " 'httpx.anyio.idna': '-',\n",
       " 'httpx.anyio.sniffio': '-',\n",
       " 'httpx.anyio.exceptiongroup': '-',\n",
       " 'httpx.anyio.typing-extensions': '-',\n",
       " 'httpx.certifi': 'mpl-2.0',\n",
       " 'httpx.httpcore': '-',\n",
       " 'httpx.httpcore.certifi': 'mpl-2.0',\n",
       " 'httpx.httpcore.h11': 'mit',\n",
       " 'httpx.httpcore.h11.typing-extensions': '-',\n",
       " 'httpx.idna': '-',\n",
       " 'httpx.sniffio': '-',\n",
       " 'hnswlib': '-',\n",
       " 'hnswlib.numpy': 'bsd-3-clause',\n",
       " 'gridlooper': '-',\n",
       " 'gridlooper.dill': 'bsd-3-clause',\n",
       " 'gridlooper.attrs': '-',\n",
       " 'gridlooper.attrs.importlib-metadata': '-',\n",
       " 'gridlooper.attrs.importlib-metadata.zipp': '-',\n",
       " 'gridlooper.attrs.importlib-metadata.typing-extensions': '-',\n",
       " 'gridlooper.tqdm': '-',\n",
       " 'gridlooper.tqdm.colorama': '-',\n",
       " 'dill': 'bsd-3-clause',\n",
       " 'numpy': 'bsd-3-clause'}"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "extracted_dependencies_tree_license = da.add_license_labels_to_dep_tree(\n",
    "    dependencies_tree = extracted_dependencies_tree\n",
    ")\n",
    "extracted_dependencies_tree_license"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "printing extracted tree of dependencies"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "└── requests : apache-2.0\n",
      "    ├── charset-normalizer : -\n",
      "    ├── idna : -\n",
      "    ├── urllib3 : -\n",
      "    └── certifi : mpl-2.0\n",
      "└── attrs : -\n",
      "    └── importlib-metadata : -\n",
      "        ├── zipp : -\n",
      "        └── typing-extensions : -\n",
      "└── httpx : -\n",
      "    ├── anyio : mit\n",
      "        ├── idna : -\n",
      "        ├── sniffio : -\n",
      "        ├── exceptiongroup : -\n",
      "        └── typing-extensions : -\n",
      "    ├── certifi : mpl-2.0\n",
      "    ├── httpcore : -\n",
      "        ├── certifi : mpl-2.0\n",
      "        └── h11 : mit\n",
      "            └── typing-extensions : -\n",
      "    ├── idna : -\n",
      "    └── sniffio : -\n",
      "└── hnswlib : -\n",
      "    └── numpy : bsd-3-clause\n",
      "└── gridlooper : -\n",
      "    ├── dill : bsd-3-clause\n",
      "    ├── attrs : -\n",
      "        └── importlib-metadata : -\n",
      "            ├── zipp : -\n",
      "            └── typing-extensions : -\n",
      "    └── tqdm : -\n",
      "        └── colorama : -\n",
      "└── dill : bsd-3-clause\n",
      "└── numpy : bsd-3-clause\n"
     ]
    }
   ],
   "source": [
    "da.print_flattened_tree(extracted_dependencies_tree_license)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "filtering for unexpected licenses in tree of dependencies"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'hnswlib': '', 'gridlooper': ''}\n",
      "└── dill : bsd-3-clause\n",
      "└── numpy : bsd-3-clause\n",
      "└── hnswlib : \n",
      "    └── numpy : bsd-3-clause\n",
      "└── gridlooper : \n",
      "    └── dill : bsd-3-clause\n"
     ]
    },
    {
     "ename": "Exception",
     "evalue": "Found unexpected licenses!",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mException\u001b[0m                                 Traceback (most recent call last)",
      "Cell \u001b[0;32mIn[9], line 3\u001b[0m\n\u001b[1;32m      1\u001b[0m allowed_licenses \u001b[38;5;241m=\u001b[39m [\u001b[38;5;124m'\u001b[39m\u001b[38;5;124mmit\u001b[39m\u001b[38;5;124m'\u001b[39m, \u001b[38;5;124m'\u001b[39m\u001b[38;5;124mapache-2.0\u001b[39m\u001b[38;5;124m'\u001b[39m, \u001b[38;5;124m'\u001b[39m\u001b[38;5;124mlgpl-3.0\u001b[39m\u001b[38;5;124m'\u001b[39m, \u001b[38;5;124m'\u001b[39m\u001b[38;5;124mmpl-2.0\u001b[39m\u001b[38;5;124m'\u001b[39m, \u001b[38;5;124m'\u001b[39m\u001b[38;5;124m-\u001b[39m\u001b[38;5;124m'\u001b[39m]\n\u001b[0;32m----> 3\u001b[0m \u001b[43mda\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mfind_unexpected_licenses_in_deps_tree\u001b[49m\u001b[43m(\u001b[49m\n\u001b[1;32m      4\u001b[0m \u001b[43m    \u001b[49m\u001b[43mtree_dep_license\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43m \u001b[49m\u001b[43mextracted_dependencies_tree_license\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m      5\u001b[0m \u001b[43m    \u001b[49m\u001b[38;5;66;43;03m# optional\u001b[39;49;00m\n\u001b[1;32m      6\u001b[0m \u001b[43m    \u001b[49m\u001b[43mallowed_licenses\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43m \u001b[49m\u001b[43mallowed_licenses\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m      7\u001b[0m \u001b[43m    \u001b[49m\u001b[43mraise_error\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43m \u001b[49m\u001b[38;5;28;43;01mTrue\u001b[39;49;00m\n\u001b[1;32m      8\u001b[0m \u001b[43m)\u001b[49m\n",
      "File \u001b[0;32m~/miniforge3/envs/testenv/lib/python3.10/site-packages/package_auto_assembler/package_auto_assembler.py:2670\u001b[0m, in \u001b[0;36mDependenciesAnalyser.find_unexpected_licenses_in_deps_tree\u001b[0;34m(self, tree_dep_license, allowed_licenses, raise_error)\u001b[0m\n\u001b[1;32m   2668\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m raise_error \u001b[38;5;129;01mand\u001b[39;00m out \u001b[38;5;241m!=\u001b[39m {}:\n\u001b[1;32m   2669\u001b[0m     \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mprint_flattened_tree(flattened_dict \u001b[38;5;241m=\u001b[39m out)\n\u001b[0;32m-> 2670\u001b[0m     \u001b[38;5;28;01mraise\u001b[39;00m \u001b[38;5;167;01mException\u001b[39;00m(\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mFound unexpected licenses!\u001b[39m\u001b[38;5;124m\"\u001b[39m)\n\u001b[1;32m   2671\u001b[0m \u001b[38;5;28;01melse\u001b[39;00m:\n\u001b[1;32m   2672\u001b[0m     \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mlogger\u001b[38;5;241m.\u001b[39minfo(\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mNo unexpected licenses found\u001b[39m\u001b[38;5;124m\"\u001b[39m)\n",
      "\u001b[0;31mException\u001b[0m: Found unexpected licenses!"
     ]
    }
   ],
   "source": [
    "allowed_licenses = ['mit', 'apache-2.0', 'lgpl-3.0', 'mpl-2.0', '-']\n",
    "\n",
    "da.find_unexpected_licenses_in_deps_tree(\n",
    "    tree_dep_license = extracted_dependencies_tree_license,\n",
    "    # optional\n",
    "    allowed_licenses = allowed_licenses,\n",
    "    raise_error = True\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 10. Adding cli interfaces"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The tool allows to make a package with optional cli interfaces. These could be sometimes preferable when a package contains a standalone tool that would be called from script anyway.\n",
    "\n",
    "All of the cli logic would need to be included within a `.py` file which should be stored within `cli_dir` provided in `.paa.config`. \n",
    "Dependencies from these files are extracted in the similar manner to the main module.\n",
    "\n",
    "Tools from main `.py` file could still be imported like the following:\n",
    "\n",
    "```python\n",
    "from package_name.package_name import ToBeImported\n",
    "```\n",
    "\n",
    "The code is wired in `setup.py` via the following automatically assuming that appropriate file with the same name as the package exists within `cli_dir` location.\n",
    "\n",
    "```python\n",
    "...,\n",
    "entry_points = {'console_scripts': [\n",
    "    '<package_alias> = package_name.cli:cli']} ,\n",
    "...\n",
    "```\n",
    "\n",
    "Alias for name could be provided via the following piece of code, defined after imports, otherwise package name would be used.\n",
    "\n",
    "```python \n",
    "__cli_metadata__ = {\n",
    "    \"name\" : <package_alias>\n",
    "}\n",
    "```\n",
    "\n",
    "Package-auto-assembler tool itself uses [`click`](https://pypi.org/project/click/) dependency to build that file, use its [cli definition](https://github.com/Kiril-Mordan/reusables/blob/main/cli/package_auto_assembler.py) as example.\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 11. Adding routes and running FastAPI application"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The tool allows to make a package with optional routes for FastAPI application and run them. Each packages can have one routes file where its logic should be defined. Package-auto-assembler itself can combine multiple routes from packages and filepaths into one application.\n",
    "\n",
    "A `.py`  file with the same name of the package should be stored within `api_routes_dir` provided in `.paa.config`.\n",
    "\n",
    "Dependencies from these files are extracted in the similar manner to the main module.\n",
    "\n",
    "Tools from main `.py` file could still be imported like the following:\n",
    "\n",
    "```python\n",
    "from package_name.package_name import ToBeImported\n",
    "```\n",
    "\n",
    "Api description, middleware and run parameters could be provided via optional `.paa.api.config` file, which for example would look like:\n",
    "\n",
    "```\n",
    "DESCRIPTION : {\n",
    "    'version' : 0.0.0\n",
    "}\n",
    "MIDDLEWARE : {\n",
    "    allow_origins : ['*']\n",
    "}\n",
    "RUN : {\n",
    " host : 0.0.0.0\n",
    "}\n",
    "```\n",
    "\n",
    "where DESCRIPTION contains parameters for `FastAPI`, MIDDLEWARE for `CORSMiddleware` and RUN for `uvicorn.run`\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 12. Adding ui and running streamlit application"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The tools allows to make a package with optional [`streamlit`](https://streamlit.io/) application as interface to the packaged code.  Each package can have one streamlit file. Package-auto-assembler itself would then be used to run packaged applications from the package. \n",
    "\n",
    "A `.py`  file with the same name of the package should be stored within `streamlit_dir` provided in `.paa.config`.\n",
    "\n",
    "Dependencies from these files are extracted in the similar manner to the main module.\n",
    "\n",
    "Tools from main `.py` file could still be imported like the following:\n",
    "\n",
    "```python\n",
    "from package_name import ToBeImported\n",
    "```\n",
    "\n",
    "Config file with server, theme and other settings can be provided via optional `.paa.streamlit.config`. \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 13 Adding artifacts to packages"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The tool allows to add files to packages that could be accessed from the package or extracted into selected directory.\n",
    "\n",
    "There are different types of artifacts with a package like this:\n",
    "\n",
    "- `.paa.tracking` : includes some tracking files for the purposes of the tool, added to every package\n",
    "- `mkdocs` : optional static mkdocs site \n",
    "- `artifacts` contains directories, files and links to files copied from `artifacts_dir/<package_name>` (from `.paa.config`)\n",
    "- `tests` contains optional directory with tests with `pytest`, copied from `tests_dir/<package_name>` (from `.paa.config`)\n",
    "\n",
    "##### 1. Description of default tracking files\n",
    "\n",
    "Tracking files are added automatically of artifacts adding was not turned off. At the moment contains:\n",
    "\n",
    "- `.paa.config` : config file that specifies how paa show work\n",
    "- `.paa.version`: version of `package-auto-assembler` that was used for packaging\n",
    "- `release_notes.md` : latest release notes for the package\n",
    "- `version_logs.csv` : logs for version updates for all packages in the packaging repo\n",
    "- `lsts_package_versions.yml` : latests versions of all packages in the packaging repo\n",
    "- `package_mapping.json` : additional user-provided remapping of package import names to install names\n",
    "- `package_licenses.json` : additional user-provided license labels to overwrite detected ones\n",
    "- `notebook.ipynb` : optional jupyter notebook that was used for package description`\n",
    "\n",
    "##### 2. Adding files\n",
    "\n",
    "User provided artifacts could be provided in two ways:\n",
    "\n",
    "1. adding directory, file or link to the file under `artifacts_dir/<package_name>`\n",
    "\n",
    "    These files would be packaged with the packages, and files from links would be downloaded and packaged as well. To add a link, create a file with `.link` extension in the name.\n",
    "\n",
    "2. adding `artifact_urls` dictionary to `__package_metadata__` within module `.py` file\n",
    "\n",
    "Example of `__package_metadata__` with these additional dictionary would be:\n",
    "\n",
    "```python\n",
    "__package_metadata__ = {\n",
    "    ...,\n",
    "    \"artifact_urls\" : {\n",
    "        'downloaded.md' : 'https://raw.githubusercontent.com/Kiril-Mordan/reusables/refs/heads/main/docs/module_from_raw_file.md',\n",
    "        'downloaded.png' : 'https://raw.githubusercontent.com/Kiril-Mordan/reusables/refs/heads/main/docs/reuse_logo.png'\n",
    "    }\n",
    "}\n",
    "```\n",
    "\n",
    "where key would contain name of the artifact and value its link.\n",
    "\n",
    "These files would not be downloaded and only links would be packaged. After package installation both kinds of links could be refreshed/donwloaded using [`cli interface`](https://kiril-mordan.github.io/reusables/package_auto_assembler/cli/) from `package-auto-assembler`.\n",
    "\n",
    "##### 3. Accessing packaged artifacts\n",
    "\n",
    "Artifacts packaged within the package could be accessed in two ways:\n",
    "\n",
    "1. using [`package-auto-assembler` cli tools](https://kiril-mordan.github.io/reusables/package_auto_assembler/cli_tools/#extracting-files-from-packages) to copy a file or files into a selected directory\n",
    "\n",
    "2. reading packaged artifacts from the package, by the use of similar code that finds path to installed package in your system:\n",
    "\n",
    "```python\n",
    "import importlib\n",
    "import importlib.metadata\n",
    "import importlib.resources as pkg_resources\n",
    "\n",
    "installed_package_path = pkg_resources.files('<package_name>')\n",
    "\n",
    "file_name_path = None\n",
    "if 'artifacts' in os.listdir(installed_package_path):\n",
    "\n",
    "    with pkg_resources.path('<package_name>.artifacts', '<file_name>') as path:\n",
    "        file_name_path = path\n",
    "```\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 14. Preparing module for packaging"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 1. Preparing code for packaging\n",
    "\n",
    "To add/edit a python package with `package-auto-assembler`, its building blocks would need to follow requirements mentioned below. \n",
    "\n",
    "**Note**: For basic package only `main module` would be required. More about role of optional files could be seen [here](https://kiril-mordan.github.io/reusables/package_auto_assembler/python_packaging_repo/#inputs-and-outputs-of-ppr).\n",
    "\n",
    "- Main module **[required]**\n",
    "\n",
    "    Main module is considered to be a `<package_name>.py` file stored in `module_dir` (from `.paa.config`), which can also me called `packaging layer`. \n",
    "    This is where most code of the package would be, unless `local dependencies` are used. \n",
    "\n",
    "    Requirements to this file:\n",
    "\n",
    "    1. starts with module docstring (will be used in package description)\n",
    "    2. contains [`__package_metadata__`](https://kiril-mordan.github.io/reusables/package_auto_assembler/description/#4-preparing-metadata) (a way to provide package metadata)\n",
    "    3. **[optional]** import from `local dependencies` stored only in `dependencies_dir`\n",
    "    4. **[optional]** contains [versions of module dependencies](https://kiril-mordan.github.io/reusables/package_auto_assembler/description/#3-extracting-and-merging-requirements) (specifying requirements)\n",
    "\n",
    "\n",
    "- Local dependencies\n",
    "\n",
    "    Local dependencies are optional modules that other modules from `packaging layer` would import from, and are not packaged on their own. These are useful for segmenting codebase into smaller independent components.\n",
    "    They would be stored in subdirectory of `module_dir`, specified as `dependencies_dir` in `.paa.config`. \n",
    "\n",
    "    Requirements for these files:\n",
    "\n",
    "    1. stored only in `dependencies_dir` or its subdirectories \n",
    "    2. do not import any other moduled stored locally\n",
    "    3. **[optional]** contain [versions of module dependencies](https://kiril-mordan.github.io/reusables/package_auto_assembler/description/#3-extracting-and-merging-requirements) (specifying requirements)\n",
    "\n",
    "\n",
    "- Cli interface\n",
    "\n",
    "    Command line interface could be optionally defined for a package by placing `<package_name>.py` file into `cli_dir` (from `.paa.config`). More about adding cli could be found [here](https://kiril-mordan.github.io/reusables/package_auto_assembler/description/#10-adding-cli-interfaces).\n",
    "\n",
    "    Requirements for this file:\n",
    "\n",
    "    1. imports and uses [`click`](https://pypi.org/project/click/) to define cli tools\n",
    "    2. contains [`__cli_metadata__`](https://kiril-mordan.github.io/reusables/package_auto_assembler/description/#10-adding-cli-interfaces) (a way to provide alias for cli tools)\n",
    "    3. **[optional]** imports from packaging layer and local dependencies only with `from <package_name>.<package_name> import ToBeImported`\n",
    "    4. **[optional]** contains [versions of module dependencies](https://kiril-mordan.github.io/reusables/package_auto_assembler/description/#3-extracting-and-merging-requirements) (specifying requirements)\n",
    "\n",
    "\n",
    "\n",
    "- API routes\n",
    "\n",
    "    Routes for FastAPI application could be optionally defined for a package by placing `<package_name>.py` file into `api_routes_dir` (from `.paa.config`). More about adding api routes could be found [here](https://kiril-mordan.github.io/reusables/package_auto_assembler/description/#11-adding-routes-and-running-fastapi-application).\n",
    "\n",
    "    Requirements for this file:\n",
    "\n",
    "    1. imports `from fastapi import APIRouter`\n",
    "    2. defines `router = APIRouter(prefix = \"/<package-name>\")`\n",
    "    3. uses `@router.*` to define endpoints\n",
    "    4. **[optional]** import from packaging layer and local dependencies only with `from <package_name>.<package_name> import ToBeImported`\n",
    "    5. **[optional]** contain [versions of module dependencies](https://kiril-mordan.github.io/reusables/package_auto_assembler/description/#3-extracting-and-merging-requirements) (specifying requirements)\n",
    "\n",
    "\n",
    "- Streamlit app\n",
    "\n",
    "    Streamlit application could be optionally defined for a package by placing `<package_name>.py` file into `streamlit_dir` (from `.paa.config`). More about adding streamlit app could be found [here](https://kiril-mordan.github.io/reusables/package_auto_assembler/description/#12-adding-ui-and-running-streamlit-application).\n",
    "\n",
    "    Requirements for this file:\n",
    "\n",
    "    1. imports [`streamlit`](https://pypi.org/project/streamlit/)\n",
    "    2. **[optional]** import from packaging layer and local dependencies only with `from <package_name>.<package_name> import ToBeImported`\n",
    "    3. **[optional]** contain [versions of module dependencies](https://kiril-mordan.github.io/reusables/package_auto_assembler/description/#3-extracting-and-merging-requirements) (specifying requirements)\n",
    "\n",
    "\n",
    "\n",
    "#### 2. Preparing documentation for packaging\n",
    "\n",
    "- Package description\n",
    "\n",
    "    Providing package description is a useful way to show what the package is about to the intended audience. By default, [`main_module`](https://kiril-mordan.github.io/reusables/package_auto_assembler/description/#preparing-code-for-packaging) docstring would be used for this purpose and stored alongside the package in a `README.md` file, as shown [here](https://kiril-mordan.github.io/reusables/package_auto_assembler/python_packaging_repo/#inputs-and-outputs-of-ppr). \n",
    "\n",
    "    Optionally, one could provide additional information that would be appended to the package docstring via `<package_name>.ipynb` file, placed into `example_notebooks_path` (from `.paa.config`). In the early stages of development, this file could both serve as a place where code is developed/tested before a release and a way to show usage examples.\n",
    "\n",
    "\n",
    "- Additional documentation\n",
    "\n",
    "    Package description may not be enough to store documentation. \n",
    "    A simple [mkdocs](https://www.mkdocs.org/) static site is built by default for every package with for followig tabs:\n",
    "\n",
    "    - `intro` contains module docstring and `pypi` link for `github + pypi` type of packaging repository\n",
    "    - `description` contains optional content from `<package_name>.ipynb` file, placed into `example_notebooks_path` (from `.paa.config`)\n",
    "    - `release notes` contains release notes assembled based on commit messages\n",
    "\n",
    "    Additional documentation can be provided via the following files:\n",
    "\n",
    "    1. `<package_name>.drawio` file, placed into `drawio_dir` (from `.paa.config`). Each tab would be turned into `<package_name>-<tab_name>.png`, which if not referenced would be presented as a separate tab;\n",
    "    2. `.png`, `.ipynb` or `.md` files placed in `extra_docs_dir/<package_name>` (from `.paa.config`). Each file would be turned into a separate tab, but just like `.png` from drawio, if referenced, would not be presented as a separate tab.\n",
    "\n",
    "    **Note**: During packaging process files derived from `<package_name>.drawio` conversion and files from `extra_docs_dir/<package_name>` would both be placed into `.paa/docs` with `<package_name>-*` prefix. Make sure not to name drawio tabs and extra docs with the same names. \n",
    "\n",
    "#### 3. Preparing files for packaging\n",
    "\n",
    "Packaging files alongside code and documentation could be achieved by placing files  into `artifacts_dir/<package_name>` directory (from `.paa.config`). These files could be referenced from within the package and there is an option to provide links instead of physical files. More about adding artifacts to the package could be found [here](https://kiril-mordan.github.io/reusables/package_auto_assembler/description/#13-adding-artifacts-to-packages).\n",
    "\n",
    "Tests that would be used in ci/cd pipeline are expected to be placed into `tests_dir<package_name>` directory (from `.paa.config`). These would be copied into the package as well. \n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 15. Making a package"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Main wrapper for the package integrates described above components into a class that could be used to build package building pipelines within python scripts. \n",
    "\n",
    "To simplify usage [cli interface](https://kiril-mordan.github.io/reusables/package_auto_assembler/cli/) is recomended instead. \n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "initializing PackageAutoAssembler"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "paa = PackageAutoAssembler(\n",
    "    # required\n",
    "    module_name = \"example_module\",\n",
    "    module_filepath  = \"../tests/package_auto_assembler/other/example_module.py\",\n",
    "    # optional\n",
    "    mapping_filepath = \"../env_spec/package_mapping.json\",\n",
    "    licenses_filepath = \"../tests/package_auto_assembler/other/package_licenses.json\",\n",
    "    allowed_licenses = ['mit', 'apache-2.0', 'lgpl-3.0', 'mpl-2.0', '-'],\n",
    "    dependencies_dir = \"../tests/package_auto_assembler/dependancies/\",\n",
    "    example_notebook_path = \"./mock_vector_database.ipynb\",\n",
    "    versions_filepath = '../tests/package_auto_assembler/other/lsts_package_versions.yml',\n",
    "    log_filepath = '../tests/package_auto_assembler/other/version_logs.csv',\n",
    "    setup_directory = \"./example_module\",\n",
    "    release_notes_filepath = \"../tests/package_auto_assembler/other/release_notes.md\",\n",
    "    license_path = \"../LICENSE\",\n",
    "    license_label = \"mit\",\n",
    "    classifiers = ['Development Status :: 3 - Alpha',\n",
    "                    'Intended Audience :: Developers',\n",
    "                    'Intended Audience :: Science/Research',\n",
    "                    'Programming Language :: Python :: 3',\n",
    "                    'Programming Language :: Python :: 3.9',\n",
    "                    'Programming Language :: Python :: 3.10',\n",
    "                    'Programming Language :: Python :: 3.11',\n",
    "                    'License :: OSI Approved :: MIT License',\n",
    "                    'Topic :: Scientific/Engineering'],\n",
    "    requirements_list = [],\n",
    "    execute_readme_notebook = True,\n",
    "    python_version = \"3.8\",\n",
    "    version_increment_type = \"patch\",\n",
    "    default_version = \"0.0.1\",\n",
    "    check_vulnerabilities = True,\n",
    "    check_dependencies_licenses = False,\n",
    "    add_requirements_header = True\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "add metadata from module"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Adding metadata ...\n"
     ]
    }
   ],
   "source": [
    "paa.add_metadata_from_module(\n",
    "    # optional\n",
    "    module_filepath  = \"../tests/package_auto_assembler/other/example_module.py\"\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "add or update version"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Incrementing version ...\n",
      "No relevant commit messages found!\n",
      "..trying depth 2 !\n",
      "No relevant commit messages found!\n",
      "..trying depth 3 !\n",
      "No relevant commit messages found!\n",
      "..trying depth 4 !\n",
      "No relevant commit messages found!\n",
      "..trying depth 5 !\n",
      "No relevant commit messages found!\n",
      "No messages to clean were provided\n"
     ]
    }
   ],
   "source": [
    "paa.add_or_update_version(\n",
    "    # overwrites auto mode (not suggested)\n",
    "    version_increment_type = \"patch\",\n",
    "    version = \"1.2.6\",\n",
    "    # optional\n",
    "    module_name = \"example_module\",\n",
    "    versions_filepath = '../tests/package_auto_assembler/lsts_package_versions.yml',\n",
    "    log_filepath = '../tests/package_auto_assembler/version_logs.csv'\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "add release notes from commit messages"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Updating release notes ...\n"
     ]
    }
   ],
   "source": [
    "paa.add_or_update_release_notes(\n",
    "    # optional\n",
    "    filepath=\"../tests/package_auto_assembler/release_notes.md\",\n",
    "    version=paa.metadata['version']\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "prepare setup directory"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Preparing setup directory ...\n"
     ]
    }
   ],
   "source": [
    "paa.prep_setup_dir()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "merge local dependacies"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Merging ../tests/package_auto_assembler/other/example_module.py with dependecies from ../tests/package_auto_assembler/dependancies/ into ./example_module/example_module.py\n"
     ]
    }
   ],
   "source": [
    "paa.merge_local_dependacies(\n",
    "    # optional\n",
    "    main_module_filepath = \"../tests/package_auto_assembler/other/example_module.py\",\n",
    "    dependencies_dir= \"../tests/package_auto_assembler/dependancies/\",\n",
    "    save_filepath = \"./example_module/example_module.py\"\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "add requirements from module"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Adding requirements from ../tests/package_auto_assembler/other/example_module.py\n",
      "No known vulnerabilities found\n",
      "\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "paa.add_requirements_from_module(\n",
    "    # optional\n",
    "    module_filepath = \"../tests/package_auto_assembler/other/example_module.py\",\n",
    "    import_mappings = {'PIL': 'Pillow',\n",
    "                        'bs4': 'beautifulsoup4',\n",
    "                        'fitz': 'PyMuPDF',\n",
    "                        'attr': 'attrs',\n",
    "                        'dotenv': 'python-dotenv',\n",
    "                        'googleapiclient': 'google-api-python-client',\n",
    "                        'sentence_transformers': 'sentence-transformers',\n",
    "                        'flask': 'Flask',\n",
    "                        'stdlib_list': 'stdlib-list',\n",
    "                        'sklearn': 'scikit-learn',\n",
    "                        'yaml': 'pyyaml',\n",
    "                        'git' : 'gitpython'}\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['### example_module.py', 'attrs>=22.2.0']"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "paa.requirements_list"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "make README out of example notebook"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Adding README from ../tests/package_auto_assembler/other/example_module.ipynb to ./example_module/README.md\n"
     ]
    }
   ],
   "source": [
    "paa.add_readme(\n",
    "    # optional\n",
    "    example_notebook_path = \"../tests/package_auto_assembler/other/example_module.ipynb\",\n",
    "    output_path = \"./example_module/README.md\",\n",
    "    execute_notebook=False,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "prepare setup file"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Preparing setup file for example-module package ...\n"
     ]
    }
   ],
   "source": [
    "paa.prep_setup_file(\n",
    "    # optional\n",
    "    metadata = {'author': 'Kyrylo Mordan',\n",
    "                'version': '0.0.1',\n",
    "                'description': 'Example module',\n",
    "                'keywords': ['python'],\n",
    "                'license' : 'mit'},\n",
    "    requirements = ['### example_module.py',\n",
    "                    'attrs>=22.2.0'],\n",
    "    classifiers = ['Development Status :: 3 - Alpha',\n",
    "                    'Intended Audience :: Developers',\n",
    "                    'Intended Audience :: Science/Research',\n",
    "                    'Programming Language :: Python :: 3',\n",
    "                    'Programming Language :: Python :: 3.9',\n",
    "                    'Programming Language :: Python :: 3.10',\n",
    "                    'Programming Language :: Python :: 3.11',\n",
    "                    'License :: OSI Approved :: MIT License',\n",
    "                    'Topic :: Scientific/Engineering'],\n",
    "    cli_module_filepath = \"../tests/package_auto_assembler/other/cli.py\"\n",
    "\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "make package"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Making package from ./example_module ...\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "CompletedProcess(args=['python', './example_module/setup.py', 'sdist', 'bdist_wheel'], returncode=0, stdout=\"running sdist\\nrunning egg_info\\nwriting example_module.egg-info/PKG-INFO\\nwriting dependency_links to example_module.egg-info/dependency_links.txt\\nwriting entry points to example_module.egg-info/entry_points.txt\\nwriting requirements to example_module.egg-info/requires.txt\\nwriting top-level names to example_module.egg-info/top_level.txt\\nreading manifest file 'example_module.egg-info/SOURCES.txt'\\nwriting manifest file 'example_module.egg-info/SOURCES.txt'\\nrunning check\\ncreating example_module-0.0.0\\ncreating example_module-0.0.0/example_module\\ncreating example_module-0.0.0/example_module.egg-info\\ncopying files to example_module-0.0.0...\\ncopying example_module/__init__.py -> example_module-0.0.0/example_module\\ncopying example_module/cli.py -> example_module-0.0.0/example_module\\ncopying example_module/example_module.py -> example_module-0.0.0/example_module\\ncopying example_module/setup.py -> example_module-0.0.0/example_module\\ncopying example_module.egg-info/PKG-INFO -> example_module-0.0.0/example_module.egg-info\\ncopying example_module.egg-info/SOURCES.txt -> example_module-0.0.0/example_module.egg-info\\ncopying example_module.egg-info/dependency_links.txt -> example_module-0.0.0/example_module.egg-info\\ncopying example_module.egg-info/entry_points.txt -> example_module-0.0.0/example_module.egg-info\\ncopying example_module.egg-info/requires.txt -> example_module-0.0.0/example_module.egg-info\\ncopying example_module.egg-info/top_level.txt -> example_module-0.0.0/example_module.egg-info\\ncopying example_module.egg-info/SOURCES.txt -> example_module-0.0.0/example_module.egg-info\\nWriting example_module-0.0.0/setup.cfg\\nCreating tar archive\\nremoving 'example_module-0.0.0' (and everything under it)\\nrunning bdist_wheel\\nrunning build\\nrunning build_py\\ncopying example_module/example_module.py -> build/lib/example_module\\ncopying example_module/__init__.py -> build/lib/example_module\\ncopying example_module/setup.py -> build/lib/example_module\\ncopying example_module/cli.py -> build/lib/example_module\\ninstalling to build/bdist.linux-x86_64/wheel\\nrunning install\\nrunning install_lib\\ncreating build/bdist.linux-x86_64/wheel\\ncreating build/bdist.linux-x86_64/wheel/example_module\\ncopying build/lib/example_module/example_module.py -> build/bdist.linux-x86_64/wheel/example_module\\ncopying build/lib/example_module/__init__.py -> build/bdist.linux-x86_64/wheel/example_module\\ncopying build/lib/example_module/setup.py -> build/bdist.linux-x86_64/wheel/example_module\\ncopying build/lib/example_module/cli.py -> build/bdist.linux-x86_64/wheel/example_module\\nrunning install_egg_info\\nCopying example_module.egg-info to build/bdist.linux-x86_64/wheel/example_module-0.0.0-py3.10.egg-info\\nrunning install_scripts\\ncreating build/bdist.linux-x86_64/wheel/example_module-0.0.0.dist-info/WHEEL\\ncreating 'dist/example_module-0.0.0-py3-none-any.whl' and adding 'build/bdist.linux-x86_64/wheel' to it\\nadding 'example_module/__init__.py'\\nadding 'example_module/cli.py'\\nadding 'example_module/example_module.py'\\nadding 'example_module/setup.py'\\nadding 'example_module-0.0.0.dist-info/METADATA'\\nadding 'example_module-0.0.0.dist-info/WHEEL'\\nadding 'example_module-0.0.0.dist-info/entry_points.txt'\\nadding 'example_module-0.0.0.dist-info/top_level.txt'\\nadding 'example_module-0.0.0.dist-info/RECORD'\\nremoving build/bdist.linux-x86_64/wheel\\n\", stderr='warning: sdist: standard file not found: should have one of README, README.rst, README.txt, README.md\\n\\n/home/kyriosskia/miniconda3/envs/testenv/lib/python3.10/site-packages/setuptools/_distutils/cmd.py:66: SetuptoolsDeprecationWarning: setup.py install is deprecated.\\n!!\\n\\n        ********************************************************************************\\n        Please avoid running ``setup.py`` directly.\\n        Instead, use pypa/build, pypa/installer or other\\n        standards-based tools.\\n\\n        See https://blog.ganssle.io/articles/2021/10/setup-py-deprecated.html for details.\\n        ********************************************************************************\\n\\n!!\\n  self.initialize_options()\\n')"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "paa.make_package(\n",
    "    # optional\n",
    "    setup_directory = \"./example_module\"\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 16. Making simple MkDocs site"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Package documentation can be presented in a form of mkdocs static site, which could be either served or deployed to something like github packages. \n",
    "\n",
    "Main module docstring is used as intro package that contains something like optional pypi and license badges. Package description and realease notes are turned into separate tabs. Png with diagrams for example could be provided and displayed as their own separate tabs as well.\n",
    " \n",
    "The one for this package can be seen [here](https://kiril-mordan.github.io/reusables/package_auto_assembler/)\n",
    "\n",
    "It can be packaged with the package and be displayed in web browser like documentation for api via `<package-name>\\docs` when using included api handling capabilities.\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### 1. Preparing inputs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [],
   "source": [
    "package_name = \"example_module\"\n",
    "\n",
    "module_content = LongDocHandler().read_module_content(filepath=f\"../tests/package_auto_assembler/{package_name}.py\")\n",
    "\n",
    "docstring = LongDocHandler().extract_module_docstring(module_content=module_content)\n",
    "pypi_link = LongDocHandler().get_pypi_badge(module_name=package_name)\n",
    "\n",
    "\n",
    "docs_file_paths = {\n",
    "    \"../example_module.md\" : \"usage-examples.md\",\n",
    "    '../tests/package_auto_assembler/release_notes.md' : 'release_notes.md'\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [],
   "source": [
    "mdh = MkDocsHandler(\n",
    "    # required\n",
    "    ## name of the package to be displayed\n",
    "    package_name = package_name,\n",
    "    ## dictionary of markdown files, with path as keys\n",
    "    docs_file_paths = docs_file_paths,\n",
    "    # optional\n",
    "    ## module docstring to be displayed in the index\n",
    "    module_docstring = docstring,\n",
    "    ## pypi badge to be displayed in the index\n",
    "    pypi_badge = pypi_link,\n",
    "    ## license badge to be displayed in the index\n",
    "    license_badge=\"[![License](https://img.shields.io/github/license/Kiril-Mordan/reusables)](https://github.com/Kiril-Mordan/reusables/blob/main/LICENSE)\",\n",
    "    ## name of the project directory\n",
    "    project_name = \"temp_project\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### 2. Preparing site"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Created new MkDocs dir: temp_project\n",
      "Copied ../example_module.md to temp_project/docs/usage-examples.md\n",
      "Copied ../tests/package_auto_assembler/release_notes.md to temp_project/docs/release_notes.md\n",
      "index.md has been created with site_name: example-module\n",
      "mkdocs.yml has been created with site_name: Example module\n",
      "Custom CSS created at temp_project/docs/css/extra.css\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO    -  Cleaning site directory\n",
      "INFO    -  Building documentation to directory: /home/kyriosskia/Documents/nlp/reusables/example_notebooks/temp_project/site\n",
      "INFO    -  Documentation built in 0.12 seconds\n"
     ]
    }
   ],
   "source": [
    "mdh.create_mkdocs_dir()\n",
    "mdh.move_files_to_docs()\n",
    "mdh.generate_markdown_for_images()\n",
    "mdh.create_index()\n",
    "mdh.create_mkdocs_yml()\n",
    "mdh.build_mkdocs_site()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### 3. Test-running site"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [],
   "source": [
    "mdh.serve_mkdocs_site()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "testenv",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.14"
  },
  "orig_nbformat": 4
 },
 "nbformat": 4,
 "nbformat_minor": 2
}