{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": [],
      "collapsed_sections": [
        "AGcTzaT_OWIN"
      ]
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "source": [
        "# Feature\n",
        "Run notebooks or their python exports from another notebook to add features and functions or use cells as templates. Tracks running notebooks(s) to prevent recursion and optionally create stats."
      ],
      "metadata": {
        "id": "IVaNo7i2gn9J"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Problem:\n",
        "Code reuse doesn't work well across notebooks in Colab, and they're atypical in terms of software development. Repetitive code wastes time with copy/paste, duplication of work, and lack of code coherence.\n",
        "\n",
        "## Constraints:\n",
        "The feature shall:\n",
        "- [ ] prevent recursion\n",
        "- [ ] add `nb_name` and `running_nb` to `c`onfig\n",
        "- [ ] ensure `features/` and `features/steps/` directories exist\n",
        "- [ ] ensure `project/notebooks/` and `project/notebooks/templates/` directories exist\n",
        "- [ ] provide `%%from_template {name}` cell magic with copy/paste output if existing version varies from saved\n",
        "- [ ] provide `%%writerun {...}` cell magic to flexibly create files from cells\n",
        "- [ ] provide `run_nb({name})` for intelligently running notebooks\n",
        "\n"
      ],
      "metadata": {
        "id": "BmN4gFjChmcu"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Imports"
      ],
      "metadata": {
        "id": "dz06rJ1OqG6a"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import os\n",
        "import re\n",
        "import requests\n",
        "from IPython.core.magic import (register_cell_magic, register_line_magic)\n",
        "from IPython import get_ipython"
      ],
      "metadata": {
        "id": "j3oYkbc3nkXt",
        "executionInfo": {
          "status": "ok",
          "timestamp": 1673963924312,
          "user_tz": 540,
          "elapsed": 116,
          "user": {
            "displayName": "Jaci ss",
            "userId": "10130648813270993501"
          }
        }
      },
      "execution_count": 1,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "# standard first cell"
      ],
      "metadata": {
        "id": "xFDwHsjfAQ3f"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# %%writerun {c['projects_dir']}{c['active_project']}/notebooks/templates/first_cell\n",
        "try:\n",
        "  c\n",
        "except:\n",
        "  c={'resources':{'jai':{}},'use_google_drive':True}\n",
        "  !wget -qO epc.py https://maya.software/epc.txt\n",
        "  import epc\n",
        "  c = epc.do_setup(c)\n",
        "#   %run {c.get('nb_dir')}notebooks_as_features.ipynb\n",
        "\n",
        "# !cd {c['projects_dir']}{c['active_project']} && git pull"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Vly4m3gZKWK7",
        "executionInfo": {
          "status": "ok",
          "timestamp": 1673965987728,
          "user_tz": 540,
          "elapsed": 210,
          "user": {
            "displayName": "Jaci ss",
            "userId": "10130648813270993501"
          }
        },
        "outputId": "75190ba8-bb61-4447-e9c7-1e575d1fb035"
      },
      "execution_count": 24,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "overwrote w /content/drive/MyDrive/Projects/jai/notebooks/templates/first_cell []\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# after git clone/drive mount/etc make sure dirs exist\n",
        "!mkdir -p {c['nb_dir']}\n",
        "!mkdir -p {c['nb_dir']}templates\n",
        "!mkdir -p {c['projects_dir']}{c['active_project']}/features\n",
        "!mkdir -p {c['projects_dir']}{c['active_project']}/features/steps\n",
        "# !ls {c['nb_dir']}\n",
        "# !ls {c['projects_dir']}{c['active_project']}/notebooks\n",
        "# !ls {c['projects_dir']}{c['active_project']}"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "1SpRgTxMmW7R",
        "executionInfo": {
          "status": "ok",
          "timestamp": 1673963928614,
          "user_tz": 540,
          "elapsed": 677,
          "user": {
            "displayName": "Jaci ss",
            "userId": "10130648813270993501"
          }
        },
        "outputId": "48f5eeae-db2e-429c-fd5e-6ac011941c8b"
      },
      "execution_count": 3,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "bdd_testing_a_notebook.ipynb\tjai_utils.ipynb\n",
            "bootstrap_a_notebook.ipynb\tsetup_project.ipynb\n",
            "demo_notebook.ipynb\t\ttemplates\n",
            "encrypted_private_config.ipynb\tvite.ipynb\n",
            "git_for_colab.ipynb\n",
            "features  LICENSE  notebooks  README.md\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Get Notebook Data (including name!)\n",
        "All the vital data on a notebook, including the file name and string to generate the URL."
      ],
      "metadata": {
        "id": "rTTy-dQrACUM"
      }
    },
    {
      "cell_type": "code",
      "execution_count": 5,
      "metadata": {
        "id": "4qkjgXa1f0by",
        "executionInfo": {
          "status": "ok",
          "timestamp": 1673963928634,
          "user_tz": 540,
          "elapsed": 571,
          "user": {
            "displayName": "Jaci ss",
            "userId": "10130648813270993501"
          }
        }
      },
      "outputs": [],
      "source": [
        "from requests.adapters import HTTPAdapter\n",
        "from urllib3.util.retry import Retry\n",
        "def get_nb_data():\n",
        "  # https://stackoverflow.com/questions/66175418/find-path-of-python-notebook-ipynb-when-running-it-with-google-colab\n",
        "  session = requests.Session()\n",
        "  retry = Retry(connect=1, backoff_factor=1)\n",
        "  adapter = HTTPAdapter(max_retries=retry)\n",
        "  session.mount('http://', adapter)\n",
        "  c['nb_ip'] = os.popen('hostname -I').read().strip()\n",
        "  nb_address = 'http://'+c['nb_ip']+':9000/api/sessions' #172.28.0.2\n",
        "  # return requests.get(nb_address).json()[0]\n",
        "  return session.get(nb_address).json()[0]"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# vital data for config\n",
        "# c['nb'] = c['nb'] if 'nb' in c else get_nb_data()\n",
        "if 'nb_name' not in c:\n",
        "  nb_data = get_nb_data()\n",
        "  c['nb_name'] = c['running_nb'] = nb_data['name'][:-6] # assuming w/running_nb; c might get cleared by anyone or this nb run not be in first_cell logic -- no way to tell which is which other than going back to defining it in every nb and changing on renames\n",
        "  c['nb_path'] = nb_data['path'][7:]\n",
        "  try:\n",
        "    logger = epc.get_logger(c['nb_name']) # use if we have it\n",
        "  except:\n",
        "    'epc not loaded, new logger'\n",
        "    import logging # extremely rare, do import here\n",
        "    logger = logging.getLogger()\n",
        "logger.info('main parent notebook: '+c['nb_name'])"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "twZOBNNhBpF_",
        "executionInfo": {
          "status": "ok",
          "timestamp": 1673963928639,
          "user_tz": 540,
          "elapsed": 564,
          "user": {
            "displayName": "Jaci ss",
            "userId": "10130648813270993501"
          }
        },
        "outputId": "b52c06cf-f4fb-4176-d18d-7de06564641a"
      },
      "execution_count": 6,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "2023-01-17 04:58:45 notebooks_as_features INFO     main parent notebook: notebooks_as_features\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Magic for Templates\n"
      ],
      "metadata": {
        "id": "AGcTzaT_OWIN"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Reuse of code is important and there aren't many tools for notebooks. You don't need much to mimic `%load` in Colab. \n",
        "\n",
        "## Create a Template \n",
        "```python\n",
        "%%writerun -e -w {c['nb_dir']}templates/template_name\n",
        "# handy cell content (-e executes and -w writes)\n",
        "```\n",
        "## Use a Template\n",
        "```python\n",
        "%%from_template template_name\n",
        "#anything, it will complain if it doesn't match the template and print out the template code to copy into the cell\n",
        "```\n",
        "## Thoughts\n",
        "Ideally code is cohesive and together in a module you can `import` or a notebook you can `%run` but there *are* legitimate use-cases where one-off templates beat [fetching and] including modules or running entire notebooks. This is why `%macro` exists, but in Colab those don't persisit after runtime restarts and to `%store` them you have to `%run` them to begin with."
      ],
      "metadata": {
        "id": "QSk4JbS5rizc"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "@register_cell_magic\n",
        "def writerun(line, cell):\n",
        "  '''like %save but for a cell'''\n",
        "  fname = line.split()[0]\n",
        "  if not os.path.isdir(os.path.dirname(fname)):\n",
        "    print('Directory does not [yet] exist:',os.path.dirname(fname))\n",
        "  else:\n",
        "    omode = 'a' if '-a' in line else 'w'\n",
        "    overw = 'over' if os.path.isfile(fname) else ''\n",
        "    with open(fname, omode) as f:\n",
        "      f.write(cell)\n",
        "    if not '-q' in line:\n",
        "      print(overw+'wrote '+omode,fname,line.split()[1:])\n",
        "  if '-e' in line:\n",
        "    get_ipython().run_cell(cell)\n",
        "\n",
        "@register_cell_magic\n",
        "def from_template(line, cell):\n",
        "  args = line.split()\n",
        "  # remove active form #@param, a mismatch there is valid https://regex101.com/r/GfyE66/1\n",
        "  p = re.compile(r'[a-zA-Z0-9_ *]{0,99}\"? #@param', re.MULTILINE)\n",
        "  print('template:',c['nb_dir']+'templates/'+args[0])\n",
        "  if not os.path.isfile(c['nb_dir']+'templates/'+args[0]):\n",
        "    print('Not found.')\n",
        "    return False\n",
        "  with open(c['nb_dir']+'templates/'+args[0], 'r') as f:\n",
        "    contents = f.read()\n",
        "  get_ipython().run_cell(cell)\n",
        "  if p.sub('', cell).strip()!=p.sub('', contents).strip():\n",
        "    # print(p.sub('', contents).strip())\n",
        "    # print('CURRENT:')\n",
        "    # print(p.sub('', cell).strip())\n",
        "    print('Mismatch, template is:\\n'+contents) #p.sub('', contents).strip())\n",
        "    # import difflib\n",
        "    # output_list = [li for li in difflib.ndiff(p.sub('', cell).strip(), p.sub('', contents).strip()) if li[0] != ' ']\n",
        "    # print(output_list)"
      ],
      "metadata": {
        "id": "YlwXzZ3Rokkg",
        "executionInfo": {
          "status": "ok",
          "timestamp": 1673963928646,
          "user_tz": 540,
          "elapsed": 501,
          "user": {
            "displayName": "Jaci ss",
            "userId": "10130648813270993501"
          }
        }
      },
      "execution_count": 7,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# c\n",
        "# del c"
      ],
      "metadata": {
        "id": "zm78Tf3Ej03-",
        "executionInfo": {
          "status": "ok",
          "timestamp": 1673963928668,
          "user_tz": 540,
          "elapsed": 514,
          "user": {
            "displayName": "Jaci ss",
            "userId": "10130648813270993501"
          }
        }
      },
      "execution_count": 8,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Replacement `%run` with Tracking\n",
        "So we know what's running from where and don't end up in infinite loops."
      ],
      "metadata": {
        "id": "ipa737cWAbrv"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "def run_nb(name):\n",
        "  print('run_nb(',name,')','nb_name:',c['nb_name'],'current running_nb set to:',c['running_nb'])\n",
        "  logger = epc.get_logger(c['running_nb'])\n",
        "  #TODO: can we catch nb_name->running-->RUNS ANOTHER? stop or give names back in order or..? array_pop, heh\n",
        "  c['running_nb'] = name # c['running_nb'] if 'running_nb' in c else c['nb_name']\n",
        "  if c['nb_name']!=c['running_nb']:\n",
        "    print('running:',c['running_nb'])\n",
        "    logger.info(c['nb_dir']+c['nb_name']+' --> '+c['nb_dir']+c['running_nb'])\n",
        "    %run {c['nb_dir']}{c['running_nb']}.ipynb #TODO: non-ipython version .replace(' ','\\ ') \n",
        "    c['running_nb']=c['nb_name'] # give name back\n",
        "  else: print('recursion, not running',c['running_nb'])\n",
        "  logger = epc.get_logger(c['nb_name'])"
      ],
      "metadata": {
        "id": "AcvJObfW5RnU",
        "executionInfo": {
          "status": "ok",
          "timestamp": 1673963928673,
          "user_tz": 540,
          "elapsed": 504,
          "user": {
            "displayName": "Jaci ss",
            "userId": "10130648813270993501"
          }
        }
      },
      "execution_count": 9,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "c"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "fK8jF-OXyIsH",
        "executionInfo": {
          "status": "ok",
          "timestamp": 1673963928677,
          "user_tz": 540,
          "elapsed": 500,
          "user": {
            "displayName": "Jaci ss",
            "userId": "10130648813270993501"
          }
        },
        "outputId": "0af6bad9-a3f5-4112-8758-4e52230cc41b"
      },
      "execution_count": 10,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{'log_level': 10,\n",
              " 'timezone': 'America/Anchorage',\n",
              " 'use_google_drive': True,\n",
              " 'epc': 'U2FsdGVkX1/xUi/hYJ+V9stP7WV+Y6fTtkJNZoDL8PX4mSWh/CuEAZUz0VI8r/BvrfQztrCI1rPmduBRSJsTLQeD9AKFNjHMNq15V1ArU7CvCVVXvuKgNsDS0aEN2hJoRMXiy4g7YzIB/b45e3vq8/DV/sFFml1GtWeTtIWLeA1c92UNrF8s3MjpvWhYB+GeD1OSQMcS3aoveW7YD8tSigzRkZQqwmoNrHlNdODtNeLq4idK4VsD9mauLGg/mWKitdfbqUi0yULYpg5pRxPMZaIy4sDFdGYbeaHYBaduf36Wr+ak3x6e7rvT7fF04EST3wv7FM54xAxZiyohgUHIyG+5oYBbvcWsKl74GZpoPYqUffxdnnH8VKURHTE28WaXwdsXwj0edbkX3cNcL/aCOdF9pfCcNPJFbRD0LM7E2sF9xYeqyNMhpnBMKWfXex7rEzj2ZRQbxlRy+f+476Vni/Qf2Ixcc/x5vCvAEJgCueD+loXol4oP3O4eDM6Q1FiPxMDOCYGxJ8TCFVb5795vqsmWsKLfa1pOqnPSJ2Id4ayiryeyxtDShRD8fTq0RplfXU7JMKnrCZJNq4CW+rDSRTOy4cHE0S3zy1EbFtPVP3Y4MCCKaWIHa/LVj9LDZSQvYSZO4xZb3Aw+fuvxqI/4Ow==',\n",
              " 'resources': {'jai': {}},\n",
              " 'active_project': 'jai',\n",
              " 'projects_dir': '/content/drive/MyDrive/Projects/',\n",
              " 'nb_dir': '/content/drive/MyDrive/Colab\\\\ Notebooks/',\n",
              " 'cmds': [],\n",
              " 'nb_ip': '172.28.0.2',\n",
              " 'nb_name': 'notebooks_as_features',\n",
              " 'running_nb': 'notebooks_as_features',\n",
              " 'nb_path': '1PDOMs91V8azSzs932x_3xKn_WBYyvJHP'}"
            ]
          },
          "metadata": {},
          "execution_count": 10
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# manual set here, running_nb/run_nb() defined here can't be looped\n",
        "if c.get('running_nb')=='notebooks_as_features':\n",
        "  run_nb('git_for_colab')\n",
        "  nb2repo()\n",
        "  # git_ppush('first cell update')"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Z924EmUs0IPE",
        "outputId": "f2b2e99e-8c3d-4afc-96de-4d6f66457306"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "2023-01-17 05:34:50 notebooks_as_features INFO     /content/drive/MyDrive/Colab\\ Notebooks/notebooks_as_features --> /content/drive/MyDrive/Colab\\ Notebooks/git_for_colab\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "run_nb( git_for_colab ) nb_name: notebooks_as_features current running_nb set to: notebooks_as_features\n",
            "running: git_for_colab\n",
            "Overwriting /root/.jupyter/nbdime_config.json\n",
            "try to write\n",
            "/content/drive/MyDrive/Projects/jai/notebooks/notebooks_as_features.ipynb\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# !cd {c.get('projects_dir')}{c.get('active_project')} && git pull"
      ],
      "metadata": {
        "id": "HP1ce5Xa9kaw",
        "executionInfo": {
          "status": "ok",
          "timestamp": 1673963985657,
          "user_tz": 540,
          "elapsed": 193,
          "user": {
            "displayName": "Jaci ss",
            "userId": "10130648813270993501"
          }
        }
      },
      "execution_count": 12,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# !cd {c['projects_dir']}{c['active_project']} && git a"
      ],
      "metadata": {
        "id": "Uu4CTvvR_LTB",
        "executionInfo": {
          "status": "ok",
          "timestamp": 1673963985683,
          "user_tz": 540,
          "elapsed": 206,
          "user": {
            "displayName": "Jaci ss",
            "userId": "10130648813270993501"
          }
        }
      },
      "execution_count": 13,
      "outputs": []
    }
  ]
}