{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": [],
      "authorship_tag": "ABX9TyM9QGIAjXQ9HycaaeFxK4B2",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/Jaciss/jai/blob/main/notebooks/encrypted_config.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Feature\n",
        "Provides configuration `c:dict` and functions to store, access, and use tokens, email addresses, and other semi-sensitive data that should not be in plain-text via passphrase-encrypted strings.\n"
      ],
      "metadata": {
        "id": "U1DT7XUbQ10W"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Problem:\n",
        "\n",
        "Notebooks:\n",
        "\n",
        "- are highly portable and may be run by anyone in a variety of environments, including offline\n",
        "- may need outside resources, some of which require tokens or passphrases to access\n",
        "- are often one-off works and should not require more than a couple of lines to establish an environment\n",
        "- may require different passphrases for various granular-access tokens depending on the user\n",
        "- are high-risk exposure points for sensitive data as they're commonly shared and stored in repositories\n",
        "- are often used by people who are not primarily programmers and cannot be expected to understand and trust complex security setups\n",
        "\n",
        "## Constraints:\n",
        "\n",
        "The feature shall:\n",
        "\n",
        "- [x] never store secured data in plain-text\n",
        "- [x] never display secured data without code modifications\n",
        "- [x] only require the notebook for configuration creation, modification, and use\n",
        "- [x] generate a stand-alone Python file less than 14 kB in size ([slightly-better-than-random number](https://www.tunetheweb.com/blog/critical-resources-and-the-first-14kb/))\n",
        "- [x] only contain data safe for storage on public servers (*see Drawbacks below)\n",
        "- [x] be uploadable using its own functions\n",
        "- [x] not require ipython to execute (python file only)\n",
        "- [x] not require third-party files or systems (python file only)\n",
        "- [x] be easy to understand; McCabe cyclomatic complexity on functions <= 10 and overall <=5\n",
        "- [x] have easily replaced auth functions (e.g., get/set passphrase or data)\n",
        "- [x] make data access via other systems easy if the user has proper credentials\n",
        "- [x] only require a single line of code to use in notebooks\n",
        "\n",
        "## Solution:\n",
        "Use `json.dumps(dictionary)` and the commonly-installed `openssl` AES 256 bit encryption with a passphrase (and salt) to create a commit-safe string containing sensitive configuration data. Notebook generates epc.py to hold the string and necessary functions to create, upload, use, and edit it. Notebook provides a convenience `do_setup()` function that will setup `c`onfig and download specified resources with a single line. See Usage section below.\n",
        "\n",
        "**Example**\n",
        "\n",
        "This would ask for a passphrase to decrypt 'jai' and, in this case, download the private repo described in it from GitHub. If 'jai' did not exist the user would be asked if they wanted to create it. If they created a dict with FTP data epc could download from FTP instead of GitHub.\n",
        "```python\n",
        "import epc\n",
        "c = epc.do_setup({'resources':{'jai':{}},'use_google_drive':False})\n",
        "```\n",
        "Sensitive data is encrypted or automatically redacted, so printing `c` is safe:\n",
        "```\n",
        "{'log_level': 10,\n",
        " 'timezone': 'America/Anchorage',\n",
        " 'use_google_drive': False,\n",
        " 'epc': 'U2FsdGVkX1/xUi/hYJ+V9stP7WV+Y6fTtkJNZoDL8PX4mSWh/CuEAZUz0VI8r/BvrfQztrCI1rPmduBRSJsTLQeD9AKFNjHMNq15V1ArU7CvCVVXvuKgNsDS0aEN2hJoRMXiy4g7YzIB/b45e3vq8/DV/sFFml1GtWeTtIWLeA1c92UNrF8s3MjpvWhYB+GeD1OSQMcS3aoveW7YD8tSigzRkZQqwmoNrHlNdODtNeLq4idK4VsD9mauLGg/mWKitdfbqUi0yULYpg5pRxPMZaIy4sDFdGYbeaHYBaduf36Wr+ak3x6e7rvT7fF04EST3wv7FM54xAxZiyohgUHIyG+5oYBbvcWsKl74GZpoPYqUffxdnnH8VKURHTE28WaXwdsXwj0edbkX3cNcL/aCOdF9pfCcNPJFbRD0LM7E2sF9xYeqyNMhpnBMKWfXex7rEzj2ZRQbxlRy+f+476Vni/Qf2Ixcc/x5vCvAEJgCueD+loXol4oP3O4eDM6Q1FiPxMDOCYGxJ8TCFVb5795vqsmWsKLfa1pOqnPSJ2Id4ayiryeyxtDShRD8fTq0RplfXU7JMKnrCZJNq4CW+rDSRTOy4cHE0S3zy1EbFtPVP3Y4MCCKaWIHa/LVj9LDZSQvYSZO4xZb3Aw+fuvxqI/4Ow==',\n",
        " 'resources': {'jai': {}},\n",
        " 'setup_commands': ['git clone --progress https://Jaciss:<REDACTED>@github.com/Jaciss/jai.git'],\n",
        " 'active_project': 'jai',\n",
        " 'projects_dir': '/content/',\n",
        " 'project_path': '/content/jai/',\n",
        " 'nb_dir': '/content/jai/notebooks/'}\n",
        "```\n",
        "\n",
        "## Drawbacks:\n",
        "- While the solution is portable it is also unique and may require scripting to use the generated strings in other systems. *Mitigation*: decryption only requires commonly-installed `openssl` and `gzip`, the passphrase, and a single line of code. Decrypted strings are common JSON.\n",
        "- Storing the encrypted strings on public servers is currently safe but AES 256 will, like all encryption methods, presumably be breakable in some distant future. *Mitigation*: practice security hygeine by keeping the two `openssl` lines using modern methods, and/or regularly change passwords and use tokens with expiration dates and/or only store `epc.py` or the encrypted string in a secure location.\n",
        "- The generated python file edits itself when the configuration is updated but does not keep backups. Data loss is possible. *Mitigation*: make secure backups.\n",
        "- This was written by someone who is *not* a security expert and intended for easily-managed tokens not truly sensitive data. *Mitigation*: get it approved for your use-case by your security expert.\n",
        "- With any system like this data must be decrypted to be used and that can happen on third-party servers like Colab: you cannot control what happens on third-party servers.\n",
        "\n",
        "## Previous Attempts:\n",
        "I tried keyrings, standard *nix `pass`, GPG/PGP, SSH keys, and considered online auth service options. None of these met all constraints or were found unreliable across systems, being designed for tty or GUI and not Jupyter notebooks."
      ],
      "metadata": {
        "id": "SJsQM0VuN8Vk"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Imports"
      ],
      "metadata": {
        "id": "ZyxRR5dycFua"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "'''basic dict-based config with optional encrypted string for sensitive data'''\n",
        "import os\n",
        "import time\n",
        "import json\n",
        "import logging\n",
        "import subprocess\n",
        "from json.decoder import JSONDecodeError\n",
        "from getpass import getpass\n",
        "from google.colab import drive\n",
        "\n",
        "def get_logger(logger_name:str, logger_level:int=logging.DEBUG):\n",
        "  '''named loggers for notebooks'''\n",
        "  glogger = logging.getLogger(logger_name)\n",
        "  glogger.setLevel(logger_level)\n",
        "  glogger.propagate = False # no double messages\n",
        "  if len(glogger.handlers)==0:\n",
        "    glogger.addHandler(logging.StreamHandler())\n",
        "    glogger.handlers[0].setFormatter(logging.Formatter(\n",
        "        '%(asctime)s %(name)s %(levelname)-8s %(message)s',\n",
        "        '%Y-%m-%d %H:%M:%S'\n",
        "    ))\n",
        "  return glogger\n",
        "logger = get_logger('epc')"
      ],
      "metadata": {
        "id": "50BVBBzCE0Hw"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Notebook Defines"
      ],
      "metadata": {
        "id": "j7Uzm4OjZp0q"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "The concept is simple but powerful. Encrypt dictionaries containing any number of fields required for a purpose. The dictionary can be decrypted and applied to a template and the resulting string used to generate a command or use a service or function. Additional arguments can be added for specific use of general resources, e.g., a generic FTP server and specific files. Since `gzip` is used on the strings, **multiple dictionaries with the same data don't add much to the string size: focus on usability**.\n",
        "\n",
        "Lambda is probably overkill. I don't know Python very well yet and it is useful to be able to use functions in the templates."
      ],
      "metadata": {
        "id": "o4aRyKZs3gbH"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# simple dict to store config, don't clobber if it exists\n",
        "if 'c' not in globals(): c = {}\n",
        "\n",
        "# redact these fields in intentional output\n",
        "hidden_fields = ['token', 'passwd'] #, 'user']\n",
        "\n",
        "# services for setup/edit functions to consume\n",
        "api_fields = {\n",
        "    'ftp': {'user': None, 'host': None, 'passwd': None, 'protocol': 'ftp', 'port': 21, 'action': '-o'},\n",
        "    'other': {'token': None},\n",
        "    'github': {'user': None, 'repo': None, 'token': None, 'filepath': ''},\n",
        "    'quit': {}\n",
        "}\n",
        "\n",
        "# bare-minimum API templates, uses encrypted dictionaries to generate commands\n",
        "api_templates = {\n",
        "    'ftp': lambda d: (f\"curl --user {d['user']}:{d['passwd']} {d['action']}\"\n",
        "    f\"{d['local_path']} {d['protocol']}://{d['host']}:{d['port']}/{d['remote_path']}\"),\n",
        "    'other': lambda d: f\"echo {d['token']}\", # you probably want get_token()\n",
        "    'github': lambda d: f\"git clone --progress https://{d['user']}:{d['token']}@github.com/{d['user']}/{d['repo']}.git\",\n",
        "    'github_file': lambda d: ['curl', '-O', '-H', 'Authorization: token ' + d['token'], '-H', 'Accept: application/vnd.github.v3.raw', f\"https://api.github.com/repos/{d['user']}/{d['repo']}/contents/{d['filepath']}\", '--http1.1'],\n",
        "    'huggingface': lambda d: ['curl', 'https://api-inference.huggingface.co/models/'+d['model'], '-X', 'POST', '-d', '{\"inputs\": '+d['inputs']+'}', '-H', 'Authorization: Bearer '+d['token']]\n",
        "}\n",
        "\n",
        "# default paths for local resources; see Pathfinding section for details\n",
        "path_templates = {\n",
        "    'projects_dir': lambda d: f\"{d.get('projects_dir','/content/')}\",\n",
        "    'active_project': lambda d: f\"{d.get('active_project','app')}\",\n",
        "    'project_path': lambda d: f\"{d.get('projects_dir','/content/')}{d.get('active_project','app')}/\",\n",
        "    'nb_dir': lambda d: f\"{d.get('projects_dir','/content/')}{d.get('active_project','app')}/notebooks/\",\n",
        "}\n",
        "\n",
        "# storing the passphrase here is less annoying but less secure\n",
        "__e = {} "
      ],
      "metadata": {
        "id": "318_VIecZoB_"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Main Functions, Encryption/Decryption"
      ],
      "metadata": {
        "id": "VnIPJ-rK19gG"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "def get_epcw(label=''):\n",
        "  '''the most basic pass input \"system\", set get_epcw=your_preference()'''\n",
        "  if label!='': label=' for '+label\n",
        "  __e['epw'] = __e['epw'] if 'epw' in __e else getpass('Save passphrase as variable, leave blank to enter each time: ')\n",
        "  return __e.get('epw') or getpass('Encrypt/decrypt passphrase'+label+': ')\n",
        "\n",
        "def exclamation(args:list):\n",
        "  '''takes str or list, approximation of `!command` syntax for .py exports'''\n",
        "  if isinstance(args,str): args=args.split() # split() string for Popen args\n",
        "  # print(' '.join(args)) # dev only, shows tokens!\n",
        "  with subprocess.Popen(args, stdout=subprocess.PIPE, stderr=subprocess.STDOUT) as popen:\n",
        "    for line in popen.stdout:\n",
        "      print(line.decode(), end='')\n",
        "\n",
        "def encode(decoded:dict, label:str='')->str:\n",
        "  '''convert dict to JSON and AES encode with a passphrase'''\n",
        "  json_string = json.dumps(decoded)\n",
        "  # logger.debug('encoding: '+json_string) # dev only, shows tokens!\n",
        "  return os.popen((f\"echo '{json_string}' |gzip -c| openssl enc -e -aes-256-cbc -base64 -A \"\n",
        "  f\"-pass pass:{get_epcw(label)} -pbkdf2\")).read()\n",
        "\n",
        "def decode(encrypted_string:str, label:str='')->dict:\n",
        "  '''decodes an AES passphrase-encrypted JSON string into a dict'''\n",
        "  decoded = os.popen((f'echo \"{encrypted_string}\" | openssl enc -d -aes-256-cbc -base64 -A '\n",
        "  f'-pass pass:{get_epcw(label)} -pbkdf2|gunzip -c')).read()\n",
        "  # logger.debug('\\ndecoded into: '+str(decoded)) # dev only, shows tokens!\n",
        "  try:\n",
        "    return json.loads(decoded)\n",
        "  except JSONDecodeError:\n",
        "    print('Bad passphrase or URI. Try again.')\n",
        "    if 'epw' not in __e: decode(encrypted_string, label) # no endless loop with bad epw\n",
        "  return None\n",
        "\n",
        "def get_token(key_name:str)->str:\n",
        "  '''primitve token fetch'''\n",
        "  ret = decode(c['epc'])[key_name]\n",
        "  # logger.debug('token return: '+str(ret)) # dev only, shows tokens!\n",
        "  if ret.get('token') is not None: return ret['token']\n",
        "  return None\n",
        "\n",
        "def redact_output(text:str,fields:dict)->str:\n",
        "  '''replace output from hidden_fields list with <REDACTED>'''\n",
        "  if isinstance(text,list): text = ' '.join(text) # out as str if list needed for cURL\n",
        "  for field in hidden_fields: # redact hidden fields\n",
        "    if fields.get(field): text = text.replace(fields[field],'<REDACTED>')\n",
        "  return text\n",
        "\n",
        "def cmd_from_template(template:str, conf_args:dict, print_cmd=False, run_cmd=False)->str:\n",
        "  '''uses `api_templates[template](conf)` to get an executable command\n",
        "  using stored conf: `conf=decode(c[epc])[key_name]`\n",
        "  '''\n",
        "  # if arg missing catch for more helpful error message\n",
        "  try: \n",
        "    cmd = api_templates[template](conf_args) # get cmd from template\n",
        "    out = redact_output(cmd, conf_args) # redact hidden_fields\n",
        "    if print_cmd == True: print(out)\n",
        "    if run_cmd == True: exclamation(cmd) # run it!\n",
        "    return out\n",
        "  except KeyError as error:\n",
        "    logger.error('make_request() missing argument: %s for %s',error,template)\n",
        "  return False\n",
        "\n",
        "def make_request(key_name:str, additional_args:dict=None, template:str='')->str:\n",
        "  '''wrapper around cmd_from_template to catch and add missing data; \n",
        "  recovery from missing key_name via `update_epc()`'''\n",
        "  try: # if key_name doesn't exist catch and ask to create\n",
        "    args = decode(c['epc'])[key_name] # get decoded dict for key_name\n",
        "    # default to template with same name as service chosen when created:\n",
        "    if template=='': template = args['service'] \n",
        "    if additional_args != None: args.update(additional_args) # update with addititonal args\n",
        "    return cmd_from_template(template, args, True, True) # get, print, run\n",
        "  except KeyError as error:\n",
        "    yn = input('resource key '+str(error)+' not in encrypted config, create? y/N: ')\n",
        "    if yn=='y': \n",
        "      update_epc(c['epc']) # add it\n",
        "      return make_request(key_name, additional_args, template)\n",
        "    return False\n",
        "\n",
        "def path_free(resource:str, config:dict=None)->bool:\n",
        "  '''basic isdir/isfile checking, will use global c if config not passed, also \n",
        "  useful with `do_setup` if encrypted key_name is a path relative to `projects_dir`'''\n",
        "  if config == None: config = c # use global\n",
        "  isdir = os.path.isdir(config['projects_dir']+resource)\n",
        "  isfile = os.path.isfile(config['projects_dir']+resource)\n",
        "  if isdir or isfile: \n",
        "    logger.warning('Resource exists: '+config['projects_dir']+resource)\n",
        "    return False\n",
        "  return True\n",
        "\n",
        "def do_setup(config:dict)->dict:\n",
        "  '''convenience method, combines `config_dict(c)` (uses `path_logic(c)`), \n",
        "  Google Drive mount (if True), and n `make_request()` from c[resources]; \n",
        "  will run os.system(cmd) on any `c[setup_commands]`'''\n",
        "  config = config_dict(config)\n",
        "  if not 'setup_commands' in config: config['setup_commands']=[]\n",
        "  if config['use_google_drive']==True and not os.path.isdir('/content/drive/'):\n",
        "    drive.mount('/content/drive/')\n",
        "  for cmd in config['setup_commands']:\n",
        "    print(cmd)\n",
        "    os.system(cmd)\n",
        "  if 'resources' in config:\n",
        "    config['setup_commands'].extend(\n",
        "        [make_request(k,d) for k,d in config['resources'].items() if path_free(k,config)]\n",
        "    )\n",
        "  return config"
      ],
      "metadata": {
        "id": "3kK8EuRpmeKf"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Pathfinding\n",
        "Unfortunately we need this here since epc downloads resources."
      ],
      "metadata": {
        "id": "T0CQI1TfbXFn"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## For Clarification (not required) \n"
      ],
      "metadata": {
        "id": "Lsnlv0OXI7Oc"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "The path conventions I'm using in my notebooks assume these are equal:\n",
        "- Google Drive: `Projects/x`\n",
        "- Local: `/home/${USER}/Projects/x`\n",
        "- GitHub Repo/PyPi/npm/etc: `x`\n",
        "\n",
        "Where local environment vars are defined in my `~/.bash_rc` like:\n",
        "```shell\n",
        "ACTIVE_PROJECT=x; export ACTIVE_PROJECT;\n",
        "PROJECTS_DIR=/home/${USER}/Projects/; export PROJECTS_DIR;\n",
        "```\n",
        "They can be anything or not used at all. The main point is **symlinking the Google Drive folder to `{project_path}/x` so paths are still simple when using Colab or SSHFS**. It also assumes the first resource in `c['resources']` is the name of the active project if `$ACTIVE_PROJECT` isn't set (because non-local will need to pull resources from somewhere). It's just a trick to save time as `active_project` can be set manually, including the default in `path_templates`.\n",
        "\n",
        "**Gotcha:** if you're in Colab *and* don't have Google Drive mounted (e.g., working from a `git clone`) you need extra logic elsewhere to pull the JSON from the 'live' notebook being edited and write it to the correct `nb_dir` file. Then you can add/commit/push...and skip all the annoying Drive and GitHub dialogues in Colab. The `%notebook` magic is useful here."
      ],
      "metadata": {
        "id": "QFwZMiw8bcIU"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Path Logic  (now less pathological)"
      ],
      "metadata": {
        "id": "_KK0H0k6aFO4"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "def path_logic(xconf:dict)->dict:\n",
        "  '''sets up paths in config; system environment vars > xconf > defaults\n",
        "  `active_project` precidence: `xconf` > first `xconf['resources']` > `path_templates`'''\n",
        "  if 'setup_commands' not in xconf: xconf['setup_commands']=[]\n",
        "  # attempt to get active_project from resources\n",
        "  if 'resources' in xconf and not 'active_project' in xconf:\n",
        "    xconf['active_project'] = next(iter(xconf['resources']))\n",
        "  # environment vars > `path_templates` defaults\n",
        "  for env_var,lfunc in path_templates.items():\n",
        "    xconf[env_var] = lfunc(xconf) # use path_templates\n",
        "    try:\n",
        "      val = os.environ[env_var.upper()] # bash exports are UPPERCASE\n",
        "      xconf[env_var] = val\n",
        "      xconf['use_google_drive'] = False # don't use drive if system env is set up\n",
        "    except KeyError: pass\n",
        "  # if use_google_drive==True setup needs to symlink active_project to /content\n",
        "  if 'use_google_drive' in xconf and xconf['use_google_drive'] == True:\n",
        "    xconf['setup_commands'].append((f\"ln -s /content/drive/MyDrive/Projects/{xconf['active_project']}/ \"\n",
        "    f\"{xconf['project_path'].rstrip('/')}\"))\n",
        "  return xconf"
      ],
      "metadata": {
        "id": "bD-YerjX9W9l"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Config"
      ],
      "metadata": {
        "id": "CFjwntS9G7aY"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "def config_dict(cxtern:dict)->dict:\n",
        "  '''local variables via optional Colab form into config'''\n",
        "  logger.info('epc.config_dict(c): may be overridden by `c[setting_name]`')\n",
        "  #@title Default Settings { run: \"auto\", vertical-output: true, form-width: \"45%\" }\n",
        "  log_level = logging.DEBUG #@param [\"logging.DEBUG\", \"logging.INFO\", \"logging.WARN\"] {type:\"raw\"}\n",
        "  timezone = 'America/Anchorage' #@param ['America/Anchorage','America/Denver'] {allow-input: true}\n",
        "  use_google_drive = False #@param {type:\"boolean\"}\n",
        "  os.environ['TZ'] = timezone\n",
        "  time.tzset()\n",
        "  logger.setLevel(log_level)\n",
        "  cxtern = {**dict(locals()), **c, **cxtern} # cx > global c > defaults here\n",
        "  cxtern = path_logic(cxtern)\n",
        "  del cxtern['cxtern']\n",
        "  # del cx['c']\n",
        "  for i,k in cxtern.items():\n",
        "    if i=='log_level': k=logging.getLevelName(k)\n",
        "    logger.info('%s: %s',i,k)\n",
        "  return cxtern"
      ],
      "metadata": {
        "id": "iziiGZse_VTl"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Convenience Functions\n",
        "These work with the encrypted config dict using `input()` and `getpass()` prompts, they are not strictly neccesary."
      ],
      "metadata": {
        "id": "-0_VT1PVm40I"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "def get_letter_options(options:dict)->str:\n",
        "  '''turns a list, of, options into a [l]ist, [o]f, [o]ptions''' \n",
        "  return ', '.join(['['+o[0:1]+']'+o[1:] for o in options])\n",
        "\n",
        "def get_input_prompt(key_name:str,name:str,field:str,is_masked:bool,default)->str:\n",
        "  '''text prompt for `input()` based on context'''\n",
        "  if name=='other': name=key_name # 'other' is not useful, use key_name\n",
        "  ret = name+' '+field # e.g., github user\n",
        "  if is_masked is True: \n",
        "    ret+= ' (hidden)' # e.g. github token (hidden)\n",
        "    # if default is not None:\n",
        "    default = str(len(default))+' chars' # e.g. (hidden) [20 chars]\n",
        "  # if default is not None: \n",
        "  ret+=f' [{default}]' # show default if not hidden\n",
        "  return ret+': '\n",
        "\n",
        "def get_user_config_input(name:str, decoded:dict)->dict:\n",
        "  '''uses global api_fields[name] to loop over fields and get input() for vals'''\n",
        "  fields = api_fields[name]\n",
        "  logger.debug(str(fields))\n",
        "  key_name = input('memorable key_name (e.g., huggingface): ')\n",
        "  if decoded.get(key_name) is not None: print(key_name,'key exists')\n",
        "  else: decoded[key_name]=fields\n",
        "  to_encode = [(getpass(get_input_prompt(key_name,name,f,True,decoded[key_name][f])) or decoded[key_name][f]) if f in hidden_fields else (input(get_input_prompt(key_name,name,f,False,decoded[key_name][f])) or decoded[key_name][f]) for f in fields]\n",
        "  to_encode = dict(zip(fields,to_encode))\n",
        "  to_encode['service']=name\n",
        "  decoded[key_name] = to_encode\n",
        "  c['epc'] = encode(decoded)\n",
        "  return decoded\n",
        "\n",
        "def update_epc(epc=None)->dict:\n",
        "  '''work with epc; accepts None, dict, or an encrypted string\n",
        "  :returns decrypted dict'''\n",
        "  deltxt=''\n",
        "  if epc is not None:\n",
        "    if isinstance(epc,dict):\n",
        "      decoded = epc\n",
        "    else: \n",
        "      decoded = decode(epc)\n",
        "    logger.info('--> Entering a key_name that exists will edit, selecting an option not listed will exit.')\n",
        "    print('Existing keys: '+', '.join(decoded.keys()))\n",
        "    deltxt=', [d]elete'\n",
        "  else:\n",
        "    decoded = {}\n",
        "    print(('\\n--> Create a service or simple {key_name:auth_token} entry. '\n",
        "    'Encrypt/decrypt pass should be unique to this system.'))\n",
        "\n",
        "  start_service = input(get_letter_options(api_fields)+deltxt+': ')\n",
        "\n",
        "  if start_service=='o':\n",
        "    decoded = get_user_config_input('other',decoded)\n",
        "  elif start_service=='g':\n",
        "    print('\\n--> Leave filepath blank to clone repo.')\n",
        "    decoded = get_user_config_input('github',decoded)\n",
        "  elif start_service=='f':\n",
        "    print('\\n--> Protocol: ftp or ftps (ports 21 & 22)')\n",
        "    decoded = get_user_config_input('ftp',decoded)\n",
        "  elif start_service=='d':\n",
        "    delete_key = input('\\n--> Enter the key_name you wish to delete: ')\n",
        "    try:\n",
        "      del decoded[delete_key]\n",
        "      encrypted_string = c['epc'] = encode(decoded)\n",
        "      print('Deleted \"',delete_key,'\", encrypted_string is now:',encrypted_string)\n",
        "    except KeyError:\n",
        "      print('Could not find key',delete_key,'please try again.')\n",
        "      update_epc(decoded)\n",
        "  else:\n",
        "    print('Quit or unknown service, confirm with \"n\" to exit.')\n",
        "  \n",
        "  add_another = input('Add/edit another? y/n: ')\n",
        "  if add_another == 'y': update_epc(decoded)\n",
        "  return decoded"
      ],
      "metadata": {
        "id": "ZPLfn3hW8AQ0"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Local Logic"
      ],
      "metadata": {
        "id": "8dddbVyM2SMe"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import os.path\n",
        "import sys\n",
        "from google.colab import _message\n",
        "\n",
        "def get_cell_contents_by_id(cell_id:str,source_or_text:str='source'):\n",
        "  '''get Jupyter cell contents by cell id'''\n",
        "  notebook_json_string = _message.blocking_request('get_ipynb', request='', timeout_sec=5)['ipynb']\n",
        "  try:\n",
        "    for u in notebook_json_string['cells']:\n",
        "      if u['metadata']['id']==cell_id:\n",
        "        if source_or_text=='source':\n",
        "          return ''.join(u['source'])\n",
        "        return u['outputs'][0]['text'][0].strip() #very simple for this\n",
        "  except (KeyError, IndexError):\n",
        "    return None\n",
        "\n",
        "def get_epc_from_nb(path:str=None)->str:\n",
        "  '''pull epc encrypted string from notebook cell'''\n",
        "  # fetch from path instead of running notebook\n",
        "  if path is not None:\n",
        "    notebook_json_string = !cat {path}\n",
        "    nbdata = json.loads(''.join(notebook_json_string))\n",
        "  else:\n",
        "    return get_cell_contents_by_id('9wG_NP_If5r9','text')\n",
        "\n",
        "def epc():\n",
        "  '''finds c['epc'], sets to an encrypted string if dict\n",
        "\n",
        "  search order: existing, notebook contents, epc.txt file \n",
        "  will use if found, otherwise will query user to set epc up\n",
        "  use `get_token()` and `update_epc()` to fetch/modify\n",
        "  '''\n",
        "  decoded = None\n",
        "  # if it's already in the notebook use it\n",
        "  # if we've downloaded it use that (run resets output):\n",
        "  if os.path.isfile('encrypted_epc.ipynb'):\n",
        "    encrypted_string = get_epc_from_nb('encrypted_epc.ipynb')\n",
        "  else:\n",
        "    encrypted_string = get_epc_from_nb() # get from running notebook\n",
        "  if encrypted_string is not None and encrypted_string.strip()!='':\n",
        "    logger.debug('Got encrypted config from notebook: '+encrypted_string)\n",
        "    c['epc'] = encrypted_string\n",
        "  else:\n",
        "    logger.debug('Encrypted string not found in notebook.')\n",
        "    # if the file exists decrypt it\n",
        "    if os.path.isfile('epc.txt'):\n",
        "      epc_txt = !cat 'epc.txt'\n",
        "      # set it to epc\n",
        "      c['epc']=str(epc_txt[0])\n",
        "      logger.debug('epc.txt exists: '+c['epc'])\n",
        "\n",
        "  # see if it's been set (here or another notebook)\n",
        "  try:\n",
        "    logger.debug('epc is set: '+str(c['epc']))\n",
        "    # if it's not encrypted set decrypted then encrypt it\n",
        "    if isinstance(c['epc'],dict):\n",
        "      print('Encrypting epc.')\n",
        "      decoded = c['epc']\n",
        "      c['epc'] = encode(c['epc'])\n",
        "  except (NameError, KeyError):\n",
        "    # if we're here we need to create a new one\n",
        "    print('Config not found.')\n",
        "    decoded = update_epc()"
      ],
      "metadata": {
        "id": "Vdx2yP3Y6ZrA"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Usage\n",
        "For a new config `epc()` and the convenience functions will walk you through creation. After that you can return to modify and create a new `epc.py` file or use `epc.encode()` and `epc.decode()` to do it from anywhere."
      ],
      "metadata": {
        "id": "xd6rj4ZSQcMR"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "c['epc'] = 'U2FsdGVkX1/xUi/hYJ+V9stP7WV+Y6fTtkJNZoDL8PX4mSWh/CuEAZUz0VI8r/BvrfQztrCI1rPmduBRSJsTLQeD9AKFNjHMNq15V1ArU7CvCVVXvuKgNsDS0aEN2hJoRMXiy4g7YzIB/b45e3vq8/DV/sFFml1GtWeTtIWLeA1c92UNrF8s3MjpvWhYB+GeD1OSQMcS3aoveW7YD8tSigzRkZQqwmoNrHlNdODtNeLq4idK4VsD9mauLGg/mWKitdfbqUi0yULYpg5pRxPMZaIy4sDFdGYbeaHYBaduf36Wr+ak3x6e7rvT7fF04EST3wv7FM54xAxZiyohgUHIyG+5oYBbvcWsKl74GZpoPYqUffxdnnH8VKURHTE28WaXwdsXwj0edbkX3cNcL/aCOdF9pfCcNPJFbRD0LM7E2sF9xYeqyNMhpnBMKWfXex7rEzj2ZRQbxlRy+f+476Vni/Qf2Ixcc/x5vCvAEJgCueD+loXol4oP3O4eDM6Q1FiPxMDOCYGxJ8TCFVb5795vqsmWsKLfa1pOqnPSJ2Id4ayiryeyxtDShRD8fTq0RplfXU7JMKnrCZJNq4CW+rDSRTOy4cHE0S3zy1EbFtPVP3Y4MCCKaWIHa/LVj9LDZSQvYSZO4xZb3Aw+fuvxqI/4Ow=='"
      ],
      "metadata": {
        "id": "rkzhTdSwSmDK"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## get epc from var, notebook, etc"
      ],
      "metadata": {
        "id": "Zx0ckxco8P7v"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "epc()\n",
        "# import epc"
      ],
      "metadata": {
        "id": "3l0A8aTXLHzg",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "a8bf7d35-f3ce-43f7-a5e8-aa1ebf8a2847"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "2023-01-18 17:04:50 epc DEBUG    Encrypted string not found in notebook.\n",
            "2023-01-18 17:04:50 epc DEBUG    epc is set: U2FsdGVkX1/xUi/hYJ+V9stP7WV+Y6fTtkJNZoDL8PX4mSWh/CuEAZUz0VI8r/BvrfQztrCI1rPmduBRSJsTLQeD9AKFNjHMNq15V1ArU7CvCVVXvuKgNsDS0aEN2hJoRMXiy4g7YzIB/b45e3vq8/DV/sFFml1GtWeTtIWLeA1c92UNrF8s3MjpvWhYB+GeD1OSQMcS3aoveW7YD8tSigzRkZQqwmoNrHlNdODtNeLq4idK4VsD9mauLGg/mWKitdfbqUi0yULYpg5pRxPMZaIy4sDFdGYbeaHYBaduf36Wr+ak3x6e7rvT7fF04EST3wv7FM54xAxZiyohgUHIyG+5oYBbvcWsKl74GZpoPYqUffxdnnH8VKURHTE28WaXwdsXwj0edbkX3cNcL/aCOdF9pfCcNPJFbRD0LM7E2sF9xYeqyNMhpnBMKWfXex7rEzj2ZRQbxlRy+f+476Vni/Qf2Ixcc/x5vCvAEJgCueD+loXol4oP3O4eDM6Q1FiPxMDOCYGxJ8TCFVb5795vqsmWsKLfa1pOqnPSJ2Id4ayiryeyxtDShRD8fTq0RplfXU7JMKnrCZJNq4CW+rDSRTOy4cHE0S3zy1EbFtPVP3Y4MCCKaWIHa/LVj9LDZSQvYSZO4xZb3Aw+fuvxqI/4Ow==\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Setup Config"
      ],
      "metadata": {
        "id": "PH2VaRry8hvj"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "config_dict(c)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "tCgA5RF08C7Q",
        "outputId": "caa41bc5-fffa-4fa7-ab59-9529de846809"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "2023-01-18 17:04:50 epc INFO     epc.config_dict(c): may be overridden by `c[setting_name]`\n",
            "2023-01-18 08:04:50 epc INFO     log_level: DEBUG\n",
            "2023-01-18 08:04:50 epc INFO     timezone: America/Anchorage\n",
            "2023-01-18 08:04:50 epc INFO     use_google_drive: False\n",
            "2023-01-18 08:04:50 epc INFO     epc: U2FsdGVkX1/xUi/hYJ+V9stP7WV+Y6fTtkJNZoDL8PX4mSWh/CuEAZUz0VI8r/BvrfQztrCI1rPmduBRSJsTLQeD9AKFNjHMNq15V1ArU7CvCVVXvuKgNsDS0aEN2hJoRMXiy4g7YzIB/b45e3vq8/DV/sFFml1GtWeTtIWLeA1c92UNrF8s3MjpvWhYB+GeD1OSQMcS3aoveW7YD8tSigzRkZQqwmoNrHlNdODtNeLq4idK4VsD9mauLGg/mWKitdfbqUi0yULYpg5pRxPMZaIy4sDFdGYbeaHYBaduf36Wr+ak3x6e7rvT7fF04EST3wv7FM54xAxZiyohgUHIyG+5oYBbvcWsKl74GZpoPYqUffxdnnH8VKURHTE28WaXwdsXwj0edbkX3cNcL/aCOdF9pfCcNPJFbRD0LM7E2sF9xYeqyNMhpnBMKWfXex7rEzj2ZRQbxlRy+f+476Vni/Qf2Ixcc/x5vCvAEJgCueD+loXol4oP3O4eDM6Q1FiPxMDOCYGxJ8TCFVb5795vqsmWsKLfa1pOqnPSJ2Id4ayiryeyxtDShRD8fTq0RplfXU7JMKnrCZJNq4CW+rDSRTOy4cHE0S3zy1EbFtPVP3Y4MCCKaWIHa/LVj9LDZSQvYSZO4xZb3Aw+fuvxqI/4Ow==\n",
            "2023-01-18 08:04:50 epc INFO     setup_commands: []\n",
            "2023-01-18 08:04:50 epc INFO     projects_dir: /content/\n",
            "2023-01-18 08:04:50 epc INFO     active_project: app\n",
            "2023-01-18 08:04:50 epc INFO     project_path: /content/app/\n",
            "2023-01-18 08:04:50 epc INFO     nb_dir: /content/app/notebooks/\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{'log_level': 10,\n",
              " 'timezone': 'America/Anchorage',\n",
              " 'use_google_drive': False,\n",
              " 'epc': 'U2FsdGVkX1/xUi/hYJ+V9stP7WV+Y6fTtkJNZoDL8PX4mSWh/CuEAZUz0VI8r/BvrfQztrCI1rPmduBRSJsTLQeD9AKFNjHMNq15V1ArU7CvCVVXvuKgNsDS0aEN2hJoRMXiy4g7YzIB/b45e3vq8/DV/sFFml1GtWeTtIWLeA1c92UNrF8s3MjpvWhYB+GeD1OSQMcS3aoveW7YD8tSigzRkZQqwmoNrHlNdODtNeLq4idK4VsD9mauLGg/mWKitdfbqUi0yULYpg5pRxPMZaIy4sDFdGYbeaHYBaduf36Wr+ak3x6e7rvT7fF04EST3wv7FM54xAxZiyohgUHIyG+5oYBbvcWsKl74GZpoPYqUffxdnnH8VKURHTE28WaXwdsXwj0edbkX3cNcL/aCOdF9pfCcNPJFbRD0LM7E2sF9xYeqyNMhpnBMKWfXex7rEzj2ZRQbxlRy+f+476Vni/Qf2Ixcc/x5vCvAEJgCueD+loXol4oP3O4eDM6Q1FiPxMDOCYGxJ8TCFVb5795vqsmWsKLfa1pOqnPSJ2Id4ayiryeyxtDShRD8fTq0RplfXU7JMKnrCZJNq4CW+rDSRTOy4cHE0S3zy1EbFtPVP3Y4MCCKaWIHa/LVj9LDZSQvYSZO4xZb3Aw+fuvxqI/4Ow==',\n",
              " 'setup_commands': [],\n",
              " 'projects_dir': '/content/',\n",
              " 'active_project': 'app',\n",
              " 'project_path': '/content/app/',\n",
              " 'nb_dir': '/content/app/notebooks/'}"
            ]
          },
          "metadata": {},
          "execution_count": 10
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## create a small helper script\n",
        "Run this cell and download/upload the `epc.py` file. Example of it uploading itself via FTP is on the last line of the cell. \n",
        "\n",
        "This hard-codes the encrypted data to `c['epc']` in the generated file."
      ],
      "metadata": {
        "id": "IXHtzJjrNunC"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "write_cells = {\n",
        "    'imports':'50BVBBzCE0Hw',\n",
        "    'lists':'318_VIecZoB_',\n",
        "    'main':'3kK8EuRpmeKf',\n",
        "    'paths':'bD-YerjX9W9l',\n",
        "    'config':'iziiGZse_VTl',\n",
        "    'utils':'ZPLfn3hW8AQ0'\n",
        "}\n",
        "\n",
        "pc = \"\\nc['epc'] = '\"+c['epc']+\"'\" # hard-code our encrypted string\n",
        "out=''\n",
        "for k,v in write_cells.items():\n",
        "  if k!='imports': out+='\\n# '+k.upper()\n",
        "  out+='\\n'+get_cell_contents_by_id(v)+'\\n' # get notebook cell content by id\n",
        "with open('epc.py', 'w') as f:\n",
        "    f.write(out+pc)"
      ],
      "metadata": {
        "id": "hqbkZkyXnMSd"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "I like fitting as much code on screen as possible so my style is not generally acceptable for public consumption. Fix it and run some checks."
      ],
      "metadata": {
        "id": "sImPDRwKf_aB"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# clean up my compressed style\n",
        "# lots more extensions https://pylint.pycqa.org/en/latest/user_guide/checkers/extensions.html\n",
        "which_pylint = !which pylint\n",
        "if len(which_pylint)==0:\n",
        "  %pip install pylint\n",
        "  %pip install autopep8\n",
        "  %pip install yapf\n",
        "\n",
        "# E121 is indentation of 4 spaces\n",
        "!autopep8 --in-place --aggressive --aggressive epc.py --indent-size=2 --ignore=E121\n",
        "!yapf -i epc.py --style='{based_on_style: pep8, indent_width: 2}'\n",
        "!pylint --good-names=k,v,c,d,i,yn epc.py --indent-string='  '\n",
        "#--load-plugins=pylint.extensions.confusing_elif,pylint.extensions.for_any_all,pylint.extensions.mccabe \n",
        "\n",
        "file_size = os.path.getsize('epc.py')/1024\n",
        "print('Size of epc.py:',round(file_size,2),'kB')\n",
        "assert file_size < 15 #1kB padding to mitigate my long testing epc\n",
        "\n",
        "# # !cat epc.py"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "fPazF4HAp06E",
        "outputId": "4fa5a503-9432-4ba5-8537-1e480479ed55"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "************* Module epc\n",
            "epc.py:391:0: C0301: Line too long (677/100) (line-too-long)\n",
            "epc.py:283:2: W0641: Possibly unused variable 'use_google_drive' (possibly-unused-variable)\n",
            "\n",
            "------------------------------------------------------------------\n",
            "Your code has been rated at 9.90/10 (previous run: 9.90/10, +0.00)\n",
            "\n",
            "Size of epc.py: 13.8 kB\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Cyclomatic complexity isn't the perfect metric but it is the one I find most helpful to keeping my compress-everything and procedural habits from creating incomprehensible code for other people."
      ],
      "metadata": {
        "id": "7nTAd_Tggb-A"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "which_radon=!which radon\n",
        "if len(which_radon)==0:\n",
        "  %pip install radon\n",
        "!radon cc epc.py -as"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "iAJ_IsWKTwgD",
        "outputId": "803d7579-c5bb-4db4-84dd-599009106f4d"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "epc.py\n",
            "    \u001b[1m\u001b[35mF \u001b[0m343:0 update_epc - \u001b[32mB (9)\u001b[0m\n",
            "    \u001b[1m\u001b[35mF \u001b[0m223:0 do_setup - \u001b[32mB (8)\u001b[0m\n",
            "    \u001b[1m\u001b[35mF \u001b[0m246:0 path_logic - \u001b[32mB (8)\u001b[0m\n",
            "    \u001b[1m\u001b[35mF \u001b[0m321:0 get_user_config_input - \u001b[32mB (6)\u001b[0m\n",
            "    \u001b[1m\u001b[35mF \u001b[0m188:0 make_request - \u001b[32mA (5)\u001b[0m\n",
            "    \u001b[1m\u001b[35mF \u001b[0m104:0 get_epcw - \u001b[32mA (4)\u001b[0m\n",
            "    \u001b[1m\u001b[35mF \u001b[0m157:0 redact_output - \u001b[32mA (4)\u001b[0m\n",
            "    \u001b[1m\u001b[35mF \u001b[0m167:0 cmd_from_template - \u001b[32mA (4)\u001b[0m\n",
            "    \u001b[1m\u001b[35mF \u001b[0m210:0 path_free - \u001b[32mA (4)\u001b[0m\n",
            "    \u001b[1m\u001b[35mF \u001b[0m113:0 exclamation - \u001b[32mA (3)\u001b[0m\n",
            "    \u001b[1m\u001b[35mF \u001b[0m133:0 decode - \u001b[32mA (3)\u001b[0m\n",
            "    \u001b[1m\u001b[35mF \u001b[0m275:0 config_dict - \u001b[32mA (3)\u001b[0m\n",
            "    \u001b[1m\u001b[35mF \u001b[0m306:0 get_input_prompt - \u001b[32mA (3)\u001b[0m\n",
            "    \u001b[1m\u001b[35mF \u001b[0m12:0 get_logger - \u001b[32mA (2)\u001b[0m\n",
            "    \u001b[1m\u001b[35mF \u001b[0m148:0 get_token - \u001b[32mA (2)\u001b[0m\n",
            "    \u001b[1m\u001b[35mF \u001b[0m301:0 get_letter_options - \u001b[32mA (2)\u001b[0m\n",
            "    \u001b[1m\u001b[35mF \u001b[0m124:0 encode - \u001b[32mA (1)\u001b[0m\n",
            "\n",
            "17 blocks (classes, functions, methods) analyzed.\n",
            "Average complexity: \u001b[32mA (4.176470588235294)\u001b[0m\n",
            "\u001b[0m"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## View Existing Keys/Configured API Services"
      ],
      "metadata": {
        "id": "p4JoWEMG8aC0"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "decode(c['epc']).keys()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "MrrqN3zT8FrB",
        "outputId": "f70e67f5-489f-485a-e5ae-c1ee7dc10178"
      },
      "execution_count": null,
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Save passphrase as variable, leave blank to enter each time: ··········\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "dict_keys(['jai', 'atomic', 'atomic.db', 'huggingface', 'ftp-m', 'test', 'dropbox_refresh', 'dropbox_app_secret', 'github_user', 'github_email', 'github', 'ngrok'])"
            ]
          },
          "metadata": {},
          "execution_count": 14
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "api_templates.keys()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "2CWoTINH-914",
        "outputId": "6e8223c5-7752-4441-f8eb-e96d46ca1c08"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "dict_keys(['ftp', 'other', 'github', 'github_file', 'huggingface'])"
            ]
          },
          "metadata": {},
          "execution_count": 15
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "api_fields['ftp']"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "23xeQxbR_V0i",
        "outputId": "3c024370-c4cb-402f-c674-6ce384cc05d7"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{'user': None,\n",
              " 'host': None,\n",
              " 'passwd': None,\n",
              " 'protocol': 'ftp',\n",
              " 'port': 21,\n",
              " 'action': '-o'}"
            ]
          },
          "metadata": {},
          "execution_count": 16
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## update/edit/add config"
      ],
      "metadata": {
        "id": "ECw4Cri_1q3i"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# update_epc(c['epc'])"
      ],
      "metadata": {
        "id": "Pg7ZFXXY5kwr"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## one-liner setup config and download resources"
      ],
      "metadata": {
        "id": "3454J302A-Vg"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "config = do_setup({'log_level':10, 'resources':{'jai':{},'ftp-m':{'local_path':'setuptest','remote_path':'site.webmanifest'}}})\n",
        "# c={'resources':{'jai':{}},'use_google_drive':False}\n",
        "# do_setup(c)"
      ],
      "metadata": {
        "id": "L_X9Fxv-FrGc",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "6576466e-5b7e-4cbc-e6e1-9834612c1caf"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "2023-01-18 08:05:16 epc INFO     epc.config_dict(c): may be overridden by `c[setting_name]`\n",
            "2023-01-18 08:05:16 epc INFO     log_level: DEBUG\n",
            "2023-01-18 08:05:16 epc INFO     timezone: America/Anchorage\n",
            "2023-01-18 08:05:16 epc INFO     use_google_drive: False\n",
            "2023-01-18 08:05:16 epc INFO     epc: U2FsdGVkX1/xUi/hYJ+V9stP7WV+Y6fTtkJNZoDL8PX4mSWh/CuEAZUz0VI8r/BvrfQztrCI1rPmduBRSJsTLQeD9AKFNjHMNq15V1ArU7CvCVVXvuKgNsDS0aEN2hJoRMXiy4g7YzIB/b45e3vq8/DV/sFFml1GtWeTtIWLeA1c92UNrF8s3MjpvWhYB+GeD1OSQMcS3aoveW7YD8tSigzRkZQqwmoNrHlNdODtNeLq4idK4VsD9mauLGg/mWKitdfbqUi0yULYpg5pRxPMZaIy4sDFdGYbeaHYBaduf36Wr+ak3x6e7rvT7fF04EST3wv7FM54xAxZiyohgUHIyG+5oYBbvcWsKl74GZpoPYqUffxdnnH8VKURHTE28WaXwdsXwj0edbkX3cNcL/aCOdF9pfCcNPJFbRD0LM7E2sF9xYeqyNMhpnBMKWfXex7rEzj2ZRQbxlRy+f+476Vni/Qf2Ixcc/x5vCvAEJgCueD+loXol4oP3O4eDM6Q1FiPxMDOCYGxJ8TCFVb5795vqsmWsKLfa1pOqnPSJ2Id4ayiryeyxtDShRD8fTq0RplfXU7JMKnrCZJNq4CW+rDSRTOy4cHE0S3zy1EbFtPVP3Y4MCCKaWIHa/LVj9LDZSQvYSZO4xZb3Aw+fuvxqI/4Ow==\n",
            "2023-01-18 08:05:16 epc INFO     resources: {'jai': {}, 'ftp-m': {'local_path': 'setuptest', 'remote_path': 'site.webmanifest'}}\n",
            "2023-01-18 08:05:16 epc INFO     setup_commands: []\n",
            "2023-01-18 08:05:16 epc INFO     active_project: jai\n",
            "2023-01-18 08:05:16 epc INFO     projects_dir: /content/\n",
            "2023-01-18 08:05:16 epc INFO     project_path: /content/jai/\n",
            "2023-01-18 08:05:16 epc INFO     nb_dir: /content/jai/notebooks/\n",
            "2023-01-18 08:05:16 epc WARNING  Resource exists: /content/jai\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "curl --user jaciss:<REDACTED> -osetuptest ftp://maya.software:21/site.webmanifest\n",
            "  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current\n",
            "                                 Dload  Upload   Total   Spent    Left  Speed\n",
            "100   263  100   263    0     0    441      0 --:--:-- --:--:-- --:--:--   441\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## get a simple token"
      ],
      "metadata": {
        "id": "_Ojspjt51upq"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "get_token('test')"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 36
        },
        "id": "bGyp_m1aLV-F",
        "outputId": "1c62f385-b651-4a08-e3a6-2fe41006c7ec"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "'token'"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            }
          },
          "metadata": {},
          "execution_count": 19
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## get a file from a private repo\n",
        "Note: someone with the passphrase or token could still access the entire repo."
      ],
      "metadata": {
        "id": "B1k85UpZ1zM4"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "fname = 'atomic.db'\n",
        "os.path.isfile(fname) or make_request(fname,{},'github_file')"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "14JYAThvRyXj",
        "outputId": "1dd905ab-5fd1-422d-e76f-437f9fe16747"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "True"
            ]
          },
          "metadata": {},
          "execution_count": 20
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "the above make_request is a fancy version of: "
      ],
      "metadata": {
        "id": "8ENeSw4cPGEA"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# decode the config, get config[key_name], pass it through the template\n",
        "cmd = api_templates['github_file'](decode(c['epc'])['atomic.db'])\n",
        "# run the command via Python's Popen\n",
        "exclamation(cmd)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "byUWntRBPPng",
        "outputId": "3f15bf61-b568-485d-be7f-a910a79d360b"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current\n",
            "                                 Dload  Upload   Total   Spent    Left  Speed\n",
            "100 4268k  100 4268k    0     0  1541k      0  0:00:02  0:00:02 --:--:-- 1541k\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## clone a repo"
      ],
      "metadata": {
        "id": "a3OZ_DFS19ac"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "repo = 'jai'\n",
        "os.path.isdir(repo) or make_request(repo)"
      ],
      "metadata": {
        "id": "2mFT3KGJR7fG",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "dc12c68c-a6f3-4e51-e5fe-b7197b0a482b"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "True"
            ]
          },
          "metadata": {},
          "execution_count": 22
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## upload a file using ftp\n",
        "Use `-T` for uploads and default `-o` for downloads.\n",
        "\n",
        "`curl` can do a lot, try `!curl -V` to get an idea of the options. "
      ],
      "metadata": {
        "id": "pm_hwjSBFj3t"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# upload our helper script via FTP\n",
        "cmd = make_request('ftp-m',{'local_path':'epc.py','remote_path':'epc.txt','action':'-T'})"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "WhEliVmFFjUd",
        "outputId": "ef26fe49-e5bc-4fae-cf2c-5771f68d724d"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "curl --user jaciss:<REDACTED> -Tepc.py ftp://maya.software:21/epc.txt\n",
            "  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current\n",
            "                                 Dload  Upload   Total   Spent    Left  Speed\n",
            "100 14129    0     0  100 14129      0  41312 --:--:-- --:--:-- --:--:-- 41192\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### forget a parameter name and be reminded"
      ],
      "metadata": {
        "id": "jjmCvAOmMMj8"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "cmd = make_request('ftp-m',{'local_file':'epc.py','remote_file':'epc.txt','action':'-T'})"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "wyATCTZ8MWrr",
        "outputId": "82a954f3-f9ff-4265-9241-30352645967d"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "2023-01-18 08:05:20 epc ERROR    make_request() missing argument: 'local_path' for ftp\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## make a Hugging Face request\n",
        "uses a simple `key_name{token:val}` dict combined with additional arguments and a template"
      ],
      "metadata": {
        "id": "0T_agcFzMbhG"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "cmd = make_request(key_name='huggingface',template='huggingface',additional_args={'model':'bert-base-uncased','inputs':'\"the answer to life the universe and everything is [MASK]\"'})"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "b-c6eT-_MTp0",
        "outputId": "5e291256-ab4d-45f2-ce55-da664fe3fc58"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "curl https://api-inference.huggingface.co/models/bert-base-uncased -X POST -d {\"inputs\": \"the answer to life the universe and everything is [MASK]\"} -H Authorization: Bearer <REDACTED>\n",
            "  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current\n",
            "                                 Dload  Upload   Total   Spent    Left  Speed\n",
            "100   690  100   620  100    70   3668    414 --:--:-- --:--:-- --:--:--  4107\n",
            "[{\"score\":0.8963303565979004,\"token\":1012,\"token_str\":\".\",\"sequence\":\"the answer to life the universe and everything is.\"},{\"score\":0.07321417331695557,\"token\":1025,\"token_str\":\";\",\"sequence\":\"the answer to life the universe and everything is ;\"},{\"score\":0.017070315778255463,\"token\":1029,\"token_str\":\"?\",\"sequence\":\"the answer to life the universe and everything is?\"},{\"score\":0.012058394961059093,\"token\":999,\"token_str\":\"!\",\"sequence\":\"the answer to life the universe and everything is!\"},{\"score\":0.0007162163965404034,\"token\":1064,\"token_str\":\"|\",\"sequence\":\"the answer to life the universe and everything is |\"}]"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## download a file using ftp"
      ],
      "metadata": {
        "id": "v6SoxLb1cKJ0"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "fname = 'ftp_download_test'\n",
        "make_request('ftp-m',{'local_path':fname,'remote_path':'site.webmanifest'})\n",
        "!cat {fname}"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "RaC8bhC0Iblv",
        "outputId": "4083c820-4a58-4876-922a-33befd05f377"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "curl --user jaciss:<REDACTED> -oftp_download_test ftp://maya.software:21/site.webmanifest\n",
            "  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current\n",
            "                                 Dload  Upload   Total   Spent    Left  Speed\n",
            "100   263  100   263    0     0    749      0 --:--:-- --:--:-- --:--:--   749\n",
            "{\"name\":\"\",\"short_name\":\"\",\"icons\":[{\"src\":\"/android-chrome-192x192.png\",\"sizes\":\"192x192\",\"type\":\"image/png\"},{\"src\":\"/android-chrome-512x512.png\",\"sizes\":\"512x512\",\"type\":\"image/png\"}],\"theme_color\":\"#ffffff\",\"background_color\":\"#ffffff\",\"display\":\"standalone\"}"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## upload a file using Dropbox\n",
        "Assumes you have a valid dropbox API setup with an dropbox_app_secret and dropbox_reresh stored in the dict.\n",
        "\n",
        "Walkthrough of how to get these: https://www.dropboxforum.com/t5/Dropbox-API-Support-Feedback/Issue-in-generating-access-token/m-p/592667\n",
        "\n",
        "Info on file uploads: https://www.dropbox.com/developers/documentation/http/documentation#files-upload for more details.\n"
      ],
      "metadata": {
        "id": "j1E04ejZxhgk"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "app_secret = get_token('dropbox_app_secret')\n",
        "refresh = get_token('dropbox_refresh')\n",
        "\n",
        "# this turns app_secret and refresh_token into json response with actual token\n",
        "json_token = !curl https://api.dropbox.com/oauth2/token -d grant_type=refresh_token -d refresh_token={refresh} -u '082mobpk9chcskt:{app_secret}'\n",
        "# print(json_token)\n",
        "access_token = json.loads(json_token[0])['access_token']\n",
        "\n",
        "# mode:add or overwrite, double-backslashes needed when using {}\n",
        "from_path = 'epc.py'\n",
        "to_path = '/epc.py'\n",
        "command = 'curl -X POST https://content.dropboxapi.com/2/files/upload --header \"Authorization: Bearer '+access_token+'\" --header \"Dropbox-API-Arg: {\\\\\"autorename\\\\\":false,\\\\\"mode\\\\\":\\\\\"overwrite\\\\\",\\\\\"mute\\\\\":false,\\\\\"path\\\\\":\\\\\"'+to_path+'\\\\\",\\\\\"strict_conflict\\\\\":false}\" --header \"Content-Type: application/octet-stream\" --data-binary @'+from_path\n",
        "!{command}\n"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "jpV3KnYrdFxu",
        "outputId": "4b5b0c32-05fe-4164-d028-aab20d2c8f95"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "{\"name\": \"epc.py\", \"path_lower\": \"/epc.py\", \"path_display\": \"/epc.py\", \"id\": \"id:xMw2KFKFNpQAAAAAAAVZCg\", \"client_modified\": \"2023-01-18T17:05:22Z\", \"server_modified\": \"2023-01-18T17:05:23Z\", \"rev\": \"5f28cd26b6d1c0010a9f5\", \"size\": 14129, \"is_downloadable\": true, \"content_hash\": \"52b76c8f7d6dbe0efdd550cb6deaa3286681f39c53a80cd2fb5dd154936c4180\"}"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## manually edit the dict\n",
        "Make sure you **clear or update the final encrypted output cell** so `epc()` doesn't use old data."
      ],
      "metadata": {
        "id": "x51MUAhmRb2C"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# decoded = decode(c['epc']) # decode the encrypted string\n",
        "# decoded['new_key']={'token':'newtoken'} # add a new key/token combination\n",
        "# c['epc']=encode(decoded) # encode it\n",
        "# decode(c['epc']).keys() # print all keys\n",
        "# # decode(c['epc']) # will print tokens in plain-text!"
      ],
      "metadata": {
        "id": "QKZO5-MBkHnf"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "It may be less error-prone to use the helpers in `update_epc()` for ad-hoc changes."
      ],
      "metadata": {
        "id": "jW_0aN6Dk8b0"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# update_epc() # use what is in the notebook or epc.txt file\n",
        "# update_epc(c['epc']) # use what is in an existing encrypted string"
      ],
      "metadata": {
        "id": "3WJRfZtgPxe0"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Find token in JSON of downloaded ipynb file\n",
        "Instead of active notebook, e.g., if you upload the entire notebook file to an FTP server and fetch with `wget` or similar instead of using `epc.py`."
      ],
      "metadata": {
        "id": "uapoHtt0CSfs"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# get_epc_from_nb('encrypted_epc.ipynb')"
      ],
      "metadata": {
        "id": "bDECx1zvCKly"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Final encrypted private config\n",
        "This cell acts as 'storage' for the encrypted string and is used by `epc()` to get/set `c['epc']`. If you use epc.py the encrypted string is hard-coded into the file and it will need to be regenerated and uploaded after modifications."
      ],
      "metadata": {
        "id": "dSdxEnS4m8nF"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# del c['epc']\n",
        "# !rm epc.txt\n",
        "try:\n",
        "  print(c['epc'])\n",
        "except KeyError:\n",
        "  print()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "9wG_NP_If5r9",
        "outputId": "1faedff7-5c21-4d40-e532-a3b6cd9907bb"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "U2FsdGVkX1/xUi/hYJ+V9stP7WV+Y6fTtkJNZoDL8PX4mSWh/CuEAZUz0VI8r/BvrfQztrCI1rPmduBRSJsTLQeD9AKFNjHMNq15V1ArU7CvCVVXvuKgNsDS0aEN2hJoRMXiy4g7YzIB/b45e3vq8/DV/sFFml1GtWeTtIWLeA1c92UNrF8s3MjpvWhYB+GeD1OSQMcS3aoveW7YD8tSigzRkZQqwmoNrHlNdODtNeLq4idK4VsD9mauLGg/mWKitdfbqUi0yULYpg5pRxPMZaIy4sDFdGYbeaHYBaduf36Wr+ak3x6e7rvT7fF04EST3wv7FM54xAxZiyohgUHIyG+5oYBbvcWsKl74GZpoPYqUffxdnnH8VKURHTE28WaXwdsXwj0edbkX3cNcL/aCOdF9pfCcNPJFbRD0LM7E2sF9xYeqyNMhpnBMKWfXex7rEzj2ZRQbxlRy+f+476Vni/Qf2Ixcc/x5vCvAEJgCueD+loXol4oP3O4eDM6Q1FiPxMDOCYGxJ8TCFVb5795vqsmWsKLfa1pOqnPSJ2Id4ayiryeyxtDShRD8fTq0RplfXU7JMKnrCZJNq4CW+rDSRTOy4cHE0S3zy1EbFtPVP3Y4MCCKaWIHa/LVj9LDZSQvYSZO4xZb3Aw+fuvxqI/4Ow==\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "len(c['epc'])"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "2eEfhaD0i5Cr",
        "outputId": "02d913f0-fddc-4894-b076-9b3429b565d2"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "664"
            ]
          },
          "metadata": {},
          "execution_count": 32
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# for finding cell ids ;)\n",
        "# notebook_json_string = _message.blocking_request('get_ipynb', request='', timeout_sec=5)['ipynb']\n",
        "# for u in notebook_json_string['cells']:\n",
        "#   print(u)"
      ],
      "metadata": {
        "id": "LjkyZThySYV9"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# c['setup_commands']=[]\n",
        "# tsetup={'use_google_drive':True}\n",
        "# tsetup={'active_project':'jai','resources':{'sample_data':{'epc_key_name'}},'use_google_drive':True}\n",
        "# tsetup={'resources':{'jai':{}},'use_google_drive':True}\n",
        "# do_setup(tsetup)\n",
        "\n",
        "# do_setup({'setup_commands':['touch iwin']})\n",
        "\n",
        "# c={}\n",
        "# config_dict({})\n",
        "# config_dict({'use_google_drive':True})\n",
        "# config_dict({'active_project':'jai','resources':{'test':{'epc_key_name'}},'use_google_drive':True})\n",
        "# config_dict({'resources':{'test':{'epc_key_name'}},'use_google_drive':False})\n",
        "# %env ACTIVE_PROJECT=tee\n",
        "# %env PROJECTS_DIR=/home/${USER}/Projects/\n",
        "# xconf = {'active_project':'jai','resources':{'test':{'epc_key_name'}},'use_google_drive':True}\n",
        "# xconf = {'resources':{'test':{'epc_key_name'}},'use_google_drive':True}\n",
        "# path_logic(xconf)\n",
        "\n",
        "# %env ACTIVE_PROJECT=beach\n",
        "# %env PROJECTS_DIR=/home/user/Projects/\n",
        "# del os.environ['ACTIVE_PROJECT']\n",
        "# del os.environ['PROJECTS_DIR']\n",
        "# del os.environ['NOTEBOOKS_DIR']\n",
        "# path_logic({'resources':['jai'], 'use_google_drive':False})\n",
        "# path_logic({})\n",
        "\n",
        "# config_dict(c) # for testing, prints output\n",
        "# config_dict({})\n",
        "# config_dict({'resources':['jai'],'use_google_drive':True})\n",
        "# config_dict({'log_level':10,'projects_dir':'/content/'})\n",
        "# config_dict({'log_level':30,'projects_dir':'/home/user/Projects/'})\n",
        "# config_dict({'active_project':'win'})\n",
        "# config_dict({'active_project':'win','projects_dir':'/home/user/Projects/'})\n",
        "# %env NOTEBOOKS_DIR=/home/user/Notebooks\n",
        "# del os.environ['NOTEBOOKS_DIR']\n",
        "# config_dict({'resources':['jai'],'use_google_drive':True,'active_project':'win'})"
      ],
      "metadata": {
        "id": "8faYN2uIghMB"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "# BDD Tests\n",
        "testkey,testtoken,testpass: U2FsdGVkX1+VPA14uYWYT+AhjF7zo8/Cu/rXqyC2rUw9hr65jrCXtCs4cVKhhk5j"
      ],
      "metadata": {
        "id": "dEt-kp7BQ8sy"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# !mkdir -p jai/features\n",
        "# !mkdir -p jai/features/steps\n",
        "# if 'error' in decoded:\n",
        "  #   print('openssl error:',decoded,' try again.')\n",
        "  #   decode(encrypted_string)"
      ],
      "metadata": {
        "id": "J_rk0clt0jRX"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# import os.path\n",
        "# from google.colab import drive\n",
        "\n",
        "# c = {'nb_name':'encrypted_epc', 'nb_run_by':nb_name, 'nb_dir':'drive/MyDrive/Colab\\ Notebooks/'} # if local set c\n",
        "# if not os.path.isdir('/content/drive/'): \n",
        "#   drive.mount('/content/drive/')\n",
        "# %run {c['nb_dir']}bootstrap_a_notebook.ipynb\n",
        "# %run {c['nb_dir']}bdd_testing_a_notebook.ipynb"
      ],
      "metadata": {
        "id": "lrlovB0_TWEP"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# print(c.get('nb_name'))\n",
        "# # %pip install coverage\n",
        "# # %pip install behave\n",
        "# # test_notebook(c.get('nb_name'))\n",
        "# !coverage run --source=jai.nbpy.encryptedprivateconfig.py -m behave"
      ],
      "metadata": {
        "id": "VTPQGsy1XO3w"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# %%writefile jai/features/{c.get('nb_name')}.feature\n",
        "# Feature: private passphrase-encrypted config for auth tokens\n",
        "\n",
        "#   Scenario: no epc.txt file or epc set\n",
        "#     Given the file \"epc.txt\" does not exist\n",
        "#     And configuration key \"epc\" is not set\n",
        "#     When we run notebook \"encrypted_epc.ipynb\"\n",
        "#     Then we see \"Config not found\" in the output\n",
        "\n",
        "# # testkey,testval,testpass: U2FsdGVkX18CQBDLSi6MGuk3EqujTslEYcJEG37zqDiO0QbsSIjPPxB1IDTJUfnx"
      ],
      "metadata": {
        "id": "4uXGuM4GJ0vZ"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "  Scenario: create config for the first time\n",
        "    Given the file \"epc.txt\" does not exist\n",
        "    And configuration key \"epc\" is not set\n",
        "    When we type \"o\"\n",
        "    And we type \"huggingface\"\n",
        "    And we type \"testtoken\"\n",
        "    Then we see \"saved\" in the output\n",
        "    And the file \"epc.txt\" exists\n",
        "\n",
        "  Scenario: no epc.txt but epc is set to dict\n",
        "    Given the file \"epc.txt\" does not exist\n",
        "    And configuration key \"epc\" contains \"{key:val}\"\n",
        "    When we run notebook \"encrypted_epc.ipynb\"\n",
        "    Then we see \"epc is set\" in the output\n",
        "    And we see \"Encrypting epc\" in the output\n",
        "    And we see \"saved\" in the output\n",
        "    And the file \"epc.txt\" exists\n",
        "  \n",
        "  Scenario: no epc.txt and epc is an encrypted JSON string\n",
        "    Given the file \"epc.txt\" does not exist\n",
        "    And configuration key \"epc\" contains \"some encrypted string\"\n",
        "    When we run notebook \"encrypted_epc.ipynb\"\n",
        "    Then we see \"epc is set\" in the output\n",
        "    And we do not see \"Encrypting epc\" in the output\n",
        "    And we see \"saved\" in the output\n",
        "    And the file \"epc.txt\" exists\n",
        "\n",
        "  Scenario: no epc.txt and epc is invalid\n",
        "    Given the file \"epc.txt\" does not exist\n",
        "    And configuration key \"epc\" contains \"not a dict or encrypted str\"\n",
        "    When we run notebook \"encrypted_epc.ipynb\"\n",
        "    And we type \"testpassphrase\"\n",
        "    Then we see \"Try again\" in the output\n",
        "    And we see \"Decryption passphrase\" in the output\n",
        "\n",
        "  Scenario: epc.txt exists and epc is not set\n",
        "    Given the file \"epc.txt\" exists\n",
        "    When we run notebook \"encrypted_epc.ipynb\"\n",
        "    Then we see \"epc.txt exists\" in the output\n",
        "    And we see \"Decryption passphrase\" in the output\n",
        "\n",
        "  Scenario: epc.txt does not exist and epc is an encrypted JSON string\n",
        "    Given the file \"epc.txt\" does not exist\n",
        "    And configuration key \"epc\" contains \"encrypted string with passphrase testpassphrase\"\n",
        "    When we run notebook \"encrypted_epc.ipynb\"\n",
        "    And we type \"testpassphrase\"\n",
        "    Then we see \"Loading resources\" in the output\n"
      ],
      "metadata": {
        "id": "svrt81trWmr5"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# %%writefile jai/features/steps/{c.get('nb_name')}.py\n",
        "# @given(u'the file \"{filename}\" does not exist')\n",
        "# def step_impl(context, filename):\n",
        "#     os.system('rm '+filename)\n",
        "#     assert os.path.isfile(filename) == False\n",
        "\n",
        "# @given(u'configuration key \"{key}\" is not set')\n",
        "# def step_impl(context, key):\n",
        "#     assert c.get(key,None) is None\n",
        "\n",
        "# @when(u'we run notebook \"encrypted_epc.ipynb\"')\n",
        "# def step_impl(context):\n",
        "#     raise NotImplementedError(u'STEP: When we run notebook \"encrypted_epc.ipynb\"')\n",
        "\n",
        "\n",
        "# @then(u'we see \"Config not found\" in the output')\n",
        "# def step_impl(context):\n",
        "#     raise NotImplementedError(u'STEP: Then we see \"Config not found\" in the output')\n",
        "\n",
        "\n",
        "# @when(u'we type \"o\"')\n",
        "# def step_impl(context):\n",
        "#     raise NotImplementedError(u'STEP: When we type \"o\"')\n",
        "\n",
        "\n",
        "# @when(u'we type \"huggingface\"')\n",
        "# def step_impl(context):\n",
        "#     raise NotImplementedError(u'STEP: When we type \"huggingface\"')\n",
        "\n",
        "\n",
        "# @when(u'we type \"testtoken\"')\n",
        "# def step_impl(context):\n",
        "#     raise NotImplementedError(u'STEP: When we type \"testtoken\"')\n",
        "\n",
        "\n",
        "# @then(u'we see \"saved\" in the output')\n",
        "# def step_impl(context):\n",
        "#     raise NotImplementedError(u'STEP: Then we see \"saved\" in the output')\n",
        "\n",
        "\n",
        "# @then(u'the file \"epc.txt\" exists')\n",
        "# def step_impl(context):\n",
        "#     raise NotImplementedError(u'STEP: Then the file \"epc.txt\" exists')\n",
        "\n",
        "\n",
        "# @given(u'configuration key \"epc\" contains \"{key:val}\"')\n",
        "# def step_impl(context):\n",
        "#     raise NotImplementedError(u'STEP: Given configuration key \"epc\" contains \"{key:val}\"')\n",
        "\n",
        "\n",
        "# @then(u'we see \"epc is set\" in the output')\n",
        "# def step_impl(context):\n",
        "#     raise NotImplementedError(u'STEP: Then we see \"epc is set\" in the output')\n",
        "\n",
        "\n",
        "# @then(u'we see \"Encrypting epc\" in the output')\n",
        "# def step_impl(context):\n",
        "#     raise NotImplementedError(u'STEP: Then we see \"Encrypting epc\" in the output')\n",
        "\n",
        "\n",
        "# @given(u'configuration key \"epc\" contains \"some encrypted string\"')\n",
        "# def step_impl(context):\n",
        "#     raise NotImplementedError(u'STEP: Given configuration key \"epc\" contains \"some encrypted string\"')\n",
        "\n",
        "\n",
        "# @then(u'we do not see \"Encrypting epc\" in the output')\n",
        "# def step_impl(context):\n",
        "#     raise NotImplementedError(u'STEP: Then we do not see \"Encrypting epc\" in the output')\n",
        "\n",
        "\n",
        "# @given(u'configuration key \"epc\" contains \"not a dict or encrypted str\"')\n",
        "# def step_impl(context):\n",
        "#     raise NotImplementedError(u'STEP: Given configuration key \"epc\" contains \"not a dict or encrypted str\"')\n",
        "\n",
        "\n",
        "# @when(u'we type \"testpassphrase\"')\n",
        "# def step_impl(context):\n",
        "#     raise NotImplementedError(u'STEP: When we type \"testpassphrase\"')\n",
        "\n",
        "\n",
        "# @then(u'we see \"Try again\" in the output')\n",
        "# def step_impl(context):\n",
        "#     raise NotImplementedError(u'STEP: Then we see \"Try again\" in the output')\n",
        "\n",
        "\n",
        "# @then(u'we see \"Decryption passphrase\" in the output')\n",
        "# def step_impl(context):\n",
        "#     raise NotImplementedError(u'STEP: Then we see \"Decryption passphrase\" in the output')\n",
        "\n",
        "\n",
        "# @given(u'the file \"epc.txt\" exists')\n",
        "# def step_impl(context):\n",
        "#     raise NotImplementedError(u'STEP: Given the file \"epc.txt\" exists')\n",
        "\n",
        "\n",
        "# @then(u'we see \"epc.txt exists\" in the output')\n",
        "# def step_impl(context):\n",
        "#     raise NotImplementedError(u'STEP: Then we see \"epc.txt exists\" in the output')\n",
        "\n",
        "\n",
        "# @given(u'configuration key \"epc\" contains \"encrypted string with passphrase testpassphrase\"')\n",
        "# def step_impl(context):\n",
        "#     raise NotImplementedError(u'STEP: Given configuration key \"epc\" contains \"encrypted string with passphrase testpassphrase\"')\n",
        "\n",
        "\n",
        "# @then(u'we see \"Loading resources\" in the output')\n",
        "# def step_impl(context):\n",
        "#     raise NotImplementedError(u'STEP: Then we see \"Loading resources\" in the output')"
      ],
      "metadata": {
        "id": "KzZzPefwQxp4"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "PWEsII_i7Gm0"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# %pip install passlib\n",
        "# %pip install bcrypt\n",
        "# from passlib.hash import bcrypt\n",
        "# from getpass import getpass\n",
        "\n",
        "# passphrase = getpass()\n",
        "# hashed_passphrase = bcrypt.hash(passphrase)\n",
        "# print(hashed_passphrase)\n",
        "# print(bcrypt.verify(passphrase, hashed_passphrase))\n",
        "# # True\n",
        "# print(bcrypt.verify(\"not-the-passphrase\", hashed_passphrase))"
      ],
      "metadata": {
        "id": "KOgf7EpIZIZn"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "\n"
      ],
      "metadata": {
        "id": "0QSSwRRKsgLS"
      },
      "execution_count": null,
      "outputs": []
    }
  ]
}