{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Tutorial Nr. 1 -- Hello World"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setup"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1. Install the SDK\n",
    "\n",
    "```bash\n",
    "pip install ratio1\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2. Create `.env` file"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Copy the `tutorials/.example_env` file to your project directory and rename it to `.env`.\n",
    "\n",
    "Fill in the empty variables with appropriate values."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3. Create new / Use test private key\n",
    "\n",
    "**Disclaimer: You should never publish sensitive information such as private keys.**\n",
    "\n",
    "To experiment on our test net, you can use the provided private key to communicate with the 3 nodes in the test network.\n",
    "\n",
    "#### Create new private key\n",
    "\n",
    "When first connecting to our network, the sdk will search in the current working directory for an existing private key. If not found, the SDK will create one at `$(cwd)/_local_cache/_data/_pk_sdk.pem`.\n",
    "\n",
    "#### Using an existing private key\n",
    "\n",
    "To use an existing private key, create in the working directory the directory tree `_local_cache/_data/` and add the `_pk_sdk.pem` file there.\n",
    "\n",
    "To use our provided key. copy it from `tutorials/_example_pk_sdk.pem` to `local_cache/_data/` and change its name to `_pk_sdk.pem`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Local Execution"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We want to find all $168$ prime numbers in the interval $1$-$1000$. For this we can run the following code on our local machine.\n",
    "\n",
    "This code has segments running on multiple threads using a ThreadPool."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "from concurrent.futures import ThreadPoolExecutor\n",
    "\n",
    "\n",
    "def local_brute_force_prime_number_generator():\n",
    "  def is_prime(n):\n",
    "    if n <= 1:\n",
    "      return False\n",
    "    for i in range(2, int(np.sqrt(n)) + 1):\n",
    "      if n % i == 0:\n",
    "        return False\n",
    "    return True\n",
    "\n",
    "  random_numbers = np.random.randint(1, 1000, 20)\n",
    "\n",
    "  thread_pool = ThreadPoolExecutor(max_workers=4)\n",
    "  are_primes = list(thread_pool.map(is_prime, random_numbers))\n",
    "\n",
    "  prime_numbers = []\n",
    "  for i in range(len(random_numbers)):\n",
    "    if are_primes[i]:\n",
    "      prime_numbers.append(random_numbers[i])\n",
    "\n",
    "  return prime_numbers\n",
    "\n",
    "\n",
    "if __name__ == \"__main__\":\n",
    "  found_so_far = []\n",
    "\n",
    "  print_step = 0\n",
    "\n",
    "  while len(found_so_far) < 168:\n",
    "    # compute a batch of prime numbers\n",
    "    prime_numbers = local_brute_force_prime_number_generator()\n",
    "\n",
    "    # keep only the new prime numbers\n",
    "    for prime_number in prime_numbers:\n",
    "      if prime_number not in found_so_far:\n",
    "        found_so_far.append(prime_number)\n",
    "    # end for\n",
    "\n",
    "    # show progress\n",
    "    if print_step % 50 == 0:\n",
    "      print(\"Found so far: {}:  {}\\n\".format(len(found_so_far), sorted(found_so_far)))\n",
    "\n",
    "    print_step += 1\n",
    "  # end while\n",
    "\n",
    "  # show final result\n",
    "  print(\"Found so far: {}:  {}\\n\".format(len(found_so_far), sorted(found_so_far)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see that we have a `local_brute_force_prime_number_generator` method which will generate a random sample of 20 numbers that will be checked if they are prime or not.\n",
    "\n",
    "The rest of the code handles how the numbers generated with this method are kept.\n",
    "Because we want to find 168 unique numbers, we append to the list of found primes only the numbers that are not present yet.\n",
    "\n",
    "At the end, we want to show a list of all the numbers found."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Remote Execution "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For this example we would like to use multiple edge nodes to find the prime numbers faster.\n",
    "\n",
    "To execute this code on our network, a series of changes must be made to the `local_brute_force_prime_number_generator` method.\n",
    "These changes are the only ones a developer has to do to deploy his own custom code on the network.\n",
    "\n",
    "For this, we will create a new method, `remote_brute_force_prime_number_generator`, which will use the exposed edge node API methods."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from ratio1 import CustomPluginTemplate\n",
    "\n",
    "# through the `plugin` object we get access to the edge node API\n",
    "# the CustomPluginTemplate class acts as a documentation for all the available methods and attributes\n",
    "# since we do not allow imports in the custom code due to security reasons, the `plugin` object \n",
    "#   exposes common modules to the user\n",
    "def remote_brute_force_prime_number_generator(plugin: CustomPluginTemplate):\n",
    "  def is_prime(n):\n",
    "    if n <= 1:\n",
    "      return False\n",
    "    # we use the `plugin.np` instead of the `np` module\n",
    "    for i in range(2, int(plugin.np.sqrt(n)) + 1):\n",
    "      if n % i == 0:\n",
    "        return False\n",
    "    return True\n",
    "  \n",
    "  # we use the `plugin.np` instead of the `np` module\n",
    "  random_numbers = plugin.np.random.randint(1, 1000, 20)\n",
    "\n",
    "  # we use the `plugin.threadapi_map` instead of the `ThreadPoolExecutor.map`\n",
    "  are_primes = plugin.threadapi_map(is_prime, random_numbers, n_threads=4)\n",
    "\n",
    "  prime_numbers = []\n",
    "  for i in range(len(random_numbers)):\n",
    "    if are_primes[i]:\n",
    "      prime_numbers.append(random_numbers[i])\n",
    "\n",
    "  return prime_numbers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This are all the changes we have to do to deploy this code in the network.\n",
    "\n",
    "Now lets connect to the network and see what nodes are online.\n",
    "We will use the `on_heartbeat` callback to print the nodes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from ratio1 import Session\n",
    "from time import sleep\n",
    "\n",
    "def on_heartbeat(session: Session, node_addr: str, heartbeat: dict):\n",
    "  # the `.P` method is used to print messages in the console and store them in the log file\n",
    "  session.P(\"{} is online\".format(node_addr))\n",
    "  return\n",
    "\n",
    "\n",
    "if __name__ == '__main__':\n",
    "  # create a session\n",
    "  # the network credentials are read from the .env file automatically\n",
    "  session = Session(\n",
    "      on_heartbeat=on_heartbeat\n",
    "  )\n",
    "\n",
    "  # run the program for 15 seconds to show all the nodes that are online\n",
    "  sleep(15)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next we will select an online node. This node will be our entrypoint in the network.\n",
    "\n",
    "The available nodes in our test net are:\n",
    "\n",
    "```\n",
    "0xai_A8SY7lEqBtf5XaGyB6ipdk5C30vSf3HK4xELp3iplwLe ratio1-1\n",
    "0xai_Amfnbt3N-qg2-qGtywZIPQBTVlAnoADVRmSAsdDhlQ-6 ratio1-2\n",
    "0xai_ApltAljEgWk3g8x2QcSa0sS3hT1P4dyCchd04zFSMy5e ratio1-3\n",
    "```\n",
    "\n",
    "We will send a task to this node. Since we want to distribute the task of finding prime numbers to multiple nodes, this selected node will handle distribution of tasks and collection of the results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "node = \"0xai_A8SY7lEqBtf5XaGyB6ipdk5C30vSf3HK4xELp3iplwLe\" # ratio1-1\n",
    "\n",
    "# we usually wait for the node to be online before sending the task\n",
    "# but in this case we are sure that the node is online because we \n",
    "# have received heartbeats from it during the sleep period\n",
    "\n",
    "# session.wait_for_node(node)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our selected node will periodically output partial results with the prime numbers found so far by the worker nodes. We want to consume these results.\n",
    "\n",
    "Thus, we need to implement a callback method that will handle this. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from ratio1 import Pipeline\n",
    "\n",
    "# a flag used to close the session when the task is finished\n",
    "finished = False\n",
    "\n",
    "def locally_process_partial_results(pipeline: Pipeline, full_payload):\n",
    "  global finished\n",
    "  found_so_far = full_payload.get(\"DATA\")\n",
    "\n",
    "  if found_so_far:\n",
    "    pipeline.P(\"Found so far: {}:  {}\\n\\n\".format(len(found_so_far), sorted(found_so_far)))\n",
    "\n",
    "  progress = full_payload.get(\"PROGRESS\")\n",
    "  if progress == 100:\n",
    "    pipeline.P(\"FINISHED\\n\\n\")\n",
    "    finished = True\n",
    "\n",
    "  return"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we are ready to deploy our job to the network."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from ratio1 import DistributedCustomCodePresets as Presets\n",
    "\n",
    "_, _ = session.create_chain_dist_custom_job(\n",
    "    # this is the main node, our entrypoint\n",
    "    node=node,\n",
    "    \n",
    "    # this function is executed on the main node\n",
    "    # this handles what we want to do with primes found by a worker node after an iteration\n",
    "    # we want to store only the unique prime numbers\n",
    "    # we cam either write a custom code to pass here or we can use a preset\n",
    "    main_node_process_real_time_collected_data=Presets.PROCESS_REAL_TIME_COLLECTED_DATA__KEEP_UNIQUES_IN_AGGREGATED_COLLECTED_DATA,\n",
    "    \n",
    "    # this function is executed on the main node\n",
    "    # this handles the finish condition of our distributed job\n",
    "    # we want to finish when we have found 168 prime numbers\n",
    "    # so more than 167 prime numbers\n",
    "    # we cam either write a custom code to pass here or we can use a preset\n",
    "    main_node_finish_condition=Presets.FINISH_CONDITION___AGGREGATED_DATA_MORE_THAN_X,\n",
    "    main_node_finish_condition_kwargs={\n",
    "        \"X\": 167\n",
    "    },\n",
    "    \n",
    "    # this function is executed on the main node\n",
    "    # this handles the final processing of the results\n",
    "    # this function prepares data for the final result of the distributed job\n",
    "    # we want to aggregate all the prime numbers found by the worker nodes in a single list\n",
    "    # we cam either write a custom code to pass here or we can use a preset\n",
    "    main_node_aggregate_collected_data=Presets.AGGREGATE_COLLECTED_DATA___AGGREGATE_COLLECTED_DATA,\n",
    "\n",
    "    # how many worker nodes we want to use for this task\n",
    "    nr_remote_worker_nodes=2,\n",
    "    \n",
    "    # this is the function that will be executed on the worker nodes\n",
    "    # this function generates prime numbers using brute force\n",
    "    # we simply pass the function reference\n",
    "    worker_node_code=remote_brute_force_prime_number_generator,\n",
    "\n",
    "    # this is the function that will be executed on the client\n",
    "    # this is the callback function that processes the partial results\n",
    "    # in our case we want to print the partial results\n",
    "    on_data=locally_process_partial_results,\n",
    "    \n",
    "    # we want to deploy the job immediately\n",
    "    deploy=True\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Last but not least, we want to close the session when the distributed job finished."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# we wait until the finished flag is set to True\n",
    "# we want to release the resources allocated on the selected node when the job is finished\n",
    "session.run(wait=lambda: not finished, close_pipelines=True)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "e2_env",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
