{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "application/vnd.databricks.v1+cell": {
          "cellMetadata": {
            "byteLimit": 2048000,
            "rowLimit": 10000
          },
          "inputWidgets": {},
          "nuid": "51adebf4-7aba-4a03-b50b-36c2ec974122",
          "showTitle": false,
          "title": ""
        },
        "id": "D6cn8o8xWMuC"
      },
      "source": [
        "![JohnSnowLabs](https://nlp.johnsnowlabs.com/assets/images/logo.png)\n",
        "\n",
        "# Automatic AWS EMR Cluster Creation with John Snow Labs libraries pre-installed\n",
        "For more details see https://nlp.johnsnowlabs.com/docs/en/jsl/install_advanced#via-json-secrets-file"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "application/vnd.databricks.v1+cell": {
          "cellMetadata": {},
          "inputWidgets": {},
          "nuid": "d93776cb-0b0a-4430-8d48-5cbb5a5c540a",
          "showTitle": false,
          "title": ""
        },
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "id": "eRI1_vt7WMuF",
        "outputId": "01d96386-2adc-4843-f0a2-f415a84b60d3"
      },
      "outputs": [],
      "source": [
        "%pip install -U johnsnowlabs"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bbF0E4tqWjh2"
      },
      "source": [
        "### Setup AWS credentials in environment variables or create a boto3 session\n",
        "\n",
        "Refer these for understanding boto3 sessions.\n",
        "1. https://boto3.amazonaws.com/v1/documentation/api/latest/guide/session.html\n",
        "2. https://boto3.amazonaws.com/v1/documentation/api/latest/reference/core/session.html"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 5,
      "metadata": {
        "id": "VVOepP7CWMuG"
      },
      "outputs": [],
      "source": [
        "# Ask your AWS admin for the AWS access keys if you don't have them\n",
        "import boto3\n",
        "\n",
        "# Creating a boto session. You can also use a profile name instead of the access key and secret key\n",
        "\n",
        "boto_session = boto3.Session(\n",
        "    aws_access_key_id=\"Your AWS Access Key\",\n",
        "    aws_secret_access_key=\"Your AWS Secret Key\",\n",
        "    # aws_session_token=\"Your AWS Session Tokenk\",\n",
        "    region_name=\"AWS Region to create the EMR cluster in\",\n",
        "    #profile_name=\"Your AWS Profile Name\"\n",
        ")\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 3,
      "metadata": {
        "application/vnd.databricks.v1+cell": {
          "cellMetadata": {
            "byteLimit": 2048000,
            "rowLimit": 10000
          },
          "inputWidgets": {},
          "nuid": "7015f047-b770-44d5-a50f-5dd7d822078d",
          "showTitle": false,
          "title": ""
        },
        "id": "l_Gp2WoNWMuH"
      },
      "outputs": [],
      "source": [
        "from johnsnowlabs import nlp\n",
        "import os\n",
        "\n",
        "cluster_license_string = \"\"\" \n",
        "Your John Snow Labs Floating JSON LIcense\n",
        "\"\"\"\n",
        "if os.path.exists('license.json'): os.remove('license.json')\n",
        "with open('license.json', 'w') as file:\n",
        "    file.write(cluster_license_string)\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### IAM Roles for EMR Cluster\n",
        "By default, we use AWS EMR default roles to create an EMR cluster. Please make sure those IAM Roles exists in your AWS account. If you don't have them in your account, create them with this command.\n",
        "\n",
        "`aws emr create-default-roles`\n",
        "\n",
        "If you want to use custom IAM Roles, it can be configured with these parameters `job_flow_role` and `service_role`. Refer [this](https://nlp.johnsnowlabs.com/docs/en/jsl/aws-emr-utils#aws-specific-parameters) for more parameters"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "application/vnd.databricks.v1+cell": {
          "cellMetadata": {
            "byteLimit": 2048000,
            "rowLimit": 10000
          },
          "inputWidgets": {},
          "nuid": "a8219a9d-8bff-4596-85ec-6477ac599ad8",
          "showTitle": false,
          "title": ""
        },
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 507
        },
        "id": "IySo3GM0WMuH",
        "outputId": "faf6f07b-b110-49a2-f14b-c6ac8122efbf"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "👷 Trying to install compatible secrets. Use nlp.settings.enforce_versions=False if you want to install outdated secrets.\n",
            "🚨 Failure Trying to read license Usage of invalid/expired license.\n",
            " Trying to use license from John Snow Labs home folder if it exists\n",
            "✅ Created EMR cluster with id=j-1B5RFL6PSZXTX\n",
            "Cluster-Id=j-1B5RFL6PSZXTX not ready, status=STARTING\n",
            "Cluster-Id=j-1B5RFL6PSZXTX not ready, status=STARTING\n",
            "Cluster-Id=j-1B5RFL6PSZXTX not ready, status=STARTING\n",
            "Cluster-Id=j-1B5RFL6PSZXTX not ready, status=STARTING\n",
            "Cluster-Id=j-1B5RFL6PSZXTX not ready, status=STARTING\n",
            "Cluster-Id=j-1B5RFL6PSZXTX not ready, status=STARTING\n",
            "Cluster-Id=j-1B5RFL6PSZXTX not ready, status=BOOTSTRAPPING\n",
            "Cluster-Id=j-1B5RFL6PSZXTX not ready, status=BOOTSTRAPPING\n",
            "Cluster-Id=j-1B5RFL6PSZXTX not ready, status=BOOTSTRAPPING\n",
            "Cluster-Id=j-1B5RFL6PSZXTX not ready, status=BOOTSTRAPPING\n",
            "Cluster-Id=j-1B5RFL6PSZXTX not ready, status=BOOTSTRAPPING\n",
            "Cluster-Id=j-1B5RFL6PSZXTX not ready, status=BOOTSTRAPPING\n",
            "Cluster-Id=j-1B5RFL6PSZXTX not ready, status=BOOTSTRAPPING\n",
            "Cluster-Id=j-1B5RFL6PSZXTX not ready, status=BOOTSTRAPPING\n",
            "Cluster-Id=j-1B5RFL6PSZXTX not ready, status=BOOTSTRAPPING\n",
            "Cluster-Id=j-1B5RFL6PSZXTX not ready, status=BOOTSTRAPPING\n",
            "Cluster-Id=j-1B5RFL6PSZXTX not ready, status=BOOTSTRAPPING\n",
            "Cluster-Id=j-1B5RFL6PSZXTX not ready, status=BOOTSTRAPPING\n",
            "Cluster-Id=j-1B5RFL6PSZXTX not ready, status=BOOTSTRAPPING\n",
            "Cluster-Id=j-1B5RFL6PSZXTX not ready, status=BOOTSTRAPPING\n",
            "Cluster-Id=j-1B5RFL6PSZXTX not ready, status=RUNNING\n",
            "👌 Cluster-Id j-1B5RFL6PSZXTX is ready!\n"
          ]
        },
        {
          "data": {
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            },
            "text/plain": [
              "'j-1B5RFL6PSZXTX'"
            ]
          },
          "execution_count": 6,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "nlp.install_to_emr(boto_session=boto_session, json_license_path='license.json')\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "application/vnd.databricks.v1+cell": {
          "cellMetadata": {},
          "inputWidgets": {},
          "nuid": "e9460868-f659-498b-9523-ceb9d2505234",
          "showTitle": false,
          "title": ""
        },
        "id": "BiuvYaquWMuI"
      },
      "outputs": [],
      "source": []
    }
  ],
  "metadata": {
    "application/vnd.databricks.v1+notebook": {
      "dashboards": [],
      "language": "python",
      "notebookMetadata": {
        "pythonIndentUnit": 4
      },
      "notebookName": "create-cluster_PUBLIC",
      "widgets": {}
    },
    "colab": {
      "provenance": []
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}
