{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {},
     "inputWidgets": {},
     "nuid": "25c10a20-71c7-4116-9f2d-f96ae4c8837d",
     "showTitle": false,
     "title": ""
    }
   },
   "source": [
    "\n",
    "![JohnSnowLabs](https://nlp.johnsnowlabs.com/assets/images/logo.png)\"\n",
    "# John Snow Labs models from Databricks Model Marketplace tutorial\n",
    "This notebook showcases how to import, deploy and query John Snow Labs models from the Databricks Marketplace. \n",
    "\n",
    "You will learn about \n",
    "- `make_ui()` and required credentials. You need a Johnsnowlabs PAYG licese, Databricks Access Token and Databricks Host\n",
    "- `deploy()` for deploying endpoints \n",
    "- `query` for querying endpoints\n",
    "\n",
    "\n",
    "## PAYG License \n",
    "You need a pay as you go (PAYG) license for using John Snow Labs models, follow these instructions to get one : \n",
    "1. Access [my.JohnSnowLabs.com](my.JohnSnowLabs.com) and log in to your account. If you don't have an account, create one. \n",
    "2. Go to the Get License page. \n",
    "3. Switch to the PAYG Subscription tab and provide your credit card details. \n",
    "4. Carefully review the End User License Agreement and the Terms and Conditions documents. If you agree, click on the Create Subscription button. \n",
    "5. Once the process is complete, you will find your PAY-As-You-GO license listed on the My Subscriptions page. \n",
    "6. Visit the My Subscriptions page and copy the PAYG license key by clicking on the copy icon in the License Key column.\n",
    "7. Go to your Databricks notebook and paste your JSL-license into the JSL-License field in the top of the notebook. You are now ready to go!\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {},
     "inputWidgets": {},
     "nuid": "1ddea1cc-7807-4e95-99f8-26b0ec02361e",
     "showTitle": false,
     "title": ""
    }
   },
   "outputs": [],
   "source": [
    "%pip install -U johnsnowlabs_for_databricks\n",
    "dbutils.library.restartPython()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {},
     "inputWidgets": {},
     "nuid": "fb2c0ab2-8559-48e6-ba64-1bcf4a01030d",
     "showTitle": false,
     "title": ""
    }
   },
   "source": [
    "# Deploy an Endpoint\n",
    "\n",
    "You must fill the following values in the top of the notebook to deploy an endpoint \n",
    "- **Databricks access token**: Access token which must have access to endpoint features. \n",
    "- **Databricks host**: URL of your Databricks host, i.e. https://abcd.cloud.databricks.com\n",
    "- **JSL-License**: Your JSL-License. Get one from [https://my.johnsnowlabs.com](https://my.johnsnowlabs.com)\n",
    "- **Model to deploy**: Johnsnowlabs NLU reference to the model, can be found in every model listing. \n",
    "- **Hardware Target**: Controls wether GPU or CPU optimized builds are used in endpoint\n",
    "\n",
    "\n",
    "Once filled, run `deploy()` to deploy an endpoint with an auto-generated name.     \n",
    "Use `deploy(my_endpoint_name)` to define the name of the deployed endpoint instead of auto-generated. \n",
    "\n",
    "`deploy` returns the name of the deployed model \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "2bc70f19-9d62-4547-9565-34c15f64a4a8",
     "showTitle": false,
     "title": ""
    }
   },
   "outputs": [],
   "source": [
    "from johnsnowlabs import nlp \n",
    "from johnsnowlabs.auto_install.databricks.marketplace import  render_ui , deploy\n",
    "# Render the Widget UI\n",
    "render_ui()\n",
    "\n",
    "# Define some medical text we can use for queries\n",
    "general_text = \"John Snow is a medical doctor from England. Peter likes America\"\n",
    "cancer_text = \"\"\"with his breathing while feeding (but negative for any perioral cyanosis or retractions). One day ago, mom also noticed a tactile temperature and gave the patient Tylenol. Baby-girl also has had some decreased p.o. intake. His normal breast-feeding is down from 20 minutes q.2h. to 5 to 10 minutes s\"\"\"\n",
    "\n",
    "ade_text = \"\"\"So glad I am off effexor, so sad it ruined my teeth. tip Please be carefull taking antideppresiva and read about it 1st\"\"\"\n",
    "\n",
    "body_re_text = \"\"\"MRI demonstrated infarction in the upper brain stem , left cerebellum and  right basil ganglia\"\"\"\n",
    "radiology_text = \"\"\"INTERPRETATION: There has been interval development of a moderate left-sided pneumothorax with near complete collapse of the left upper lobe. The lower lobe appears aerated. There is stable, diffuse, bilateral interstitial thickening with no definite acute air space consolidation. The heart and pulmonary vascularity are within normal limits. Left-sided port is seen with Groshong tip at the SVC/RA junction. No evidence for acute fracture, malalignment, or dislocation.\"\"\"\n",
    "\n",
    "\n",
    "long_text = \"\"\"\n",
    "Patient with hypertension, syncope, and spinal stenosis - for recheck.\n",
    "(Medical Transcription Sample Report)\n",
    "SUBJECTIVE:\n",
    "The patient is a 78-year-old female who returns for recheck. She has hypertension. She denies difficulty with chest pain, palpations, orthopnea, nocturnal dyspnea, or edema.\n",
    "PAST MEDICAL HISTORY / SURGERY / HOSPITALIZATIONS:\n",
    "Reviewed and unchanged from the dictation on 12/03/2003.\n",
    "MEDICATIONS:\n",
    "Atenolol 50 mg daily, Premarin 0.625 mg daily, calcium with vitamin D two to three pills daily, multivitamin daily, aspirin as needed, and TriViFlor 25 mg two pills daily. She also has Elocon cream 0.1% and Synalar cream 0.01% that she uses as needed for rash\n",
    "\"\"\"\n",
    "all_data = [general_text,cancer_text,ade_text,body_re_text, radiology_text, long_text ]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "8e0d098a-d604-4fa6-a282-8c0295a346e7",
     "showTitle": false,
     "title": ""
    }
   },
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Writing license to scope JSL_SCOPE\ncreating model john_snow_labs_clinical_de_identification_mask.johnsnowlabs_folder.en_de_identify_clinical_pipeline_cpu\ncreating endpoint my_deid_endpoint\nCreating new serving endpoint: my_deid_endpoint\nDeployment starting, this may take 10 to 20 minutes...\n"
     ]
    },
    {
     "output_type": "display_data",
     "data": {
      "text/html": [
       "Created the <a href=\"/#mlflow/endpoints/my_deid_endpoint\" target=\"_blank\">my_deid_endpoint</a> serving endpoint"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "'my_deid_endpoint'"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "endpoint_name = deploy('my_deid_endpoint')\n",
    "endpoint_name"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {},
     "inputWidgets": {},
     "nuid": "950981bd-44b5-4355-a1cc-2854424a273f",
     "showTitle": false,
     "title": ""
    }
   },
   "source": [
    "# Query deployed endpoint\n",
    "\n",
    "Once deployed we can use `query_endpoint` to send data to the endpoint and get the models predictions back as pandas DF "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "4d286c95-598a-4e74-8d26-d2a0fcf75228",
     "showTitle": false,
     "title": ""
    }
   },
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>index</th>\n",
       "      <th>finished_masked</th>\n",
       "      <th>finished_masked_fixed_length_chars</th>\n",
       "      <th>finished_masked_with_chars</th>\n",
       "      <th>finished_ner_chunk</th>\n",
       "      <th>finished_obfuscated</th>\n",
       "      <th>finished_sentence</th>\n",
       "      <th>text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>[&lt;PATIENT&gt; is a medical doctor from &lt;COUNTRY&gt;....</td>\n",
       "      <td>[**** is a medical doctor from ****.,  ****]</td>\n",
       "      <td>[[*******] is a medical doctor from [*****]., ...</td>\n",
       "      <td>[John Snow, England, Peter likes America]</td>\n",
       "      <td>[Gustavus Bryant is a medical doctor from Guad...</td>\n",
       "      <td>[John Snow is a medical doctor from England., ...</td>\n",
       "      <td>John Snow is a medical doctor from England. Pe...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>[with his breathing while feeding (but negativ...</td>\n",
       "      <td>[with his breathing while feeding (but negativ...</td>\n",
       "      <td>[with his breathing while feeding (but negativ...</td>\n",
       "      <td>[]</td>\n",
       "      <td>[with his breathing while feeding (but negativ...</td>\n",
       "      <td>[with his breathing while feeding (but negativ...</td>\n",
       "      <td>with his breathing while feeding (but negative...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2</td>\n",
       "      <td>[So glad I am off effexor, so sad it ruined my...</td>\n",
       "      <td>[So glad I am off effexor, so sad it ruined my...</td>\n",
       "      <td>[So glad I am off effexor, so sad it ruined my...</td>\n",
       "      <td>[]</td>\n",
       "      <td>[So glad I am off effexor, so sad it ruined my...</td>\n",
       "      <td>[So glad I am off effexor, so sad it ruined my...</td>\n",
       "      <td>So glad I am off effexor, so sad it ruined my ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>3</td>\n",
       "      <td>[MRI demonstrated infarction in the upper brai...</td>\n",
       "      <td>[MRI demonstrated infarction in the upper brai...</td>\n",
       "      <td>[MRI demonstrated infarction in the upper brai...</td>\n",
       "      <td>[]</td>\n",
       "      <td>[MRI demonstrated infarction in the upper brai...</td>\n",
       "      <td>[MRI demonstrated infarction in the upper brai...</td>\n",
       "      <td>MRI demonstrated infarction in the upper brain...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>4</td>\n",
       "      <td>[INTERPRETATION: There has been interval devel...</td>\n",
       "      <td>[INTERPRETATION: There has been interval devel...</td>\n",
       "      <td>[INTERPRETATION: There has been interval devel...</td>\n",
       "      <td>[]</td>\n",
       "      <td>[INTERPRETATION: There has been interval devel...</td>\n",
       "      <td>[INTERPRETATION: There has been interval devel...</td>\n",
       "      <td>INTERPRETATION: There has been interval develo...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>5</td>\n",
       "      <td>[Patient with hypertension, syncope, and spina...</td>\n",
       "      <td>[Patient with hypertension, syncope, and spina...</td>\n",
       "      <td>[Patient with hypertension, syncope, and spina...</td>\n",
       "      <td>[78, 12/03/2003]</td>\n",
       "      <td>[Patient with hypertension, syncope, and spina...</td>\n",
       "      <td>[Patient with hypertension, syncope, and spina...</td>\n",
       "      <td>\\nPatient with hypertension, syncope, and spin...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   index  ...                                               text\n",
       "0      0  ...  John Snow is a medical doctor from England. Pe...\n",
       "1      1  ...  with his breathing while feeding (but negative...\n",
       "2      2  ...  So glad I am off effexor, so sad it ruined my ...\n",
       "3      3  ...  MRI demonstrated infarction in the upper brain...\n",
       "4      4  ...  INTERPRETATION: There has been interval develo...\n",
       "5      5  ...  \\nPatient with hypertension, syncope, and spin...\n",
       "\n",
       "[6 rows x 8 columns]"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "nlp.query_endpoint(endpoint_name, all_data)"
   ]
  }
 ],
 "metadata": {
  "application/vnd.databricks.v1+notebook": {
   "dashboards": [],
   "language": "python",
   "notebookMetadata": {
    "mostRecentlyExecutedCommandWithImplicitDF": {
     "commandId": 2549815365552276,
     "dataframes": [
      "_sqldf"
     ]
    },
    "pythonIndentUnit": 4,
    "widgetLayout": []
   },
   "notebookName": "Databricks Model Marketplace",
   "widgets": {
    "Databricks access token": {
     "currentValue": "",
     "nuid": "0752ddaa-571b-4076-898a-08a2710bbbae",
     "widgetInfo": {
      "widgetType": "text",
      "defaultValue": "",
      "label": null,
      "name": "Databricks access token",
      "options": {
       "widgetType": "text",
       "autoCreated": null,
       "validationRegex": null
      }
     }
    },
    "Databricks host": {
     "currentValue": "",
     "nuid": "e8e5fcd6-6308-4de8-a6a9-093cf0f3125f",
     "widgetInfo": {
      "widgetType": "text",
      "defaultValue": "",
      "label": null,
      "name": "Databricks host",
      "options": {
       "widgetType": "text",
       "autoCreated": null,
       "validationRegex": null
      }
     }
    },
    "JSL-License": {
     "currentValue": "",
     "nuid": "f9e40950-8046-4bf0-b41a-6b545b3d3995",
     "widgetInfo": {
      "widgetType": "text",
      "defaultValue": "",
      "label": "",
      "name": "JSL-License",
      "options": {
       "widgetType": "text",
       "autoCreated": false,
       "validationRegex": null
      }
     }
    },
    "The model": {
     "currentValue": "1. Clinical De-identification (Mask)",
     "nuid": "2e5dc5b2-4cd2-4e11-aa29-3d92159c9c1a",
     "widgetInfo": {
      "widgetType": "dropdown",
      "defaultValue": "1. Clinical De-identification (Mask)",
      "label": null,
      "name": "The model",
      "options": {
       "widgetType": "dropdown",
       "autoCreated": null,
       "choices": [
        "1. Clinical De-identification (Mask)",
        "2. Clinical De-identification (Obfuscate)",
        "3. Clinical De-identification for German (Mask)",
        "4. Clinical De-identification for German (Obfuscate)",
        "5. Clinical De-identification for Spanish (Mask)",
        "6. Clinical De-identification for Spanish (Obfuscate)",
        "7. Clinical De-identification forItalian (Mask)",
        "8. Clinical De-identification for Italian (Obfuscate)",
        "9. Clinical De-identification for French (Mask)",
        "10. Clinical De-identification for French (Obfuscate)",
        "11. Clinical De-identification for Portuguese (Mask)",
        "12. Clinical De-identification for Portuguese (Obfuscate)",
        "13. Clinical De-identification for Arabic (Mask)",
        "14. Clinical De-identification for Arabic (Obfuscate)",
        "15. Clinical De-identification for Romanian (Mask)",
        "16. Clinical De-identification for Romanian (Obfuscate)",
        "17. Summarize Radiology Reports",
        "18. Guideline Summarizer for Asthma & Breast Cancer",
        "19. Summarize Clinical Notes in Laymen Terms ",
        "20. Summarize Clinical Question Notes",
        "21. Summarize PubMed Research",
        "22. Clinical Text Summarization",
        "23. Medical Question Answering (Open-Book on Clinical Notes)",
        "24. Extract clinical events and find temporal relations",
        "25. Extract findings in radiology reports",
        "26. Extract Adverse Drug Events (ADE)",
        "27. Extract oncological entities and relations",
        "28. Extract patient information from patient narratives",
        "29. Detect Healthcare Professional Consult documents",
        "30. Detect Drug Side Effect Narratives",
        "31. Extract Social Determinants of Health",
        "32. Extract neurologic deficits related to Stroke Scale (NIHSS)",
        "33. Extract clinical risk factors",
        "34. Detect Cancer Genetics",
        "35. Detect Pathogen, Medical Condition and Medicine",
        "36. Extract Living Species",
        "37. Extract Anatomical Structures",
        "38. Extract  Drugs and Substances and their UMLS Codes",
        "39. Extract Drugs and the correspondingt RXNORM codes",
        "40. Extract Clinical Findings and the corresponding ICD10CM codes",
        "41. Medication Coder",
        "42. Extract Disease and Syndrom entities and their UMLS Codes",
        "43. ICD-10-CM to UMLS Code Mapper",
        "44. Extract Clinical Findings and the corresponding ICD 9 codes",
        "45. Extract Clinical Findings and their UMLS Codes",
        "46. Extract Drugs and their UMLS Codes",
        "47. Extract phenotypic abnormalities and the coresponding HPO codes ",
        "48. Extract procedures and measurements and their corresponding CPT codes",
        "49. ICD-10-CM to SNOMED Code Mapper",
        "50. ICD-10 to ICD-9 Code Converter",
        "51. MeSH to UMLS Code Mapper",
        "52. Resolve Oncology terminology with ICD-O Taxonomy",
        "53. RxNorm to MeSH Code Mapper",
        "54. RxNorm to NDC Code Converter",
        "55. RxNorm to UMLS Code Mapper",
        "56. SNOMED Clinical Health Information Coder",
        "57. SNOMED Codes to ICDO Codes Mapper",
        "58. SNOMED to ICD-10-CM Code Mapper",
        "59. SNOMED to UMLS Code Mapper",
        "60. Extract Laboratory Observations and the corresponding LOINC codes"
       ]
      }
     }
    },
    "hardware_target": {
     "currentValue": "CPU",
     "nuid": "86f10dfd-6480-487f-ae37-cfd722fc0e60",
     "widgetInfo": {
      "widgetType": "dropdown",
      "defaultValue": "CPU",
      "label": null,
      "name": "hardware_target",
      "options": {
       "widgetType": "dropdown",
       "autoCreated": null,
       "choices": [
        "CPU",
        "GPU"
       ]
      }
     }
    }
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}
