{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Loading Documents\n",
    "When documents are uploaded into the platform, they are uploaded as a special Class type `BlobInput`. \n",
    "\n",
    "To test Abacus functionality locally in your notebook, transform them to `BlobInput` class"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Here, we upload training file from the current location of the notebook\n",
    "# You can add files to Jupyter Notebook by drag and drop\n",
    "from abacusai.client import BlobInput\n",
    "import abacusai\n",
    "client = abacusai.ApiClient('YOUR_API_KEY')\n",
    "document = BlobInput.from_local_file(\"YOUR_DOCUMENT.pdf/word/etc\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Extract text from a local document\n",
    "You can extract text using two methods:\n",
    "1. Embedded Text Extraction --> That means extracting the text that is already in the document. It's fast and works well for modern documents.\n",
    "2. OCR ---> Extracts the text as seen from end user. Works very well for scanned documents, when there are tables involved, etc.\n",
    "\n",
    "First, let's take a look at **Embedded Text Extraction**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "UNITED STATES\n",
      "SECURITIES AND EXCHANGE COMMISSION\n",
      "Washington, D.C. 20549\n",
      "____________________________\n",
      "\n",
      "UNITED STATES\n",
      "SECURITIES AND EXCHANGE COMMISSION\n",
      "Washington, D.C. 20549\n",
      "____________________________\n"
     ]
    }
   ],
   "source": [
    "extracted_doc_data = client.extract_document_data(document.contents)\n",
    "\n",
    "# print first 100 chracters of page 0\n",
    "print(extracted_doc_data.pages[0][0:100])\n",
    "print()\n",
    "# print first 100 chracters of all embedded text\n",
    "print(extracted_doc_data.embedded_text[0:100]) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, let's extract data using **OCR**. Please note that there are multiple `ocr_mode` values and multiple settings. Refer to the official Python SDK API for all of them."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "UNITED STATES\n",
      "SECURITIES AND EXCHANGE COMMISSION\n",
      "Washington, D.C. 20549\n",
      "(Mark One)\n",
      "ANNUAL REPORT PUR\n"
     ]
    }
   ],
   "source": [
    "extracted_doc_data = client.extract_document_data(document.contents, \n",
    "                                                  document_processing_config={'extract_bounding_boxes': True,'ocr_mode': 'DEFAULT', 'use_full_ocr':True})\n",
    "\n",
    "# Print first 100 characters of extracted_page_text\n",
    "print(extracted_doc_data.extracted_text[0:100])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Extract Text from a document that has already been uploaded into the platform\n",
    "\n",
    "When you upload documents directly into the platform, depending on the settings you choose, you will already have access to `embedded_text` or `extracted_text`, etc. Here is how you can load the text of a file that has already been uploaded into the file:\n",
    "\n",
    "1. Find the `doc_id`. You can find that in the feature group where the documents where uploaded under `doc_id` column.\n",
    "2. Use `get_docstore_document_data` to get document's data.\n",
    "\n",
    "If OCR is not used when ingesting the document, then `extracted_text` won't exist"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "------------------------------\n",
      "Embedded Text:\n",
      "\n",
      "UNITED STATES\n",
      "SECURITIES AND EXCHANGE COMMISSION\n",
      "Washington, D.C. 20549\n",
      "FORM 10-K\n",
      "(Mark One)\n",
      "☒\n",
      "ANNUA\n",
      "------------------------------\n",
      "Extracted (OCR) Text:\n",
      "\n",
      "UNITED STATES\n",
      "SECURITIES AND EXCHANGE COMMISSION\n",
      "Washington, D.C. 20549\n",
      "(Mark One)\n",
      "ANNUAL REPORT PUR\n"
     ]
    }
   ],
   "source": [
    "doc_data = client.get_docstore_document_data('115fd750d0-000000000-bde8f7f6ce6065337e599fcac194739685fb3d3060650f6d7ef95bac914c72bc')\n",
    "# print first 100 chracters from embedded text\n",
    "print('------------------------------')\n",
    "print('Embedded Text:\\n')\n",
    "print(doc_data.embedded_text[0:100])\n",
    "print('------------------------------')\n",
    "# print first 100 chracters from OCR detected text\n",
    "print('Extracted (OCR) Text:\\n')\n",
    "print(extracted_doc_data.extracted_text[0:100]) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Load a feature group with documents locally as a pandas dataframe"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# To access docstore later, or when it was created outside of this notebook, we may use the name or id of it by functions describe_feature_group_by_table_name or describe_feature_group, respectively\n",
    "\n",
    "df = client.describe_feature_group_by_table_name('YOUR_FEATURE_GROUP_NAME').load_as_pandas_documents(doc_id_column = 'doc_id',document_column = 'page_infos')\n",
    "df['page_infos'][0].keys()\n",
    "# dict_keys(['pages', 'tokens', 'metadata', 'extracted_text'])\n",
    "\n",
    "#pages: This is the embedded text from the document on a per page level\n",
    "#extracted_text: This is the OCR extracted text from the document"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
