# Extractlib

This is a Python package that provides a set of tools and utilities for processing and analyzing PDF documents. It includes functionality for extracting text and tables from PDFs, cleaning and preprocessing text data, and analyzing content for keywords and patterns. The package also provides a number of configuration options for customizing the behavior of the tools and utilities, making it flexible and easy to use in a variety of different contexts. Whether you need to extract data from PDF documents for data analysis, or analyze PDF content for specific keywords or patterns, this package provides the tools you need to get the job done quickly and efficiently.


## python dependency overview
### Python Dependency Install
<pre>pip install nltk==3.8.1 PyMuPDF==1.21.1 camelot-py==0.11.0 opencv-python==4.7.0.72 ghostscript==0.7</pre>

## manually install supporting binaries

### camelot dependencies
- https://camelot-py.readthedocs.io/en/master/user/install-deps.html#install-deps

#### Ubuntu
<pre>$ apt install ghostscript python3-tk</pre>

#### MacOS
<pre>$ brew install ghostscript tcl-tk</pre>

## windows dependency installations
- Install ghostscript: https://ghostscript.com/releases/gsdnld.html
- Install Tinker: https://platform.activestate.com/activestate/activetcl-8.6/auto-fork?_ga=2.93217438.2024444162.1679060315-1994225326.1678735799
 
# Configuration
- configuration file should be placed in the root of the project and named 'extractlib.config.json'

## Example 'extractlib.config.json' File
### this file should be located in the root of your project
<pre>
{
  "std_out_logging": true,
  "supported_file_types": [".pdf"],
  "invalid_content_regexs": ["X{2,}"],
  "stop_words": [
    "na",
    "dependent",
    "address",
    "plans",
    "network",
    "nonnetwork",
    "additional",
    "covered"
  ],
  "keywords": {
    "dental": 5,
    "vision": 5,    
    "life": 5,
    "disability": 5
  },
  "keyword_synonyms": {
    "dental": ["orthodontic", "Endo", "Perio", "Oral"],
    "vision": ["eye", "vision", "lens", "lenses", "contact", "contacts"],
    "life": [
      "accident",
      "critical",
      "illness",
      "accidental",
      "dismember",
      "AD&D"
    ],
    "disability": []
  },
  "word_min_length": 3
}
</pre>

# access config variables
<pre>
from extractlib.settings import config

print(json.dump(config.config_raw, indent=4))
</pre>
# Example implementation
<pre>
from extractlib.document.process import process_document
import json

def main(file: str):
    result = process_document(file,  exclude_pages=[2,3], use_multithreading=False, split_pages_output_dir='./output', delete_split_pages=False)
    # Save the HTML content to a temporary file
    with open('temp.json', 'w') as f:
        json.dump(result, f, indent=4)


if __name__ == '__main__':

    # get working directory
    import os
    target_dir = os.path.dirname(os.path.abspath(__file__))
    main(f'{target_dir}/_testdata/PDF.pdf')
</pre>