abacusai.api_class.dataset — AbacusAI documentation

# `abacusai.api_class.dataset`

## Module Contents

### Classes

`ParsingConfig` | Helper class that provides a standard way to create an ABC using
`DocumentProcessingConfig` | Document processing configuration.
`DatasetDocumentProcessingConfig` | Document processing configuration for dataset imports.
`IncrementalDatabaseConnectorConfig` | Config information for incremental datasets from database connectors

class abacusai.api_class.dataset.ParsingConfig

  Bases: `abacusai.api_class.abstract.ApiClass`

  Helper class that provides a standard way to create an ABC using inheritance.

  escape: str
  csv_delimiter: str
  file_path_with_schema: str

class abacusai.api_class.dataset.DocumentProcessingConfig

  Bases: `abacusai.api_class.abstract.ApiClass`

  Document processing configuration.

  Parameters:

    - extract_bounding_boxes (bool) – Whether to perform OCR and extract bounding boxes. If False, no OCR will be done but only the embedded text from digital documents will be extracted. Defaults to False.

    - ocr_mode (OcrMode) – OCR mode. There are different OCR modes available for different kinds of documents and use cases. This option only takes effect when extract_bounding_boxes is True.

    - use_full_ocr (bool) – Whether to perform full OCR. If True, OCR will be performed on the full page. If False, OCR will be performed on the non-text regions only. By default, it will be decided automatically based on the OCR mode and the document type. This option only takes effect when extract_bounding_boxes is True.

    - remove_header_footer (bool) – Whether to remove headers and footers. Defaults to False. This option only takes effect when extract_bounding_boxes is True.

    - remove_watermarks (bool) – Whether to remove watermarks. By default, it will be decided automatically based on the OCR mode and the document type. This option only takes effect when extract_bounding_boxes is True.

    - convert_to_markdown (bool) – Whether to convert extracted text to markdown. Defaults to False. This option only takes effect when extract_bounding_boxes is True.

  extract_bounding_boxes: bool = False
  ocr_mode: abacusai.api_class.enums.OcrMode
  use_full_ocr: bool
  remove_header_footer: bool = False
  remove_watermarks: bool = True
  convert_to_markdown: bool = False

class abacusai.api_class.dataset.DatasetDocumentProcessingConfig

  Bases: `DocumentProcessingConfig`

  Document processing configuration for dataset imports.

  Parameters:

    - extract_bounding_boxes (bool) – Whether to perform OCR and extract bounding boxes. If False, no OCR will be done but only the embedded text from digital documents will be extracted. Defaults to False.

    - ocr_mode (OcrMode) – OCR mode. There are different OCR modes available for different kinds of documents and use cases. This option only takes effect when extract_bounding_boxes is True.

    - use_full_ocr (bool) – Whether to perform full OCR. If True, OCR will be performed on the full page. If False, OCR will be performed on the non-text regions only. By default, it will be decided automatically based on the OCR mode and the document type. This option only takes effect when extract_bounding_boxes is True.

    - remove_header_footer (bool) – Whether to remove headers and footers. Defaults to False. This option only takes effect when extract_bounding_boxes is True.

    - remove_watermarks (bool) – Whether to remove watermarks. By default, it will be decided automatically based on the OCR mode and the document type. This option only takes effect when extract_bounding_boxes is True.

    - convert_to_markdown (bool) – Whether to convert extracted text to markdown. Defaults to False. This option only takes effect when extract_bounding_boxes is True.

    - page_text_column (str) – Name of the output column which contains the extracted text for each page. If not provided, no column will be created.

  page_text_column: str

class abacusai.api_class.dataset.IncrementalDatabaseConnectorConfig

  Bases: `abacusai.api_class.abstract.ApiClass`

  Config information for incremental datasets from database connectors

  Parameters:

    timestamp_column (str) – If dataset is incremental, this is the column name of the required column in the dataset. This column must contain timestamps in descending order which are used to determine the increments of the incremental dataset.

  timestamp_column: str