---
title: HuggingFace Dataset API Reference
---

# HuggingFace Dataset API Reference

This section provides detailed API documentation for the HuggingFace dataset integration with CaDaR.

## Module: cadar.hf_dataset

The `cadar.hf_dataset` module provides classes and functions for integrating CaDaR with HuggingFace datasets.

### Classes

#### DatasetLoader

```python
class DatasetLoader:
    """
    A class for loading datasets from HuggingFace Hub or local files.
    
    This class provides methods to load datasets with various options including
    streaming, filtering, and column mapping.
    """
    
    def __init__(self, cache_dir: Optional[str] = None):
        """
        Initialize the DatasetLoader.
        
        Args:
            cache_dir (Optional[str]): Directory to cache downloaded datasets.
                                    If None, uses default HuggingFace cache.
        """
        
    def load_from_hub(
        self,
        dataset_name: str,
        subset: Optional[str] = None,
        split: Optional[str] = None,
        cache_dir: Optional[str] = None,
        filter_func: Optional[Callable] = None,
        column_mapping: Optional[Dict[str, str]] = None,
        streaming: bool = False
    ) -> Dataset:
        """
        Load a dataset from HuggingFace Hub.
        
        Args:
            dataset_name (str): Name of the dataset on HuggingFace Hub.
            subset (Optional[str]): Subset or configuration of the dataset.
            split (Optional[str]): Dataset split to load (train, test, validation).
            cache_dir (Optional[str]): Directory to cache the dataset.
            filter_func (Optional[Callable]): Function to filter examples.
            column_mapping (Optional[Dict[str, str]]): Mapping to rename columns.
            streaming (bool): Whether to use streaming mode for large datasets.
            
        Returns:
            Dataset: The loaded dataset.
            
        Raises:
            DatasetNotFoundError: If the dataset is not found.
            ValueError: If invalid parameters are provided.
        """
        
    def load_from_file(
        self,
        file_path: str,
        format: str = "json",
        split: Optional[str] = None,
        filter_func: Optional[Callable] = None,
        column_mapping: Optional[Dict[str, str]] = None
    ) -> Dataset:
        """
        Load a dataset from a local file.
        
        Args:
            file_path (str): Path to the dataset file.
            format (str): Format of the file (json, csv, parquet, etc.).
            split (Optional[str]): Dataset split name.
            filter_func (Optional[Callable]): Function to filter examples.
            column_mapping (Optional[Dict[str, str]]): Mapping to rename columns.
            
        Returns:
            Dataset: The loaded dataset.
            
        Raises:
            FileNotFoundError: If the file is not found.
            ValueError: If the file format is not supported.
        """
        
    def load_streaming(
        self,
        dataset_name: str,
        subset: Optional[str] = None,
        cache_dir: Optional[str] = None,
        column_mapping: Optional[Dict[str, str]] = None
    ) -> IterableDataset:
        """
        Load a dataset in streaming mode.
        
        Args:
            dataset_name (str): Name of the dataset on HuggingFace Hub.
            subset (Optional[str]): Subset or configuration of the dataset.
            cache_dir (Optional[str]): Directory to cache the dataset.
            column_mapping (Optional[Dict[str, str]]): Mapping to rename columns.
            
        Returns:
            IterableDataset: The loaded streaming dataset.
        """
```

#### CaDaRProcessor

```python
class CaDaRProcessor:
    """
    A class for applying CaDaR transliteration to HuggingFace datasets.
    
    This class provides methods to process datasets with CaDaR's transliteration
    capabilities, supporting various operations like transliteration,
    standardization, and bidirectional processing.
    """
    
    def __init__(
        self,
        darija: str = "Ma",
        batch_size: int = 32,
        num_proc: Optional[int] = None,
        progress: bool = True
    ):
        """
        Initialize the CaDaRProcessor.
        
        Args:
            darija (str): Darija dialect to use (default: "Ma" for Moroccan).
            batch_size (int): Batch size for processing.
            num_proc (Optional[int]): Number of processes for parallelization.
                                    If None, uses all available cores.
            progress (bool): Whether to show progress bars.
        """
        
    def process_dataset(
        self,
        dataset: Dataset,
        text_column: str,
        target_script: str,
        output_column: Optional[str] = None,
        standardize_input: bool = True,
        detect_script: bool = True,
        batch_size: Optional[int] = None,
        num_proc: Optional[int] = None,
        progress: Optional[bool] = None,
        filter_func: Optional[Callable] = None,
        preprocess_callback: Optional[Callable] = None,
        postprocess_callback: Optional[Callable] = None
    ) -> Dataset:
        """
        Process a dataset with CaDaR transliteration.
        
        Args:
            dataset (Dataset): The dataset to process.
            text_column (str): Name of the column containing text to process.
            target_script (str): Target script ("arabic" or "latin").
            output_column (Optional[str]): Name of the output column.
                                        If None, automatically generated.
            standardize_input (bool): Whether to standardize input text.
            detect_script (bool): Whether to auto-detect input script.
            batch_size (Optional[int]): Batch size for processing.
                                    If None, uses instance default.
            num_proc (Optional[int]): Number of processes for parallelization.
                                    If None, uses instance default.
            progress (Optional[bool]): Whether to show progress.
                                    If None, uses instance default.
            filter_func (Optional[Callable]): Function to filter examples.
            preprocess_callback (Optional[Callable]): Function called before processing.
            postprocess_callback (Optional[Callable]): Function called after processing.
            
        Returns:
            Dataset: The processed dataset with transliterated text.
            
        Raises:
            ValueError: If invalid parameters are provided.
            ColumnNotFoundError: If the specified text column doesn't exist.
        """
        
    def process_bidirectional(
        self,
        dataset: Dataset,
        text_column: str,
        arabic_output: str,
        latin_output: str,
        standardize_input: bool = True,
        batch_size: Optional[int] = None,
        num_proc: Optional[int] = None,
        progress: Optional[bool] = None,
        filter_func: Optional[Callable] = None,
        preprocess_callback: Optional[Callable] = None,
        postprocess_callback: Optional[Callable] = None
    ) -> Dataset:
        """
        Process a dataset bidirectionally, creating both Arabic and Latin versions.
        
        Args:
            dataset (Dataset): The dataset to process.
            text_column (str): Name of the column containing text to process.
            arabic_output (str): Name of the output column for Arabic text.
            latin_output (str): Name of the output column for Latin text.
            standardize_input (bool): Whether to standardize input text.
            batch_size (Optional[int]): Batch size for processing.
            num_proc (Optional[int]): Number of processes for parallelization.
            progress (Optional[bool]): Whether to show progress.
            filter_func (Optional[Callable]): Function to filter examples.
            preprocess_callback (Optional[Callable]): Function called before processing.
            postprocess_callback (Optional[Callable]): Function called after processing.
            
        Returns:
            Dataset: The processed dataset with both Arabic and Latin text.
        """
        
    def standardize_dataset(
        self,
        dataset: Dataset,
        text_column: str,
        output_column: Optional[str] = None,
        batch_size: Optional[int] = None,
        num_proc: Optional[int] = None,
        progress: Optional[bool] = None,
        filter_func: Optional[Callable] = None,
        preprocess_callback: Optional[Callable] = None,
        postprocess_callback: Optional[Callable] = None
    ) -> Dataset:
        """
        Standardize text in a dataset without changing script.
        
        Args:
            dataset (Dataset): The dataset to process.
            text_column (str): Name of the column containing text to process.
            output_column (Optional[str]): Name of the output column.
                                        If None, overwrites the input column.
            batch_size (Optional[int]): Batch size for processing.
            num_proc (Optional[int]): Number of processes for parallelization.
            progress (Optional[bool]): Whether to show progress.
            filter_func (Optional[Callable]): Function to filter examples.
            preprocess_callback (Optional[Callable]): Function called before processing.
            postprocess_callback (Optional[Callable]): Function called after processing.
            
        Returns:
            Dataset: The processed dataset with standardized text.
        """
```

#### ConfigurableProcessor

```python
class ConfigurableProcessor:
    """
    A processor that can be configured via YAML configuration files.
    
    This class provides a way to configure dataset loading and processing
    through configuration files, making it easier to reproduce experiments.
    """
    
    def __init__(self):
        """Initialize the ConfigurableProcessor."""
        
    @classmethod
    def from_config(cls, config_path: str) -> "ConfigurableProcessor":
        """
        Create a ConfigurableProcessor from a configuration file.
        
        Args:
            config_path (str): Path to the YAML configuration file.
            
        Returns:
            ConfigurableProcessor: The configured processor instance.
            
        Raises:
            FileNotFoundError: If the configuration file is not found.
            ValueError: If the configuration is invalid.
        """
        
    def set_dataset_config(
        self,
        name: str,
        subset: Optional[str] = None,
        split: Optional[str] = None,
        cache_dir: Optional[str] = None,
        filter_func: Optional[Callable] = None,
        column_mapping: Optional[Dict[str, str]] = None,
        streaming: bool = False
    ) -> None:
        """
        Configure dataset loading parameters.
        
        Args:
            name (str): Name of the dataset on HuggingFace Hub.
            subset (Optional[str]): Subset or configuration of the dataset.
            split (Optional[str]): Dataset split to load.
            cache_dir (Optional[str]): Directory to cache the dataset.
            filter_func (Optional[Callable]): Function to filter examples.
            column_mapping (Optional[Dict[str, str]]): Mapping to rename columns.
            streaming (bool): Whether to use streaming mode.
        """
        
    def set_processing_config(
        self,
        darija: str = "Ma",
        target_script: str = "latin",
        text_column: str = "text",
        output_column: Optional[str] = None,
        standardize_input: bool = True,
        detect_script: bool = True,
        batch_size: int = 32,
        num_proc: Optional[int] = None,
        progress: bool = True
    ) -> None:
        """
        Configure processing parameters.
        
        Args:
            darija (str): Darija dialect to use.
            target_script (str): Target script for processing.
            text_column (str): Name of the text column.
            output_column (Optional[str]): Name of the output column.
            standardize_input (bool): Whether to standardize input.
            detect_script (bool): Whether to auto-detect script.
            batch_size (int): Batch size for processing.
            num_proc (Optional[int]): Number of processes for parallelization.
            progress (bool): Whether to show progress.
        """
        
    def set_output_config(
        self,
        format: str = "parquet",
        save_path: str,
        overwrite: bool = True
    ) -> None:
        """
        Configure output parameters.
        
        Args:
            format (str): Output format (parquet, json, csv).
            save_path (str): Path to save the processed dataset.
            overwrite (bool): Whether to overwrite existing files.
        """
        
    def load_dataset(self) -> Dataset:
        """
        Load the dataset using the configured parameters.
        
        Returns:
            Dataset: The loaded dataset.
        """
        
    def process(self) -> Dataset:
        """
        Process the dataset using the configured parameters.
        
        Returns:
            Dataset: The processed dataset.
        """
        
    def save(self) -> None:
        """
        Save the processed dataset using the configured output parameters.
        """
```

### Functions

#### Convenience Functions

```python
def process_hf_dataset(
    dataset_name: str,
    text_column: str,
    target_script: str,
    subset: Optional[str] = None,
    split: Optional[str] = None,
    darija: str = "Ma",
    output_dir: Optional[str] = None,
    batch_size: int = 32,
    num_proc: Optional[int] = None,
    progress: bool = True
) -> Dataset:
    """
    Convenience function to process a HuggingFace dataset with CaDaR.
    
    Args:
        dataset_name (str): Name of the dataset on HuggingFace Hub.
        text_column (str): Name of the column containing text to process.
        target_script (str): Target script ("arabic" or "latin").
        subset (Optional[str]): Subset or configuration of the dataset.
        split (Optional[str]): Dataset split to load.
        darija (str): Darija dialect to use.
        output_dir (Optional[str]): Directory to save the processed dataset.
        batch_size (int): Batch size for processing.
        num_proc (Optional[int]): Number of processes for parallelization.
        progress (bool): Whether to show progress bars.
        
    Returns:
        Dataset: The processed dataset.
    """

def create_parallel_corpus(
    dataset_name: str,
    text_column: str,
    arabic_output: str,
    latin_output: str,
    subset: Optional[str] = None,
    split: Optional[str] = None,
    darija: str = "Ma",
    output_dir: Optional[str] = None,
    batch_size: int = 32,
    num_proc: Optional[int] = None,
    progress: bool = True
) -> Dataset:
    """
    Create a parallel corpus from a monolingual dataset.
    
    Args:
        dataset_name (str): Name of the dataset on HuggingFace Hub.
        text_column (str): Name of the column containing text to process.
        arabic_output (str): Name of the output column for Arabic text.
        latin_output (str): Name of the output column for Latin text.
        subset (Optional[str]): Subset or configuration of the dataset.
        split (Optional[str]): Dataset split to load.
        darija (str): Darija dialect to use.
        output_dir (Optional[str]): Directory to save the parallel corpus.
        batch_size (int): Batch size for processing.
        num_proc (Optional[int]): Number of processes for parallelization.
        progress (bool): Whether to show progress bars.
        
    Returns:
        Dataset: The parallel corpus with both Arabic and Latin text.
    """
```

## Exceptions

```python
class DatasetProcessingError(Exception):
    """Base exception for dataset processing errors."""
    
class ColumnNotFoundError(DatasetProcessingError):
    """Raised when a specified column is not found in the dataset."""
    
class InvalidScriptError(DatasetProcessingError):
    """Raised when an invalid script is specified."""
    
class UnsupportedFormatError(DatasetProcessingError):
    """Raised when an unsupported file format is specified."""
```

## Type Hints

```python
from typing import (
    Optional, Dict, Callable, Union, List, Any, 
    Tuple, Generator, Iterator
)
from datasets import Dataset, IterableDataset
```

## Usage Examples

```python
# Basic processing
from cadar.hf_dataset import process_hf_dataset

processed = process_hf_dataset(
    dataset_name="arabic_speech_corpus",
    text_column="text",
    target_script="latin",
    split="train"
)

# Advanced configuration
from cadar.hf_dataset import DatasetLoader, CaDaRProcessor

loader = DatasetLoader(cache_dir="./cache")
dataset = loader.load_from_hub(
    dataset_name="arabic_news",
    subset="default",
    split="train",
    filter_func=lambda x: len(x["text"]) > 50
)

processor = CaDaRProcessor(
    darija="Ma",
    batch_size=64,
    num_proc=8
)

processed = processor.process_dataset(
    dataset=dataset,
    text_column="text",
    target_script="latin",
    standardize_input=True
)
```
