# HuggingFace Dataset User Guide

This guide provides comprehensive instructions on using the HuggingFace dataset application with CaDaR for processing Darija text in various formats.

## Overview

The HuggingFace dataset application enables seamless integration between CaDaR's transliteration capabilities and HuggingFace's vast collection of datasets. This allows you to:

- Process Darija datasets from HuggingFace Hub
- Apply CaDaR's bidirectional transliteration to dataset text
- Create standardized versions of mixed-script Darija datasets
- Generate new features for NLP models by adding transliterated versions of text

## Installation

### Prerequisites

Before using the HuggingFace dataset application, ensure you have:

1. CaDaR installed (see [Installation Guide](../installation.md))
2. HuggingFace datasets library installed:
   ```bash
   pip install datasets
   ```

### Optional Dependencies

For enhanced functionality, you may want to install:

```bash
pip install tqdm  # For progress bars
pip install pandas  # For data manipulation
pip install pyarrow  # For fast dataset operations
```

## Quick Start

### Basic Usage

```python
from cadar.hf_dataset import DatasetLoader, CaDaRProcessor
from datasets import load_dataset

# Load a dataset from HuggingFace
dataset = load_dataset("arabic_speech_corpus", split="train")

# Initialize CaDaR processor
processor = CaDaRProcessor(darija="Ma")

# Process the dataset
processed_dataset = processor.process_dataset(
    dataset, 
    text_column="text",
    target_script="latin"  # Convert Arabic to Latin
)

print(processed_dataset[0]["text_latin"])  # Transliterated text
```

### Command Line Interface

```bash
# Process a dataset from HuggingFace Hub
cadar-dataset-process \
    --dataset-name "arabic_speech_corpus" \
    --dataset-config "default" \
    --split "train" \
    --text-column "text" \
    --target-script "latin" \
    --output-dir "./processed_data" \
    --darija "Ma"
```

## Core Components

### 1. DatasetLoader

The `DatasetLoader` class handles loading datasets from HuggingFace Hub or local files.

```python
from cadar.hf_dataset import DatasetLoader

# Initialize loader
loader = DatasetLoader()

# Load from HuggingFace Hub
dataset = loader.load_from_hub(
    dataset_name="arabic_speech_corpus",
    subset="default",
    split="train",
    cache_dir="./cache"
)

# Load from local file
local_dataset = loader.load_from_file(
    file_path="./data/my_dataset.json",
    format="json"
)

# Load with streaming for large datasets
streaming_dataset = loader.load_streaming(
    dataset_name="large_arabic_corpus",
    subset="default"
)
```

#### Advanced Loading Options

```python
# Apply filtering while loading
filtered_dataset = loader.load_from_hub(
    dataset_name="arabic_speech_corpus",
    subset="default",
    split="train",
    filter_func=lambda example: len(example["text"]) > 10
)

# Load with custom column mapping
mapped_dataset = loader.load_from_hub(
    dataset_name="multilingual_corpus",
    subset="arabic",
    split="train",
    column_mapping={
        "ar_text": "text",  # Rename ar_text to text
        "dialect": "metadata"  # Rename dialect to metadata
    }
)
```

### 2. CaDaRProcessor

The `CaDaRProcessor` class applies CaDaR's transliteration capabilities to dataset text.

```python
from cadar.hf_dataset import CaDaRProcessor

# Initialize processor
processor = CaDaRProcessor(
    darija="Ma",  # Moroccan Darija
    batch_size=32,  # Batch processing size
    num_proc=4  # Number of processes for parallelization
)

# Simple transliteration
processed = processor.process_dataset(
    dataset,
    text_column="text",
    target_script="latin",
    output_column="text_latin"
)

# Bidirectional transliteration (both Arabic to Latin and Latin to Arabic)
bidirectional = processor.process_bidirectional(
    dataset,
    text_column="text",
    arabic_output="text_arabic",
    latin_output="text_latin"
)

# Standardization only
standardized = processor.standardize_dataset(
    dataset,
    text_column="text",
    output_column="text_standardized"
)
```

#### Advanced Processing Options

```python
# Custom processing with callback
def custom_callback(example):
    # Apply custom logic before or after processing
    example["word_count"] = len(example["text"].split())
    return example

custom_processed = processor.process_dataset(
    dataset,
    text_column="text",
    target_script="latin",
    output_column="text_latin",
    preprocess_callback=custom_callback
)

# Process only specific subset
filtered_processing = processor.process_dataset(
    dataset,
    text_column="text",
    target_script="latin",
    output_column="text_latin",
    filter_func=lambda example: example["dialect"] == "Moroccan"
)
```

## Configuration

### Configuration File

Create a configuration file to specify common processing parameters:

```yaml
# cadar_dataset_config.yaml
dataset:
  name: "arabic_speech_corpus"
  subset: "default"
  split: "train"
  cache_dir: "./cache"

processing:
  darija: "Ma"
  target_script: "latin"
  batch_size: 32
  num_proc: 4
  text_column: "text"
  output_column: "text_latin"

output:
  format: "parquet"
  save_path: "./processed_data"
  overwrite: true

advanced:
  standardize_input: true
  detect_script: true
  preserve_special_chars: true
```

### Using Configuration

```python
from cadar.hf_dataset import ConfigurableProcessor

# Load from configuration file
processor = ConfigurableProcessor.from_config("cadar_dataset_config.yaml")
dataset = processor.load_dataset()
processed = processor.process()

# Or configure programmatically
processor = ConfigurableProcessor()
processor.set_dataset_config(
    name="arabic_speech_corpus",
    subset="default",
    split="train"
)
processor.set_processing_config(
    darija="Ma",
    target_script="latin",
    batch_size=64
)
processor.process()
```

## Common Workflows

### 1. Creating a Parallel Corpus

```python
from cadar.hf_dataset import DatasetLoader, CaDaRProcessor

# Load monolingual dataset
loader = DatasetLoader()
arabic_dataset = loader.load_from_hub("arabic_news", split="train")

# Create parallel version
processor = CaDaRProcessor(darija="Ma")
parallel_corpus = processor.process_bidirectional(
    arabic_dataset,
    text_column="text",
    arabic_output="text_arabic_standardized",
    latin_output="text_latin"
)

# Save the parallel corpus
parallel_corpus.save_to_disk("./parallel_corpus")
```

### 2. Preparing Data for Training

```python
# Mix of original and transliterated versions for training
def prepare_training_data(dataset):
    processor = CaDaRProcessor(darija="Ma")
    
    # Create multiple versions
    arabic_standardized = processor.standardize_dataset(
        dataset, text_column="text", output_column="text_std"
    )
    
    latin_version = processor.process_dataset(
        dataset, text_column="text", target_script="latin", output_column="text_latin"
    )
    
    # Combine features
    training_data = dataset.add_column("text_std", arabic_standardized["text_std"])
    training_data = training_data.add_column("text_latin", latin_version["text_latin"])
    
    return training_data

# Apply to your dataset
training_data = prepare_training_data(original_dataset)
```

### 3. Large-Scale Processing

```python
from datasets import load_dataset
from cadar.hf_dataset import DatasetLoader, CaDaRProcessor

# Stream processing for very large datasets
loader = DatasetLoader()
streaming_dataset = loader.load_streaming(
    dataset_name="massive_arabic_corpus",
    subset="default"
)

# Process in batches
processor = CaDaRProcessor(darija="Ma")

# Save processed batches
output_dir = "./processed_large_dataset"
for i, batch in enumerate(streaming_dataset):
    processed_batch = processor.process_dataset(
        batch, 
        text_column="text", 
        target_script="latin"
    )
    
    # Save batch to disk
    batch_path = f"{output_dir}/batch_{i}.parquet"
    processed_batch.to_parquet(batch_path)
    
    print(f"Processed batch {i}, saved to {batch_path}")
```

### 4. Quality Assessment

```python
from cadar.hf_dataset import DatasetLoader, CaDaRProcessor

loader = DatasetLoader()
dataset = loader.load_from_hub("arabic_social_media", split="test")

processor = CaDaRProcessor(darija="Ma")

# Process and assess quality
processed = processor.process_dataset(
    dataset,
    text_column="text",
    target_script="latin"
)

# Analyze transliteration quality
def assess_quality(example):
    original = example["text"]
    transliterated = example["text_latin"]
    
    # Check for script consistency
    arabic_chars = sum(1 for c in transliterated if '\u0600' <= c <= '\u06FF')
    
    # Simple quality metric
    quality_score = 1.0 - (arabic_chars / len(transliterated)) if len(transliterated) > 0 else 0
    
    example["quality_score"] = quality_score
    return example

assessed = processed.map(assess_quality)

# Filter by quality
high_quality = assessed.filter(lambda x: x["quality_score"] > 0.95)
print(f"High quality examples: {len(high_quality)} out of {len(assessed)}")
```

## Tips and Best Practices

1. **Memory Management**: Use streaming for large datasets to avoid memory issues
2. **Batch Processing**: Adjust batch size based on your system's memory and CPU
3. **Parallelization**: Use multiple processes for CPU-bound tasks
4. **Caching**: Enable HuggingFace's caching to avoid re-downloading datasets
5. **Validation**: Always validate a sample of processed data before full-scale processing
6. **Error Handling**: Implement proper error handling for malformed text

## Troubleshooting

For common issues and solutions, see the [Troubleshooting Guide](hf-dataset-troubleshooting.md).

## Examples

For detailed examples and use cases, see the [Examples Guide](hf-dataset-examples.md).
