# HuggingFace Dataset Configuration Guide

This guide provides comprehensive documentation for configuring the HuggingFace dataset application with CaDaR.

## Configuration Overview

The HuggingFace dataset application can be configured through:

1. Command-line arguments
2. YAML configuration files
3. Environment variables
4. Python code

Configuration options can be combined, with the following precedence:
1. Command-line arguments (highest)
2. Python code
3. Environment variables
4. YAML configuration file (lowest)

## YAML Configuration

The most comprehensive way to configure the application is through a YAML configuration file:

```yaml
# Example configuration file
dataset:
  # Dataset loading configuration
  name: "arabic_speech_corpus"
  subset: "default"
  split: "train"
  cache_dir: "./cache"
  streaming: false
  
  # Data filtering and transformation
  filter_func: null
  column_mapping:
    "ar_text": "text"
    "dialect": "metadata"

processing:
  # CaDaR processing configuration
  darija: "Ma"
  target_script: "latin"
  text_column: "text"
  output_column: "text_latin"
  
  # Performance settings
  batch_size: 32
  num_proc: 4
  standardize_input: true
  detect_script: true
  
  # Advanced options
  progress: true
  preserve_special_chars: true

output:
  # Output configuration
  format: "parquet"
  save_path: "./processed_data"
  overwrite: true
  compression: "gzip"

advanced:
  # Advanced options
  memory_limit: "8GB"
  prefetch_factor: 2
  error_handling: "skip"  # skip, raise, or fallback
```

## Configuration Sections

### Dataset Configuration

Controls how datasets are loaded and prepared.

```yaml
dataset:
  # Required: Name of the dataset on HuggingFace Hub
  name: "arabic_speech_corpus"
  
  # Optional: Subset or configuration of the dataset
  subset: "default"
  
  # Optional: Dataset split to load
  split: "train"
  
  # Optional: Directory to cache the dataset
  cache_dir: "./cache"
  
  # Optional: Whether to use streaming mode for large datasets
  streaming: false
  
  # Optional: Function to filter examples (as string)
  # Must be importable function in your Python environment
  filter_func: "my_module.filter_function"
  
  # Optional: Mapping to rename columns
  column_mapping:
    "ar_text": "text"
    "dialect": "metadata"
```

### Processing Configuration

Controls how the data is processed with CaDaR.

```yaml
processing:
  # Required: Darija dialect to use
  # Options: "Ma" (Moroccan), "Tn" (Tunisian), "Al" (Algerian), etc.
  darija: "Ma"
  
  # Required: Target script for processing
  # Options: "arabic", "latin", or "bidirectional"
  target_script: "latin"
  
  # Required: Name of the column containing text to process
  text_column: "text"
  
  # Required: Name of the output column
  # For bidirectional processing, see bidirectional_output below
  output_column: "text_latin"
  
  # Optional: Batch size for processing (default: 32)
  batch_size: 32
  
  # Optional: Number of processes for parallelization (default: CPU count)
  num_proc: 4
  
  # Optional: Whether to standardize input text (default: true)
  standardize_input: true
  
  # Optional: Whether to auto-detect input script (default: true)
  detect_script: true
  
  # Optional: Whether to show progress bars (default: true)
  progress: true
  
  # Optional: Whether to preserve special characters (default: true)
  preserve_special_chars: true
  
  # For bidirectional processing
  bidirectional_output:
    arabic: "text_arabic"
    latin: "text_latin"
```

### Output Configuration

Controls how processed data is saved.

```yaml
output:
  # Required: Output format
  # Options: "parquet", "json", "csv", "arrow"
  format: "parquet"
  
  # Required: Path to save the processed dataset
  save_path: "./processed_data"
  
  # Optional: Whether to overwrite existing files (default: true)
  overwrite: true
  
  # Optional: Compression format (default: depends on format)
  # Options: "gzip", "bz2", "xz", "none"
  compression: "gzip"
  
  # Optional: For CSV output
  csv_options:
    delimiter: ","
    quotechar: "\""
    encoding: "utf-8"
  
  # Optional: For JSON output
  json_options:
    orient: "records"
    lines: false
```

### Advanced Configuration

Controls advanced processing and performance options.

```yaml
advanced:
  # Optional: Memory limit for processing
  memory_limit: "8GB"
  
  # Optional: Prefetch factor for streaming
  prefetch_factor: 2
  
  # Optional: Error handling strategy
  # Options: "skip", "raise", "fallback"
  error_handling: "skip"
  
  # Optional: Custom preprocessing function
  preprocess_func: "my_module.preprocess"
  
  # Optional: Custom postprocessing function
  postprocess_func: "my_module.postprocess"
  
  # Optional: Logging configuration
  logging:
    level: "INFO"
    file: "./processing.log"
    format: "%(asctime)s - %(name)s - %(levelname)s - %(message)s"
```

## Environment Variables

You can use environment variables to override configuration:

```bash
# Dataset configuration
export CADAR_DATASET_NAME="arabic_speech_corpus"
export CADAR_DATASET_SUBSET="default"
export CADAR_DATASET_SPLIT="train"
export CADAR_DATASET_CACHE_DIR="./cache"

# Processing configuration
export CADAR_PROCESSING_DARIJA="Ma"
export CADAR_PROCESSING_TARGET_SCRIPT="latin"
export CADAR_PROCESSING_BATCH_SIZE=32
export CADAR_PROCESSING_NUM_PROC=4

# Output configuration
export CADAR_OUTPUT_FORMAT="parquet"
export CADAR_OUTPUT_SAVE_PATH="./processed_data"
```

## Command-Line Interface

### Basic Commands

```bash
# Process a dataset with basic options
cadar-dataset-process \
    --dataset-name "arabic_speech_corpus" \
    --dataset-split "train" \
    --text-column "text" \
    --target-script "latin" \
    --output-dir "./processed_data" \
    --darija "Ma"

# Process with a configuration file
cadar-dataset-process --config "config.yaml"

# Override specific config values
cadar-dataset-process \
    --config "config.yaml" \
    --batch-size 64 \
    --num-proc 8
```

### Complete CLI Reference

```bash
cadar-dataset-process [OPTIONS]

Required Options:
  --dataset-name TEXT        Name of the dataset on HuggingFace Hub
  --text-column TEXT        Name of the column containing text to process
  --target-script TEXT      Target script (arabic, latin, or bidirectional)
  
Dataset Options:
  --dataset-subset TEXT     Subset or configuration of the dataset
  --dataset-split TEXT      Dataset split to load
  --cache-dir TEXT         Directory to cache the dataset
  --streaming             Enable streaming mode for large datasets
  
Processing Options:
  --darija TEXT           Darija dialect (default: Ma)
  --output-column TEXT    Name of the output column
  --batch-size INTEGER    Batch size for processing (default: 32)
  --num-proc INTEGER      Number of processes for parallelization
  --standardize-input     Enable input standardization
  --no-standardize-input  Disable input standardization
  --detect-script         Enable script auto-detection
  --no-detect-script      Disable script auto-detection
  
Output Options:
  --output-dir TEXT       Directory to save the processed dataset
  --format TEXT           Output format (parquet, json, csv)
  --overwrite            Overwrite existing files
  --compression TEXT      Compression format
  
Bidirectional Options:
  --bidirectional        Enable bidirectional processing
  --arabic-output TEXT   Name of the Arabic output column
  --latin-output TEXT    Name of the Latin output column
  
Configuration:
  --config TEXT          Path to YAML configuration file
  
Other Options:
  --help                Show help message
  --version             Show version
```

## Python Configuration

You can also configure the application directly in Python:

```python
from cadar.hf_dataset import ConfigurableProcessor

# Create processor
processor = ConfigurableProcessor()

# Configure dataset loading
processor.set_dataset_config(
    name="arabic_speech_corpus",
    subset="default",
    split="train",
    cache_dir="./cache"
)

# Configure processing
processor.set_processing_config(
    darija="Ma",
    target_script="latin",
    text_column="text",
    output_column="text_latin",
    batch_size=32,
    num_proc=4
)

# Configure output
processor.set_output_config(
    format="parquet",
    save_path="./processed_data",
    overwrite=True
)

# Load and process
dataset = processor.load_dataset()
processed = processor.process()
processor.save()
```

## Configuration Templates

### Template 1: Small Dataset Quick Processing

```yaml
# quick_processing.yaml
dataset:
  name: "arabic_news"
  subset: "default"
  split: "train"

processing:
  darija: "Ma"
  target_script: "latin"
  text_column: "text"
  batch_size: 16

output:
  format: "json"
  save_path: "./quick_output"
```

### Template 2: Large Dataset Production Processing

```yaml
# production_processing.yaml
dataset:
  name: "massive_arabic_corpus"
  subset: "default"
  split: "train"
  streaming: true
  cache_dir: "/data/cache"

processing:
  darija: "Ma"
  target_script: "latin"
  text_column: "text"
  batch_size: 256
  num_proc: 16
  standardize_input: true

output:
  format: "parquet"
  save_path: "/data/processed"
  compression: "gzip"

advanced:
  memory_limit: "32GB"
  error_handling: "skip"
  logging:
    level: "INFO"
    file: "/logs/processing.log"
```

### Template 3: Research Experiment Configuration

```yaml
# research_experiment.yaml
dataset:
  name: "darija_sentiment"
  subset: "annotated"
  split: "train"
  filter_func: "research.filter_high_quality"

processing:
  darija: "Ma"
  target_script: "bidirectional"
  text_column: "text"
  bidirectional_output:
    arabic: "text_arabic"
    latin: "text_latin"
  batch_size: 32
  standardize_input: true
  detect_script: true

output:
  format: "parquet"
  save_path: "./experiments/sentiment"

advanced:
  preprocess_func: "research.add_features"
  postprocess_func: "research.calculate_metrics"
```

### Template 4: Development and Debugging

```yaml
# debug_processing.yaml
dataset:
  name: "arabic_speech_corpus"
  subset: "default"
  split: "train[:100]"  # Only first 100 examples

processing:
  darija: "Ma"
  target_script: "latin"
  text_column: "text"
  batch_size: 8  # Small batch size
  num_proc: 1    # Single process for easier debugging

output:
  format: "json"
  save_path: "./debug_output"

advanced:
  logging:
    level: "DEBUG"
    file: "./debug.log"
  error_handling: "raise"  # Fail on errors
```

## Common Configuration Patterns

### 1. Creating Multiple Script Versions

```yaml
# multi_script.yaml
dataset:
  name: "darija_news"
  split: "train"

processing:
  darija: "Ma"
  target_script: "bidirectional"
  text_column: "content"
  bidirectional_output:
    arabic: "text_arabic_std"
    latin: "text_latin"
  standardize_input: true

output:
  format: "parquet"
  save_path: "./multi_script_corpus"
```

### 2. Processing Multiple Datasets with Similar Configuration

```python
# process_multiple.py
from cadar.hf_dataset import ConfigurableProcessor
import yaml

# Base configuration
base_config = yaml.safe_load(open("base_config.yaml"))

datasets = [
    {"name": "arabic_news", "subset": "sports"},
    {"name": "arabic_news", "subset": "politics"},
    {"name": "arabic_news", "subset": "entertainment"}
]

for dataset_config in datasets:
    # Create a new processor for each dataset
    processor = ConfigurableProcessor()
    
    # Merge base config with dataset-specific config
    config = base_config.copy()
    config["dataset"].update(dataset_config)
    
    # Configure processor
    processor.set_dataset_config(**config["dataset"])
    processor.set_processing_config(**config["processing"])
    processor.set_output_config(
        save_path=f"./processed/{dataset_config['subset']}",
        **config["output"]
    )
    
    # Process dataset
    print(f"Processing {dataset_config['name']}/{dataset_config['subset']}")
    dataset = processor.load_dataset()
    processed = processor.process()
    processor.save()
```

### 3. Custom Preprocessing and Postprocessing

```yaml
# custom_processing.yaml
dataset:
  name: "social_media_darija"
  split: "train"

processing:
  darija: "Ma"
  target_script: "latin"
  text_column: "tweet_text"
  
advanced:
  preprocess_func: "my_module.preprocess_tweets"
  postprocess_func: "my_module.analyze_sentiment"
```

```python
# my_module.py
import re

def preprocess_tweets(example):
    # Clean up tweet text
    text = example["tweet_text"]
    
    # Remove URLs
    text = re.sub(r'http\S+', '', text)
    
    # Remove hashtags and mentions but keep the text
    text = re.sub(r'#([^\s]+)', r'\1', text)
    text = re.sub(r'@([^\s]+)', r'\1', text)
    
    # Store cleaned text
    example["cleaned_text"] = text
    return example

def analyze_sentiment(example):
    # Add sentiment analysis as postprocessing
    text = example["text_latin"]
    sentiment = analyze_sentiment_text(text)  # Your sentiment function
    
    example["sentiment"] = sentiment
    return example
```

## Validation

You can validate your configuration before processing:

```python
from cadar.hf_dataset import ConfigurableProcessor
import yaml

# Load configuration
with open("config.yaml", "r") as f:
    config = yaml.safe_load(f)

# Create processor
processor = ConfigurableProcessor()

# Validate and set configuration
try:
    processor.set_dataset_config(**config["dataset"])
    processor.set_processing_config(**config["processing"])
    processor.set_output_config(**config["output"])
    print("Configuration is valid!")
except Exception as e:
    print(f"Configuration error: {e}")
```

This comprehensive guide covers all aspects of configuring the HuggingFace dataset application with CaDaR. Use the examples and templates as starting points for your specific use case.
