# HuggingFace Dataset Troubleshooting Guide

This guide provides solutions to common issues encountered when using the HuggingFace dataset application with CaDaR.

## Installation Issues

### Issue: "No module named 'cadar.hf_dataset'"

**Problem**: The HuggingFace dataset module cannot be imported.

**Solutions**:
1. Ensure you have the latest version of CaDaR installed:
   ```bash
   pip install --upgrade cadar
   ```

2. If installing from source, make sure to install the additional requirements:
   ```bash
   pip install -e .[hf_dataset]
   ```

3. Verify the installation:
   ```python
   import cadar
   print(cadar.__version__)
   ```

### Issue: HuggingFace datasets library not found

**Problem**: `datasets` module cannot be imported.

**Solutions**:
1. Install the HuggingFace datasets library:
   ```bash
   pip install datasets
   ```

2. For specific features, install additional dependencies:
   ```bash
   pip install datasets[audio,vision]
   ```

## Dataset Loading Issues

### Issue: Dataset download fails

**Problem**: Dataset cannot be downloaded from HuggingFace Hub.

**Solutions**:
1. Check your internet connection:
   ```bash
   ping huggingface.co
   ```

2. Configure HuggingFace cache directory:
   ```python
   import os
   os.environ['HF_HOME'] = '/path/to/cache/directory'
   ```

3. Use a different mirror if available:
   ```python
   import os
   os.environ['HF_ENDPOINT'] = 'https://huggingface.co'
   ```

### Issue: Memory error when loading large datasets

**Problem**: Running out of memory when loading datasets.

**Solutions**:
1. Use streaming mode:
   ```python
   from cadar.hf_dataset import DatasetLoader
   
   loader = DatasetLoader()
   dataset = loader.load_streaming(
       dataset_name="large_dataset_name"
   )
   ```

2. Process in batches:
   ```python
   from cadar.hf_dataset import CaDaRProcessor
   
   processor = CaDaRProcessor(batch_size=16)  # Smaller batch size
   ```

3. Use disk-based processing:
   ```python
   from datasets import load_dataset
   
   dataset = load_dataset(
       "dataset_name",
       split="train",
       cache_dir="/path/to/disk/cache"
   )
   ```

### Issue: Dataset column not found

**Problem**: The specified text column doesn't exist in the dataset.

**Solutions**:
1. Check available columns:
   ```python
   dataset = load_dataset("dataset_name", split="train")
   print(dataset.column_names)
   ```

2. Use column mapping:
   ```python
   loader = DatasetLoader()
   dataset = loader.load_from_hub(
       dataset_name="dataset_name",
       column_mapping={
           "content": "text",  # Rename content to text
           "label": "target"    # Rename label to target
       }
   )
   ```

## Processing Issues

### Issue: Slow processing speed

**Problem**: Dataset processing is very slow.

**Solutions**:
1. Increase parallelization:
   ```python
   processor = CaDaRProcessor(num_proc=8)  # Use 8 processes
   ```

2. Use larger batch sizes:
   ```python
   processor = CaDaRProcessor(batch_size=128)
   ```

3. Disable progress bars for slight speedup:
   ```python
   processor = CaDaRProcessor(progress=False)
   ```

4. Use GPU if available (future feature):
   ```python
   processor = CaDaRProcessor(device="cuda")
   ```

### Issue: Poor transliteration quality

**Problem**: The output quality of transliteration is poor.

**Solutions**:
1. Ensure standardization is enabled:
   ```python
   processed = processor.process_dataset(
       dataset,
       text_column="text",
       target_script="latin",
       standardize_input=True
   )
   ```

2. Use the correct dialect:
   ```python
   processor = CaDaRProcessor(darija="Ma")  # Moroccan Darija
   ```

3. Filter out non-Darija text:
   ```python
   def is_darija(example):
       # Add logic to identify Darija text
       return check_if_darija(example["text"])
   
   darija_dataset = dataset.filter(is_darija)
   ```

### Issue: Processing fails on specific examples

**Problem**: Processing crashes on certain examples.

**Solutions**:
1. Add error handling:
   ```python
   def safe_process(example):
       try:
           text = example["text"]
           # Process text
           example["processed"] = processor.process_text(text)
           return example
       except Exception as e:
           print(f"Error processing example: {e}")
           example["processed"] = ""  # Fallback value
           return example
   
   safe_dataset = dataset.map(safe_process)
   ```

2. Filter problematic examples:
   ```python
   def is_valid(example):
       text = example["text"]
       # Check for issues like very long text or special characters
       return 0 < len(text) < 10000 and is_valid_text(text)
   
   valid_dataset = dataset.filter(is_valid)
   ```

## Output Issues

### Issue: Output file is too large

**Problem**: The processed output file is too large.

**Solutions**:
1. Use compressed formats:
   ```python
   processed.save_to_disk("./output", compression="gzip")
   ```

2. Split output into smaller files:
   ```python
   for i, shard in enumerate(processed.shard(num_shards=10, index=i)):
       shard.to_csv(f"./output_batch_{i}.csv")
   ```

3. Use Parquet format which is more space-efficient:
   ```python
   processed.to_parquet("./output.parquet")
   ```

### Issue: Cannot read output file

**Problem**: The output file cannot be read or is corrupted.

**Solutions**:
1. Verify the output was created successfully:
   ```python
   import os
   
   output_path = "./output.parquet"
   if os.path.exists(output_path):
       print(f"File exists, size: {os.path.getsize(output_path)} bytes")
   else:
       print("File does not exist")
   ```

2. Try different output formats:
   ```python
   # Try JSON format
   processed.to_json("./output.json")
   
   # Try CSV format
   processed.to_csv("./output.csv")
   ```

## CLI Issues

### Issue: Command not found

**Problem**: `cadar-dataset-process` command is not found.

**Solutions**:
1. Reinstall CaDaR with CLI support:
   ```bash
   pip install -e .
   ```

2. Check if the command is in your PATH:
   ```bash
   which cadar-dataset-process
   ```

3. Run the module directly:
   ```bash
   python -m cadar.cli.hf_dataset_process --help
   ```

### Issue: CLI arguments not recognized

**Problem**: The CLI doesn't recognize certain arguments.

**Solutions**:
1. Check the help documentation:
   ```bash
   cadar-dataset-process --help
   ```

2. Use argument names exactly as shown in help:
   ```bash
   # Correct
   cadar-dataset-process --dataset-name "my_dataset"
   
   # Incorrect
   cadar-dataset-process --dataset "my_dataset"
   ```

3. Use a configuration file for complex options:
   ```bash
   cadar-dataset-process --config my_config.yaml
   ```

## Performance Optimization

### Issue: Processing takes too long

**Problem**: The entire processing pipeline is slow.

**Solutions**:
1. Profile your processing:
   ```python
   import time
   
   start_time = time.time()
   processed = processor.process_dataset(dataset, ...)
   end_time = time.time()
   
   print(f"Processing took {end_time - start_time:.2f} seconds")
   ```

2. Process a smaller sample first:
   ```python
   # Process just 100 examples first
   small_dataset = dataset.select(range(100))
   small_processed = processor.process_dataset(small_dataset, ...)
   ```

3. Use caching to avoid reprocessing:
   ```python
   processed = dataset.map(
       lambda x: processor.process_text(x["text"]),
       cache_file_name="./processed_cache.arrow"
   )
   ```

## Environment-Specific Issues

### Issue: Docker/Mac M1 compatibility

**Problem**: Issues running in Docker or on Mac M1.

**Solutions**:
1. For Mac M1, ensure you have the right version:
   ```bash
   pip install --upgrade datasets datasets[fsspec]
   ```

2. For Docker, use the right image:
   ```dockerfile
   FROM python:3.9-slim
   
   # Install system dependencies
   RUN apt-get update && apt-get install -y \
       build-essential \
       curl \
       && rm -rf /var/lib/apt/lists/*
   
   # Install Python packages
   RUN pip install cadar datasets
   ```

3. Set environment variables for Docker:
   ```bash
   export HF_HOME=/app/cache
   export NUMBA_CACHE_DIR=/app/numba_cache
   ```

## Debugging Tips

1. Enable verbose logging:
   ```python
   import logging
   
   logging.basicConfig(level=logging.DEBUG)
   ```

2. Use a small sample for debugging:
   ```python
   debug_dataset = dataset.select(range(10))
   ```

3. Save intermediate results:
   ```python
   # Save before processing
   dataset.save_to_disk("./before_processing")
   
   # Process
   processed = processor.process_dataset(dataset, ...)
   
   # Save after processing
   processed.save_to_disk("./after_processing")
   ```

4. Check data types:
   ```python
   print(dataset.features)
   print(dataset[0])
   ```

## Getting Help

If you encounter an issue not covered in this guide:

1. Check the [GitHub Issues](https://github.com/Oit-Technologies/CaDaR/issues)
2. Create a new issue with:
   - The exact error message
   - A minimal reproducible example
   - Your environment information (OS, Python version, CaDaR version)
3. Include debug logs if possible
