"""
Spark Simplicity - Parquet File Reader
======================================

High-performance Parquet file reader optimized for enterprise-grade analytics and big data
processing. This module provides advanced Parquet reading capabilities with intelligent
column selection, distributed processing optimization, and comprehensive cluster validation.
Designed for production data lake environments and large-scale analytics workflows.

Key Features:
    - **Columnar Performance**: Optimized reading of Apache Parquet columnar format
    - **Predicate Pushdown**: Advanced column selection for query optimization
    - **Distributed Processing**: Full cluster parallelization for large datasets
    - **Schema Validation**: Automatic schema inference and column validation
    - **Cluster Compatibility**: Integrated path validation for shared storage
    - **Production Safety**: Comprehensive error handling and logging

Parquet Format Advantages:
    **Performance Optimization**:
    - Columnar storage for analytical query performance
    - Built-in compression reducing I/O overhead
    - Predicate pushdown optimization for selective reading
    - Schema evolution support for data lake scenarios

    **Data Lake Integration**:
    - Native support for complex nested data structures
    - Efficient handling of large datasets with partitioning
    - Schema preservation across different processing engines
    - Cross-platform compatibility for data sharing

Advanced Features:
    **Column Selection Optimization**:
    - Selective column reading for improved performance
    - Automatic validation of requested columns against schema
    - Predicate pushdown to reduce data transfer and processing
    - Memory optimization through targeted data loading

    **Cluster Processing**:
    - Distributed file reading across Spark executors
    - Automatic partitioning for optimal parallelization
    - Shared storage validation for enterprise environments
    - Resource optimization for large-scale processing

Enterprise Integration:
    - **Data Lake Compatibility**: Optimized for data lake architectures
    - **Big Data Processing**: Efficient handling of multi-gigabyte files
    - **Analytics Workflows**: Integration with business intelligence tools
    - **ETL Pipeline Support**: High-performance data ingestion capabilities
    - **Multi-Format Ecosystem**: Seamless integration with other data formats

Usage:
    This module provides the primary interface for Parquet data ingestion in
    Spark Simplicity, optimized for analytics workloads and data lake processing.

    from spark_simplicity.io.readers.parquet_reader import load_parquet
"""

from pathlib import Path
from typing import List, Optional, Union

from pyspark.sql import DataFrame, SparkSession

from ...logger import get_logger
from ..validation.path_utils import configure_spark_path

# Logger for Parquet reader
_parquet_logger = get_logger("spark_simplicity.io.readers.parquet")


def load_parquet(
    spark: SparkSession,
    file_path: Union[str, Path],
    columns: Optional[List[str]] = None,
    shared_mount: bool = False,
    **options,
) -> DataFrame:
    """
    Load Parquet files with enterprise-grade performance optimization and cluster validation.

    Provides high-performance Parquet data ingestion optimized for analytics workloads and
    data lake processing. This function leverages Spark's native columnar processing capabilities
    while adding intelligent column selection, comprehensive error handling, and cluster-wide
    validation essential for production data processing environments.

    The function is specifically designed for large-scale analytics scenarios where Parquet's
    columnar format provides significant performance advantages through predicate pushdown,
    compression efficiency, and schema evolution capabilities.

    Args:
        spark: Active SparkSession instance configured for distributed processing.
              Must have appropriate executor resources allocated for the expected
              data volume. Used for both DataFrame creation and cluster validation.
        file_path: Path to Parquet file or directory containing Parquet files.
                  Can be provided as string or Path object. Supports single files,
                  partitioned directories, and multi-file datasets. Compatible with
                  local storage, network mounts, HDFS, and cloud storage systems.
        columns: Optional list of specific column names to load for performance optimization.
                When specified, enables predicate pushdown to read only required columns,
                significantly improving performance and reducing memory usage for wide tables.
                Column names must exist in the Parquet schema or ValueError will be raised.
        shared_mount: Boolean indicating whether the file resides on shared storage
                     accessible by all cluster nodes. When True, triggers comprehensive
                     cluster validation to ensure all executors can access the data.
                     When False, uses local file URI scheme for single-node processing.
        **options: Additional Spark DataFrameReader options for fine-tuning Parquet processing:
                  - 'mergeSchema': Boolean to merge schemas across multiple files
                  - 'pathGlobFilter': Glob pattern to filter files in directory
                  - 'recursiveFileLookup': Boolean to recursively scan subdirectories
                  - 'datetimeRebaseMode': Handling of legacy datetime formats
                  - 'int96RebaseMode': Handling of legacy timestamp formats

    Returns:
        Spark DataFrame containing the loaded Parquet data:
        - Schema automatically inferred from Parquet metadata
        - Data types preserved from original Parquet format
        - Partitioning information maintained for query optimization
        - Column order preserved from file schema
        - Null values properly handled according to Parquet specification

    Raises:
        FileNotFoundError: If the specified file or directory does not exist.
                          Error message includes full path for troubleshooting.
        ValueError: If requested columns are not found in the Parquet schema.
                   The error specifies which columns are missing for easy correction.
        RuntimeError: If Parquet loading fails due to format corruption, permission issues,
                     cluster validation failures, or resource constraints. Includes
                     detailed error context for troubleshooting and resolution.

    Performance Optimization Features:
        **Column Selection (Predicate Pushdown)**:
        - Only specified columns are read from storage
        - Significant I/O reduction for wide tables
        - Memory usage optimization for large datasets
        - Network transfer reduction in distributed environments

        **Distributed Processing**:
        - Automatic file splitting across cluster executors
        - Parallel processing of partitioned datasets
        - Optimal resource utilization for large files
        - Load balancing across available cluster nodes

    Examples:
        Load complete Parquet dataset for full analysis:

         df = load_parquet(spark, "sales_data.parquet")
         print(f"Loaded {df.count()} records with {len(df.columns)} columns")
         df.printSchema()

        Optimized loading with column selection for analytics:

         # Load only required columns for performance
         analytics_df = load_parquet(
        ...     spark,
        ...     "large_dataset.parquet",
        ...     columns=["customer_id", "purchase_amount", "transaction_date"]
        ... )
         # Significantly faster than loading all columns

        Load partitioned dataset from shared storage:

         partitioned_df = load_parquet(
        ...     spark,
        ...     "/data/warehouse/sales/",  # Partitioned directory
        ...     shared_mount=True,
        ...     recursiveFileLookup=True,  # Include subdirectories
        ...     pathGlobFilter="*.parquet"  # Filter specific files
        ... )

        Load with schema merging for evolving datasets:

         evolved_df = load_parquet(
        ...     spark,
        ...     "evolving_dataset/",
        ...     mergeSchema=True,  # Handle schema evolution
        ...     datetimeRebaseMode="CORRECTED"  # Handle date formats
        ... )

        Performance comparison - column selection vs full load:

         # Full dataset load (slower)
         full_df = load_parquet(spark, "wide_table.parquet")

         # Optimized column selection (faster)
         subset_df = load_parquet(
        ...     spark,
        ...     "wide_table.parquet",
        ...     columns=["id", "name", "value"]
        ... )
         # Can be 10x faster for wide tables

    Data Lake Integration Patterns:
        **Partitioned Dataset Processing**:
        - Automatic partition pruning for query optimization
        - Efficient processing of time-series data
        - Support for Hive-style partitioning schemes
        - Dynamic partition discovery and filtering

        **Multi-File Dataset Handling**:
        - Seamless processing of distributed datasets
        - Automatic schema inference across multiple files
        - Consistent data type handling across partitions
        - Efficient parallel processing of file collections

    Performance Characteristics:
        **Small Datasets (< 100MB)**:
        - Excellent performance with immediate loading
        - Minimal overhead for schema inference
        - Efficient memory usage

        **Medium Datasets (100MB - 10GB)**:
        - Good performance with distributed processing
        - Column selection provides significant benefits
        - Automatic partitioning optimization

        **Large Datasets (> 10GB)**:
        - Optimal performance with cluster parallelization
        - Essential to use column selection for memory efficiency
        - Partitioned datasets provide best performance

    Analytics Workload Optimization:
        **Business Intelligence Integration**:
        - Fast aggregation queries through columnar format
        - Efficient filtering and projection operations
        - Schema consistency for reporting tools

        **Data Science Workflows**:
        - High-performance feature extraction
        - Efficient sampling for model training
        - Fast exploratory data analysis

        **ETL Processing**:
        - Efficient data transformation pipelines
        - Fast data quality validation
        - Optimized data movement between systems

    Storage System Compatibility:
        **Local Storage**: High-performance local file processing
        **HDFS**: Native Hadoop ecosystem integration
        **Cloud Storage**: S3, Azure, GCS compatibility with mounted storage
        **Network Storage**: NFS, SMB support with cluster validation

    Error Handling and Validation:
        **Schema Validation**: Automatic verification of column existence
        **Format Verification**: Parquet format integrity checking
        **Access Validation**: Comprehensive permission and accessibility testing
        **Resource Monitoring**: Memory and processing resource validation

    See Also:
        - CSV readers: ``load_csv()`` for comma-separated data
        - JSON readers: ``load_json()`` for semi-structured data
        - Text readers: ``load_text()`` for unstructured data
        - Parquet writers: ``write_parquet()`` for high-performance output

    Note:
        This function is optimized for production analytics workloads and provides
        the foundation for high-performance data lake processing. It leverages Parquet's
        columnar advantages while ensuring reliability and cluster compatibility for
        enterprise environments.
    """
    file_path = Path(file_path)

    if not file_path.exists():
        raise FileNotFoundError(
            f"File not found — please check that the path is correct and the "
            f"file exists: {file_path}"
        )

    try:
        reader = spark.read

        # Add any additional options
        for key, value in options.items():
            reader = reader.option(key, value)

        # Configure Spark path with validation
        spark_path = configure_spark_path(file_path, shared_mount, spark)

        df = reader.parquet(spark_path)

        # Select specific columns if requested (predicate pushdown optimization)
        if columns:
            missing_cols = set(columns) - set(df.columns)
            if missing_cols:
                raise ValueError(f"Columns not found in Parquet file: {missing_cols}")
            df = df.select(*columns)

        _parquet_logger.info("Parquet loaded successfully: %s", file_path.name)
        return df

    except Exception as e:
        raise RuntimeError(
            f"Could not load the file (please check file format and accessibility) : "
            f"Parquet {file_path}: {str(e)}"
        ) from e
