# Getting Started

This tutorial will walk you through your first steps with `fsspeckit`. You'll learn how to install the library, work with local and cloud storage, and perform basic dataset operations.

> **Package Structure Note:** fsspeckit has been refactored to use a package-based structure. While legacy import paths still work for backward compatibility, the new structure provides better organization and discoverability. See the [Architecture Guide](../explanation/architecture.md) for details.

## Prerequisites

- Python 3.11 or higher
- Basic familiarity with Python and data concepts

## Installation

First, install `fsspeckit` with the dependencies you need:

```bash
# Basic installation
pip install fsspeckit

# With cloud storage support
pip install "fsspeckit[aws,gcp,azure]"

# With all optional dependencies for data processing
pip install "fsspeckit[aws,gcp,azure]" duckdb pyarrow polars sqlglot
```

For detailed installation instructions, see the [Installation Guide](../installation.md).

## Your First Local Filesystem

Let's start by creating a local filesystem and performing basic operations:

```python
# New preferred import path
from fsspeckit.core import filesystem
import os

# Legacy import (still works but deprecated)
# from fsspeckit.core.filesystem import filesystem

# Create a local filesystem
# Note: filesystem() wraps the filesystem in DirFileSystem by default (dirfs=True)
# for path safety, confining all operations to the specified directory
fs = filesystem("file")

# Define a directory path
local_dir = "./my_data/"
os.makedirs(local_dir, exist_ok=True)

# Create and write a file
with fs.open(f"{local_dir}example.txt", "w") as f:
    f.write("Hello, fsspeckit!")

# Read the file
with fs.open(f"{local_dir}example.txt", "r") as f:
    content = f.read()
print(f"Content: {content}")

# List files in directory
files = fs.ls(local_dir)
print(f"Files: {files}")
```

**Path Safety:** The `filesystem()` function wraps filesystems in `DirFileSystem` by default (`dirfs=True`), which confines all operations to the specified directory path. This prevents accidental access to paths outside the intended directory.

## Working with Cloud Storage

Now let's configure cloud storage. We'll use environment variables for credentials:

```python
from fsspeckit.storage_options import storage_options_from_env
from fsspeckit.core import filesystem

# Legacy import (still works but deprecated)
# from fsspeckit.core.filesystem import filesystem

# Set environment variables (or set them in your environment)
import os
os.environ["AWS_ACCESS_KEY_ID"] = "your_access_key"
os.environ["AWS_SECRET_ACCESS_KEY"] = "your_secret_key"
os.environ["AWS_DEFAULT_REGION"] = "us-east-1"

# Load AWS options from environment
aws_options = storage_options_from_env("s3")
fs = filesystem("s3", storage_options=aws_options.to_dict())

print(f"Created S3 filesystem in region: {aws_options.region}")
```

You can also configure storage manually:

```python
from fsspeckit.storage_options import AwsStorageOptions

# Configure AWS S3
aws_options = AwsStorageOptions(
    region="us-east-1",
    access_key_id="YOUR_ACCESS_KEY",
    secret_access_key="YOUR_SECRET_KEY"
)

# Create filesystem
aws_fs = aws_options.to_filesystem()
```

## Protocol Inference

The `filesystem()` function can automatically detect protocols from URIs:

```python
# Auto-detect protocols
s3_fs = filesystem("s3://bucket/path")      # S3
gcs_fs = filesystem("gs://bucket/path")      # Google Cloud Storage
az_fs = filesystem("az://container/path")    # Azure Blob Storage
github_fs = filesystem("github://owner/repo") # GitHub

# All work with the same interface
for name, fs in [("S3", s3_fs), ("GCS", gcs_fs)]:
    try:
        files = fs.ls("/")
        print(f"{name} files: {len(files)}")
    except Exception as e:
        print(f"{name} error: {e}")
```

## Your First Dataset Operation

Let's perform basic dataset operations using both DuckDB and PyArrow handlers:

### DuckDB Approach

```python
from fsspeckit.datasets import DuckDBParquetHandler
import polars as pl

# Initialize handler with storage options
storage_options = {"key": "value", "secret": "secret"}
handler = DuckDBParquetHandler(storage_options=storage_options)

# Create sample data
data = pl.DataFrame({
    "id": [1, 2, 3, 4],
    "category": ["A", "B", "A", "B"],
    "value": [10.5, 20.3, 15.7, 25.1]
})

# Write dataset
handler.write_parquet_dataset(data, "s3://bucket/my-dataset/")

# Execute SQL queries
result = handler.execute_sql("""
    SELECT category, COUNT(*) as count, AVG(value) as avg_value
    FROM parquet_scan('s3://bucket/my-dataset/')
    GROUP BY category
""")

print(result)
```

### PyArrow Approach

```python
from fsspeckit.datasets.pyarrow import PyarrowDatasetIO, PyarrowDatasetHandler
import pyarrow as pa

# Class-based approach
io = PyarrowDatasetIO()

# Handler wrapper approach with context manager
with PyarrowDatasetHandler() as handler:
    # Create sample data
    data = pa.table({
        "id": [1, 2, 3, 4],
        "category": ["A", "B", "A", "B"],
        "value": [10.5, 20.3, 15.7, 25.1]
    })

    # Write dataset with merge strategy
    handler.write_parquet_dataset(data, "s3://bucket/my-dataset/", strategy="upsert", key_columns=["id"])

    # Read dataset
    table = handler.read_parquet("s3://bucket/my-dataset/", columns=["category", "value"])
    print(f"Read {table.num_rows} rows with columns: {table.column_names}")

# Direct class usage for maintenance operations
stats = io.compact_parquet_dataset("s3://bucket/my-dataset/", target_mb_per_file=64)
print(f"Compaction stats: {stats}")
```

## Domain Package Structure

`fsspeckit` is organized into domain-specific packages. Import from the appropriate package for your use case:

```python
# Filesystem creation and core functionality
from fsspeckit.core.filesystem import filesystem

# Storage configuration
from fsspeckit.storage_options import AwsStorageOptions, storage_options_from_env

# Dataset operations
from fsspeckit.datasets import DuckDBParquetHandler, PyarrowDatasetHandler
from fsspeckit.datasets.duckdb import DuckDBDatasetIO
from fsspeckit.datasets.pyarrow import PyarrowDatasetIO

# SQL filter translation
from fsspeckit.sql.filters import sql2pyarrow_filter, sql2polars_filter

# Common utilities
from fsspeckit.common.misc import run_parallel
from fsspeckit.common.types import dict_to_dataframe

# Backwards compatibility (legacy)
from fsspeckit.utils import DuckDBParquetHandler  # Still works
```

## Next Steps

Congratulations! You've completed the basic fsspeckit tutorial. Here are some recommended next steps:

### Explore More Features

- **How-to Guides**: Dive into specific tasks with our [How-to Guides](../how-to/)
- **API Reference**: Browse the [API Reference](../reference/api-guide.md) for detailed documentation
- **Architecture & Concepts**: Understand the design principles in [Architecture & Concepts](../explanation/)

### Common Use Cases

1. **Cloud Data Processing**: Use `storage_options_from_env()` for production deployments
2. **Dataset Operations**: Use `DuckDBParquetHandler` for SQL-based operations or `PyarrowDatasetHandler` for memory-efficient PyArrow operations
3. **SQL Filtering**: Use `sql2pyarrow_filter()` and `sql2polars_filter()` for cross-framework compatibility
4. **Safe Operations**: Use `DirFileSystem` for security-critical applications
5. **Performance**: Use `run_parallel()` for concurrent file processing

### Production Tips

1. **Use Domain Packages**: Import from `fsspeckit.datasets`, `fsspeckit.storage_options`, etc. instead of utils
2. **Environment Configuration**: Load credentials from environment variables in production
3. **Error Handling**: Always wrap remote filesystem operations in try-except blocks
4. **Type Safety**: Use structured `StorageOptions` classes instead of raw dictionaries
5. **Testing**: Use `LocalStorageOptions` and `DirFileSystem` for isolated test environments

For more detailed information, explore the other sections of the documentation.