# Design: Role-based documentation architecture for fsspeckit

## Goals

- Make it obvious where a new user should start and where an experienced user should look for task recipes or API details.
- Implement clear documentation roles (tutorials, how-to guides, reference, explanation) in both file layout and MkDocs navigation.
- Reduce duplication and drift between guides and API reference by treating mkdocstrings-generated pages as the primary source of truth for API signatures.
- Preserve, where practical, existing URLs or at least provide clear pointers from old entrypoints to new locations.

## Proposed Structure

### Tutorials (`docs/tutorials/`)

- `tutorials/getting-started.md`
  - Single, opinionated onboarding path:
    - Install fsspeckit (with a link out to `installation.md` for details).
    - Create a local filesystem and perform basic operations.
    - Configure one remote backend (e.g., S3) and perform a simple read/write.
    - Use a basic dataset helper (e.g., `DuckDBParquetHandler`) for an end-to-end example.
  - Avoids comprehensive option listings and advanced edge cases; instead links to how-to guides and reference.

### How-to Guides (`docs/how-to/`)

- `how-to/configure-cloud-storage.md`
  - Env-var based configuration (`storage_options_from_env`).
  - Options-object configuration (`AwsStorageOptions`, `GcsStorageOptions`, `AzureStorageOptions`, etc.).
  - URI-derived options (`storage_options_from_uri`), including example URIs.
- `how-to/work-with-filesystems.md`
  - Creating `filesystem()` instances for local and remote protocols.
  - DirFileSystem semantics: base path confinement, composition via `base_fs`.
  - High-level caching usage (`cached=True`, `cache_size` access, `sync_cache()`).
- `how-to/read-and-write-datasets.md`
  - Reading JSON/CSV/Parquet via extended I/O helpers and batching patterns.
  - Writing datasets (Parquet, CSV, JSON) from common DataFrame/table types.
  - Core `DuckDBParquetHandler` workflows (write, query via SQL, read back).
- `how-to/use-sql-filters.md`
  - Translating SQL WHERE clauses into PyArrow and Polars expressions.
  - Applying filters to datasets/tables with realistic schemas.
- `how-to/sync-and-manage-files.md`
  - File and directory synchronization between backends with `sync_files` and `sync_dir`.
  - Partition/path helpers for organizing partitioned datasets.
- `how-to/optimize-performance.md`
  - When and how to enable caching and parallelism.
  - Dataset optimization helpers (merge/compact/optimize) and recommended patterns.

Each how-to is task-oriented, assumes some familiarity with Python and fsspec-like concepts, and links to reference pages for API details as needed.

### Reference (`docs/reference/` and `docs/api/`)

- `reference/api-guide.md`
  - Capability index oriented around user questions (e.g., “Configure storage”, “Work with datasets”, “Use SQL filters”).
  - For each capability:
    - Short description.
    - Link to one or more `docs/api/fsspeckit.*.md` pages.
    - Links to relevant how-to guides and tutorials.
- `reference/utils.md`
  - Explains `fsspeckit.utils` as a façade.
  - Provides a mapping table:
    - Legacy import (from `fsspeckit.utils`).
    - Canonical import (from `fsspeckit.common`, `fsspeckit.datasets`, `fsspeckit.sql.filters`, etc.).
  - Encourages new code to use domain modules directly while reassuring existing users about compatibility.
- `installation.md`
  - Remains at the top level but functions as reference:
    - Supported Python versions.
    - Installation/upgrade instructions.
    - Optional dependency mapping and extras.
    - Brief troubleshooting (no usage examples).
- `docs/api/*`
  - Continues to be generated by mkdocstrings from source code.
  - Treated as the canonical source for signatures and detailed behavior.

### Explanation (`docs/explanation/`)

- `explanation/architecture.md`
  - Domain model and bounding context for core, storage_options, datasets, sql, common, and utils.
  - Integration patterns between packages (filesystem + storage options + datasets + sql + security).
  - Performance and security architecture at a conceptual level.
  - References to security/migration docs and how-to guides for concrete usage.
- `explanation/concepts.md`
  - Concepts that clarify why the APIs look the way they do:
    - Path safety and DirFileSystem.
    - Dataset vs file-level operations.
    - SQL filter abstraction over multiple backends.
    - Lazy optional dependencies and their impact on usage patterns.
- `explanation/migration-0.5.md`
  - Migration from pre–0.5.x to the domain package architecture.
  - Already exists as `docs/migration-0.5.md`; this change formalizes its role as explanation and improves discoverability via nav links.

## Landing Page and Navigation

- `index.md` becomes a concise landing page:
  - Short description of fsspeckit’s purpose.
  - Prominent links:
    - “Getting Started” (tutorial).
    - “How-to Guides”.
    - “API Reference”.
    - “Architecture & Concepts”.
    - “Installation”, “Migration Guide”, and “Contributing”.
- `mkdocs.yml` navigation:
  - Groups pages by documentation role rather than historical file names.
  - Keeps the number of top-level items small and clearly labeled.
  - Ensures that domain packages are the primary API surface in the API reference section and that `fsspeckit.utils` is marked as compatibility.

## Mapping from Existing Pages

- `docs/index.md`
  - Trim to landing page content; move detailed “Purpose/Key Features/Get Started” narrative into Getting Started and reference pages as appropriate.
- `docs/quickstart.md`
  - Source material for `tutorials/getting-started.md`.
  - After extraction, becomes either a wrapper or stub with links to the new tutorial/how-to pages.
- `docs/installation.md`
  - Retained as-is with minor tweaks to ensure it stays purely reference and matches `pyproject.toml`.
- `docs/api-guide.md`
  - Source for both `reference/api-guide.md` (capability index) and multiple how-to guides (filesystems, datasets, SQL filters, sync, performance).
- `docs/advanced.md`
  - Donates conceptual content to `explanation/architecture.md` and `explanation/concepts.md`.
  - Donates task recipes to how-to guides.
- `docs/examples.md`
  - Donates runnable examples to specific how-to guides (cloud storage, datasets, SQL filters, sync, performance).
  - Optionally replaced by a short index page that links into how-to guides, or removed entirely if redundant.
- `docs/utils.md`
  - Re-scoped to a compatibility-focused `reference/utils.md` with a mapping table and links to canonical modules.
- `docs/migration-0.5.md`
  - Relocated or reclassified as `explanation/migration-0.5.md` with improved cross-links from index, architecture, and utils/reference docs.
- `docs/architecture.md`
  - Becomes `explanation/architecture.md` with reduced implementation detail and better linkage to how-to and reference pages.

## Spec Interaction

- The existing `project-docs` specification already defines high-level requirements for:
  - Branding and naming.
  - Domain vs utils emphasis in documentation and API reference.
  - Security and migration documentation.
  - Documentation layering and content roles.
- This change primarily implements those requirements by:
  - Making doc roles explicit in both file layout and navigation.
  - Reducing duplication between guides and API reference.
  - Clarifying the intended entrypoints for different user journeys.
- To make the structure enforceable, this change may add narrowly scoped requirements that:
  - Require the homepage and navigation to expose each documentation layer (tutorials, how-to, reference, explanation).
  - Encourage, but do not rigidly mandate, a folder layout that aligns with these roles.
