# Internal Docs

These documents capture design notes and operational guidance that are intended
for internal use. Some content references internal infrastructure and may not be
appropriate for public distribution.

## Index
- [architecture_overview.md](architecture_overview.md) - Architecture diagrams: deployment models, cloud providers, security, scaling, operations.
- [admission_control.md](admission_control.md) - Admission control goals, config, and behavior.
- [backfills.md](backfills.md) - Backfill pipeline design and parameters.
- [backfill_conflicts.md](backfill_conflicts.md) - Conflict resolution details for backfills.
- [checkpointing.md](checkpointing.md) - Checkpoint storage and recovery design.
- [compaction_lineage_plan.md](compaction_lineage_plan.md) - Tracking lineage during compaction.
- [console.md](console.md) - Geneva console access and setup notes.
- [dependency_verification.md](dependency_verification.md) - Dependency packaging and verification.
- [deployment.md](deployment.md) - Deployment options and environments.
- [fragment_writers.md](fragment_writers.md) - Fragment writer responsibilities and flows.
- [job_metrics.md](job_metrics.md) - JobTracker metric definitions and debugging guidance.
- [materialized_views.md](materialized_views.md) - Materialized view design and behavior.
- [resource_guide.md](resource_guide.md) - Ray resource sizing guidance.
- [udf_retry_policy_design.md](udf_retry_policy_design.md) - Retry and error handling policies.
- [udtf.md](udtf.md) - Batch UDTF reference (API, execution model, production infrastructure).

- [udtf_chunking_design.md](udtf_chunking_design.md) - Scalar UDTF design for per-row 1:N expansion (implemented).

## Design Docs

Design docs for planned or not-yet-implemented features live in `designs/`:

- [designs/gpu_pipelining_overview.md](designs/gpu_pipelining_overview.md) - GPU underutilization problems and pipelining strategy overview.
- [designs/gpu_pipelining_phase1.md](designs/gpu_pipelining_phase1.md) - DataLoaderActors to prefetch batches and keep GPUs busy.
- [designs/gpu_pipelining_phase1_1.md](designs/gpu_pipelining_phase1_1.md) - Auto-tuning loader actor parameters.
- [designs/gpu_pipelining_phase2.md](designs/gpu_pipelining_phase2.md) - Separating CPU preprocessing from GPU work.
- [designs/gpu_pipelining_phase3.md](designs/gpu_pipelining_phase3.md) - CUDA streams for overlapping data transfer and computation.
- [designs/inference-udf-design.md](designs/inference-udf-design.md) - ML inference architecture options (API, local GPU, GPU-as-a-Service).
- [designs/multi_column_udf_design.md](designs/multi_column_udf_design.md) - UDFs that output multiple columns instead of single values.
- [designs/udf_ratelimiter.md](designs/udf_ratelimiter.md) - Client-side rate limiting for API-based UDFs.
- [designs/udf_type_ambiguity.md](designs/udf_type_ambiguity.md) - Resolving implicit SCALAR/ARRAY/RECORD_BATCH type detection issues.
- [designs/udtf_future.md](designs/udtf_future.md) - Unimplemented UDTF features: hardening, incremental refresh, schema evolution.

## Assets
- Diagrams and images live under `internal_docs/assets/`.
