# Query Internals

This page describes how `src/geneva/query.py` extends LanceDB queries with Geneva-specific behavior.

## Responsibilities
- Wrap the LanceDB query builder to add Geneva features.
- Serialize and deserialize query state for jobs and materialized views.
- Inject computed columns backed by UDFs.
- Handle blob columns and optional row address metadata.

## Key Types
- `GenevaQuery` stores the base LanceDB query plus Geneva extras.
- `PydanticUDFSpec` and `ColumnUDF` serialize UDFs for transport.
- `ExtractedTransform` represents a fully loaded UDF for execution.

## GenevaQueryBuilder
`GenevaQueryBuilder` subclasses `LanceEmptyQueryBuilder` and adds the following behavior.

- `select` accepts a list of column names or a mapping of output names to SQL expressions or UDFs.
- UDFs in `select` are stored with their output index so the output order is stable.
- `enable_internal_api` must be called before using internal-only methods.
- `with_fragments`, `with_row_address`, and `with_where_as_bool_column` are internal APIs.
- `to_query_object` and `from_query_object` serialize and rebuild basic queries.

## Schema Construction
- `_schema_for_query` resolves column names, SQL expressions, and UDF outputs.
- Dotted paths like `info.left` are resolved into struct fields.
- SQL expressions are type-inferred by scanning a single row.
- Meta columns `_rowid` and `_rowaddr` are appended when requested.

## Batch Execution Flow
`to_batches` orchestrates scanning, UDF execution, and result assembly.

- It computes a schema without meta columns to detect blob fields.
- It enables `_rowid` when needed for blob access or filter masks.
- It adds extra input columns required by UDFs and drops them after execution.
- It rejects shuffle, vector search, and full-text search because those are not implemented.
- It scans fragments one by one and applies global offset and limit across fragments.
- It can compute a `BACKFILL_SELECTED` column when `with_where_as_bool_column` is enabled.
- It materializes blob columns using `dataset.take_blobs`.
- It inserts UDF output columns into RecordBatch results.
- It returns a `RecordBatchReader` unless blobs are present, in which case batches are lists.

## Materialized Views
`create_materialized_view` builds a view table that stores query results.

- It validates the source table is non-empty.
- It warns when stable row IDs are not enabled.
- It stores metadata on the view schema, including the source query and source version.
- It seeds the view with `__source_row_id` and `__is_set` columns.
- It uses Lance directly to enable stable row IDs when namespaces are not used.

## Limitations
- Shuffle, vector search, full-text search, and rerank are not implemented.
- Internal APIs require explicit opt-in via `enable_internal_api`.
- Blob batches are lists and do not receive UDF output columns in that path.

## Example
```python
query = (
    table.search()
    .select({"text": "text", "embedding": my_udf})
    .where("score > 0.5")
    .limit(100)
)

result = query.to_arrow()
```
