---
title: 'User-Defined Functions'
description: 'The core concepts to understand multi-engine system'
---

## Custom UD(X)F System

xorq provides a powerful system for extending query engines with custom User-Defined Functions (UDFs). Here are three key types supported:

### 1. Scalar UDF with Model Integration

```python
import xorq as xo
from xorq.expr.ml import make_quickgrove_udf
from pathlib import Path
from xorq import _

t = xo.examples.diamonds.fetch()

model_path = Path(xo.options.pins.get_path("diamonds-model"))
model = make_quickgrove_udf(model_path, model_name="diamonds_model")
expr = t.mutate(pred=model.on_expr).filter(_.carat < 1).select(_.pred).execute()
```

### 2. User Defined Aggregate Functions
```python
from xorq.expr import udf
import xorq.vendor.ibis.expr.datatypes as dt

alltypes = xo.examples.functional_alltypes.fetch()
cols = (by, _) = ["year", "month"]
name = "sum_sum"

@udf.agg.pandas_df(
    schema=alltypes[cols].schema(),
    return_type=dt.int64(),
    name=name,
)
def sum_sum(df):
    return df.sum().sum()

actual = (
    alltypes.group_by(by)
    .agg(sum_sum(*(alltypes[c] for c in cols)).name(name))
    .execute()
)
```

Additionally you can use UDAF for training ML models, see the [example](https://github.com/letsql/xorq/blob/main/examples/xgboost_udaf.py) for training an XGBoost model

### 3. Window UDF for Analysis
```python
import pyarrow as pa
from xorq.expr.udf import pyarrow_udwf
from xorq.vendor import ibis


@pyarrow_udwf(
    schema=ibis.schema({"a": float}),
    return_type=ibis.dtype(float),
    alpha=0.9,
)
def exp_smooth(self, values: list[pa.Array], num_rows: int) -> pa.Array:
    results = []
    curr_value = 0.0
    values = values[0]
    for idx in range(num_rows):
        if idx == 0:
            curr_value = values[idx].as_py()
        else:
            curr_value = values[idx].as_py() * self.alpha + curr_value * (
                1.0 - self.alpha
            )
        results.append(curr_value)

    return pa.array(results)


df = pd.DataFrame(
    [
        (0, 7, "A"),
        (1, 4, "A"),
        (2, 3, "A"),
        (3, 8, "A"),
        (4, 9, "B"),
        (5, 1, "B"),
        (6, 6, "B"),
    ],
    columns=["a", "b", "c"],
)

t = xo.register(df, table_name="t")

expr = t.select(
    t.a,
    udwf=exp_smooth.on_expr(t).over(ibis.window()),
).order_by(t.a)

result = expr.execute()
```

## Ephemeral Flight Service

xorq's Ephemeral Flight Service provides a high-performance data transfer mechanism between engines using Apache Arrow Flight. Unlike traditional data transfer methods, this service provides:

1. **Automatic Lifecycle Management**
2. **Zero-Copy Data Movement**
    - Direct memory transfer between processes
    - No serialization/deserialization overhead
    - Efficient handling of large datasets
3. **Process Isolation**
    - Separate processes for different engines
    - Independent resource management
    - Fault isolation
4. **Resource Management**
5. **Security Integration**
