---
title: 'Pipelines as Expressions'
---

In this tutorial, we'll build an end-to-end machine learning pipeline using xorq expressions to predict the number of comments a Hacker News story will receive based on its title. The pipeline fetches live data, processes text, trains a model, and makes predictions - all expressed as a single composable expression.

## Why Use Expression-Based ML Pipelines?

Traditional ML pipelines typically involve multiple disjointed steps that require manual orchestration, leading to complex codebases and difficult-to-maintain systems. An expression-based approach using xorq offers significant advantages:

- **Multi-engine optimization:** Execute different parts of your pipeline on the most appropriate backends (PostgreSQL, DuckDB, DataFusion)
- **Smart caching:** Automatically cache intermediate results to avoid redundant computations
- **Lazy evaluation:** Define the entire pipeline before execution, enabling optimization across the full workflow

Unlike traditional ETL or ML pipeline frameworks that focus on tasks and scheduling, xorq treats data transformations as first-class expressions, making your pipelines more readable, maintainable, and efficient.

## Fetching Live Data

First, we set up our environment to fetch the latest stories from Hacker News:

```python
import pandas as pd
import toolz
import xgboost as xgb
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.metrics import mean_absolute_error

import xorq as xo
import xorq.vendor.ibis.expr.datatypes as dt
from xorq.caching import ParquetStorage
from xorq.common.utils.defer_utils import deferred_read_parquet
from xorq.common.utils.import_utils import import_python
from xorq.expr.ml import (
    deferred_fit_predict,
    deferred_fit_transform_series_sklearn,
    train_test_splits,
)

# Import helper module with UDFs
m = import_python(xo.options.pins.get_path("hackernews_lib"))

# Set up connection and storage for caching
con = xo.connect()
storage = ParquetStorage(source=con)

# Load data and split into train/test sets
train_expr, test_expr = (
    deferred_read_parquet(
        con,
        xo.options.pins.get_path("hn-fetcher-input-small.parquet"),
        "fetcher-input",
    )
    .pipe(m.do_hackernews_fetcher_udxf, inner_name="inner-named-flight-udxf")
    .pipe(
        train_test_splits,
        unique_key="id",
        test_sizes=(0.9, 0.1),
        random_seed=0,
    )
)
```

Using `deferred_read_parquet` allows us to load the data without immediate execution, keeping everything as expressions that can be optimized.

## Text Vectorization and Model Training

To process the story titles, we use TF-IDF vectorization with a deferred approach:

```python
transform_col = "title"
features = (transformed_col,) = (f"{transform_col}_transformed",)
target = "descendants"
target_predicted = f"{target}_predicted"

# Create deferred transformer for TF-IDF
deferred_fit_transform_tfidf = deferred_fit_transform_series_sklearn(
    col=transform_col,
    cls=TfidfVectorizer,
    return_type=dt.Array(dt.float64),
)
```

For our XGBoost model, we define custom functions for training and prediction:

```python
@toolz.curry
def fit_xgboost_model(feature_df, target_series, seed=0):
    xgb_r = xgb.XGBRegressor(
        objective="reg:squarederror",
        eval_metric=mean_absolute_error,
        n_estimators=20,
        seed=seed,
    )
    X = pd.DataFrame(feature_df.squeeze().tolist())
    xgb_r.fit(X, target_series)
    return xgb_r

@toolz.curry
def predict_xgboost_model(model, df):
    return model.predict(df.squeeze().tolist())

# Create deferred predictor using our functions
deferred_fit_predict_xgb = deferred_fit_predict(
    target=target,
    features=list(features),
    fit=fit_xgboost_model,
    predict=predict_xgboost_model,
    return_type=dt.float32,
)
```

## Building the Pipeline Expression

Finally, we compose everything together into a single expressions, that works as a pipeline:

```python
# Fit and transform with TF-IDF
(deferred_tfidf_model, tfidf_udaf, deferred_tfidf_transform) = (
    deferred_fit_transform_tfidf(
        train_expr,
        storage=storage,
    )
)

# Apply the transformation to training data
train_tfidf_transformed = train_expr.mutate(
    **{transformed_col: deferred_tfidf_transform.on_expr}
)

# Fit XGBoost and get prediction function
(deferred_xgb_model, xgb_udaf, deferred_xgb_predict) = deferred_fit_predict_xgb(
    train_tfidf_transformed,
    storage=storage,
)

# Apply transformation and prediction to test data
test_xgb_predicted = (
    test_expr.mutate(**{transformed_col: deferred_tfidf_transform.on_expr})
    .into_backend(xo.connect())
    .mutate(**{target_predicted: deferred_xgb_predict.on_expr})
)
```

## Executing and Evaluating the Pipeline

To run the pipeline and get predictions:

```python
test_xgb_predicted.execute()
```

The key advantage of this approach is that everything - from data fetching through prediction - is expressed as a single composable pipeline. xorq handles the execution details, optimization, and can even cache intermediate results.

This makes it easy to:
- Update the pipeline with new data
- Modify individual steps without rewriting the whole pipeline
- Cache and reuse expensive computations
- Execute different parts of the pipeline on different engines

This expression-based approach provides a clean, declarative way to build ML pipelines while maintaining the flexibility to use powerful ML libraries like scikit-learn and XGBoost.