---
title: 'Caching'
description: 'The core concepts to understand caching'
---

## Caching System

xorq provides a sophisticated caching system that enables efficient iterative
development of ML pipelines. The caching system allows you to:

- Cache results from upstream query engines
- Persist data locally or in remote storage
- Automatically invalidate cache when source data changes
- Chain caches across multiple engines

### Storage Types

xorq supports two main types of cache storage:

#### 1. SourceStorage

- Automatically invalidates cache when upstream data changes
- Persistence depends on the source backend
- Supports both remote (Snowflake, Postgres) and in-process (pandas, DuckDB) backends

```python
import xorq as xo
from xorq.caching import SourceStorage

# Connect to source database
pg = xo.postgres.connect_env()
con = xo.connect()  # empty connection

# Create source storage
storage = SourceStorage(source=con)

# Register table from postgres and cache it
batting = pg.table("batting")

# Cache the filtered data in the source backend
cached = (
    batting.filter(batting.yearID == 2015)
    .cache(storage=storage)  # cache expression
)

# Execute the query - results will be cached
result = xo.execute(cached)
```

#### 2. SnapshotStorage

- No automatic invalidation
- Ideal for one-off analyses
- Persistence depends on source backend

#### 3. ParquetStorage

- Special case of SourceStorage
- Caches results as Parquet files on local disk
- Uses source backend for writing and reading
- Ensures durable persistence

### Hashing Strategies

Cache invalidation uses different hashing strategies based on the storage type:

| Storage Type | Hash Components |
| --- | --- |
| In-Memory | Data bytes + Schema |
| Disk-Based | Query plan + Schema |
| Remote | Table metadata + Last modified time |
