Metadata-Version: 2.3
Name: metaxy
Version: 0.1.6
Summary: Pluggable versioned metadata management for incremental multimodal pipelines.
Author: Daniel Gafni
Author-email: Daniel Gafni <danielgafni16@gmail.com>
Classifier: Development Status :: 4 - Beta
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: Environment :: Console
Classifier: Operating System :: OS Independent
Classifier: Topic :: Scientific/Engineering
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Software Development :: Libraries
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Typing :: Typed
Requires-Dist: cyclopts>=4.4.0
Requires-Dist: narwhals>=2.14.0
Requires-Dist: typing-extensions>=4.0.0
Requires-Dist: pydantic-core>=2.0.0
Requires-Dist: polars>=1.35.0
Requires-Dist: polars-hash>=0.5.1
Requires-Dist: pydantic>=2.11.9
Requires-Dist: pydantic-settings>=2.11.0
Requires-Dist: python-dotenv>=1.0.0
Requires-Dist: pyyaml>=6.0.0
Requires-Dist: pytest-cases>=3.9.1
Requires-Dist: sqlglot>=28.1.0
Requires-Dist: tomli>=2.3.0
Requires-Dist: rich>=13.0.0
Requires-Dist: packaging>=25.0
Requires-Dist: tomli-w>=1.2.0
Requires-Dist: tomlkit>=0.13.2
Requires-Dist: ibis-framework[bigquery]>=11.0.0 ; extra == 'bigquery'
Requires-Dist: google-auth>=2.0.0 ; extra == 'bigquery'
Requires-Dist: ibis-framework[clickhouse]>=11.0.0 ; extra == 'clickhouse'
Requires-Dist: sqlalchemy>=2.0 ; extra == 'clickhouse'
Requires-Dist: dagster>=1.12.9 ; extra == 'dagster'
Requires-Dist: dagster-shared>=1.12.9 ; extra == 'dagster'
Requires-Dist: deltalake>=1.2.1 ; extra == 'delta'
Requires-Dist: ibis-framework>=11.0.0 ; extra == 'delta'
Requires-Dist: ibis-framework[duckdb]>=11.0.0 ; extra == 'duckdb'
Requires-Dist: duckdb>=1.4.1 ; extra == 'duckdb'
Requires-Dist: pyarrow>=18.0.0 ; extra == 'ibis'
Requires-Dist: ibis-framework>=11.0.0 ; extra == 'ibis'
Requires-Dist: lancedb>=0.15.0 ; extra == 'lancedb'
Requires-Dist: polars>=1.36.1 ; extra == 'lancedb'
Requires-Dist: fastmcp>=2.0.0 ; extra == 'mcp'
Requires-Dist: mermaid-py>=0.8.0 ; extra == 'mermaid'
Requires-Dist: ibis-framework[postgres]>=11.0.0 ; extra == 'postgres'
Requires-Dist: psycopg>=3.2.0 ; extra == 'postgres'
Requires-Dist: ray>2.36.0 ; (python_full_version < '3.14' and sys_platform != 'win32' and extra == 'ray') or (python_full_version < '3.13' and sys_platform == 'win32' and extra == 'ray')
Requires-Dist: sqlalchemy>=2.0 ; extra == 'sqlalchemy'
Requires-Dist: sqlmodel>=0.0.27 ; extra == 'sqlmodel'
Requires-Dist: sqlalchemy>=2.0 ; extra == 'sqlmodel'
Requires-Python: >=3.10
Project-URL: Changelog, https://docs.metaxy.io/latest/changelog
Project-URL: Documentation, https://docs.metaxy.io
Project-URL: Issues, https://github.com/anam-org/metaxy/issues
Project-URL: Repository, https://github.com/anam-org/metaxy
Provides-Extra: bigquery
Provides-Extra: clickhouse
Provides-Extra: dagster
Provides-Extra: delta
Provides-Extra: duckdb
Provides-Extra: ibis
Provides-Extra: lancedb
Provides-Extra: mcp
Provides-Extra: mermaid
Provides-Extra: postgres
Provides-Extra: ray
Provides-Extra: sqlalchemy
Provides-Extra: sqlmodel
Description-Content-Type: text/markdown

<p align="center">
  <img src="https://raw.githubusercontent.com/anam-org/metaxy/main/docs/assets/metaxy.svg" alt="Metaxy Logo" width="100"/>
</p>

<h1 align="center">Metaxy</h1>

<p align="center">
  <a href="https://pypi.org/project/metaxy/"><img src="https://img.shields.io/pypi/v/metaxy.svg?color=4644ad" alt="PyPI version"></a>
  <a href="https://pypi.org/project/metaxy/"><img src="https://img.shields.io/pypi/pyversions/metaxy.svg?color=4644ad" alt="Python versions"></a>
  <a href="https://pypi.org/project/metaxy/"><img src="https://img.shields.io/pypi/dm/metaxy.svg?color=4644ad" alt="PyPI downloads"></a>
  <a href="https://github.com/anam-org/metaxy/actions/workflows/main.yml"><img src="https://github.com/anam-org/metaxy/actions/workflows/main.yml/badge.svg" alt="CI"></a>
  <a href="https://codecov.io/gh/anam-org/metaxy"><img src="https://codecov.io/gh/anam-org/metaxy/graph/badge.svg" alt="codecov"></a>
  <a href="https://docs.astral.sh/ruff/"><img src="https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/astral-sh/ruff/main/assets/badge/v2.json" alt="Ruff"></a>
  <a href="https://docs.astral.sh/ty/"><img src="https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/astral-sh/ty/main/assets/badge/v0.json" alt="Ty"></a>
  <a href="https://prek.j178.dev"><img src="https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/j178/prek/master/docs/assets/badge-v0.json" alt="prek"></a>
</p>

Metaxy is a metadata layer for **multimodal Data and ML pipelines**. Metaxy tracks lineage and versioning across complex computational graphs for multimodal datasets. Metaxy can cache every single sample and scale to handle millions of them.

Metaxy manages **metadata** while **data** typically lives elsewhere:

```
┌─────────────────────────────────┐          ┌─────────────────────────┐
│      Metadata (Metaxy)          │          │   Data (e.g., S3)       │
├──────┬──────────┬──────┬────────┤          │                         │
│  ID  │   path   │ size │version │          │  📦 s3://my-bucket/     │
├──────┼──────────┼──────┼────────┤          │                         │
│ img1 │ s3://... │ 2.1M │a3fdsf  │ ────────>│    ├─ img1.jpg          │
│ img2 │ s3://... │ 1.8M │b7e123  │ ────────>│    ├─ img2.jpg          │
└──────┴──────────┴──────┴────────┘          └─────────────────────────┘
```

The feature that makes Metaxy stand out is the ability to track **partial data dependencies** and detect **prunable updates** — updates that don't trigger change propagation through certain paths in the dependency graph because they modify fields that aren't dependencies of those downstream features. For example, updating audio upstream of a face recognition step allows pruning the face recognition branch since it only depends on video frames. This problem is specific to multimodal pipelines and doesn't typically emerge in traditional data engineering.

Metaxy's goal is to provide a standard instrument for any kind of multimodal (or purely tabular) **incremental** pipelines, standardizing dependency specification, versioning, partial data dependencies, and manipulations over metadata. Or, in short, to be a universal glue for incremental data pipelines.

Metaxy is very reliable and is fanatically tested across all supported Python versions and platforms [^1].

## Documentation

Read the [docs](https://docs.metaxy.io) to learn more.

## Installation

Install Metaxy from [PyPI](https://pypi.org/project/metaxy/):

```shell
uv add metaxy
```

## Using Metaxy

Metaxy is highly pluggable and generally can be used with any kind of incremental pipelines, storage, metadata storage, and dataframe libraries.

Metaxy provides integrations with popular tools such as [Dagster](https://docs.metaxy.io/latest/integrations/orchestration/dagster), [Ray](https://docs.metaxy.io/latest/integrations/compute/ray), [ClickHouse](https://docs.metaxy.io/latest/integrations/metadata-stores/databases/clickhouse), [DeltaLake](https://docs.metaxy.io/latest/integrations/metadata-stores/storage/delta/), [SQLModel](https://docs.metaxy.io/latest/integrations/plugins/sqlmodel/).

The full list can be found [here](https://docs.metaxy.io/latest/integrations).

## Blog Posts

- [Announcement post](https://anam.ai/blog/metaxy) by Anam

- [Dagster + Metaxy](https://dagster.io/blog/building-real-time-interactive-avatars-with-metaxy) by Dagster labs

- [Docling + Slurm + Metaxy](https://georgheiler.com/2026/02/22/metaxy-dagster-slurm-multimodal/) by ASCII

## Contributing

See [CONTRIBUTING.md](CONTRIBUTING.md).

<!-- scarf.sh for telemetry collection (it does not record personal data such as IP addresses) -->
<img referrerpolicy="no-referrer" src="https://telemetry.metaxy.io/a.png?x-pxid=22cb75dc-201e-4a72-9fb2-c3a53ce9207e&page=README.md" />

[^1]: The CLI is not tested on Windows yet.
