<img src="/resources/tiders_logo2.png" alt="Tiders" width="1000">

[![PyPI](https://img.shields.io/badge/PyPI-lightgreen?style=for-the-badge&logo=pypi&labelColor=white)](https://pypi.org/project/tiders/)
[![tiders-core](https://img.shields.io/badge/github-black?style=for-the-badge&logo=github)](https://github.com/yulesa/tiders-core)
[![tiders-rpc-client](https://img.shields.io/badge/github-black?style=for-the-badge&logo=github)](https://github.com/yulesa/tiders-rpc-client)
[![Documentation](https://img.shields.io/badge/documentation-blue?style=for-the-badge&logo=readthedocs)](https://yulesa.github.io/tiders-docs/)

Tiders is an open-source framework that simplifies getting data out of blockchains and into your favorite tools. Whether you are building a DeFi dashboard, tracking NFT transfers, or running complex analytics, Tiders handles the heavy lifting of fetching, cleaning, transforming and storing blockchain data.

Tiders is modular. A Tiders pipeline is built from four components:

| Component | Description |
|---|---|
| `Provider` | Data source (HyperSync, SQD, or RPC) |
| `Query` | What data to fetch (block range, transaction, logs, filters, field selection) |
| `Steps` | Transformations to apply (decode, cast, encode, custom) |
| `Writer` | Output destination |

<img src="/resources/four_pipeline_components.png" alt="four_pipeline_components" width="1000">

**Why Tiders?**

Most indexers lock you into a specific platform or database. Tiders is built to be modular, meaning you can swap parts in and out without breaking your setup:
- Swap Providers: Don't like your current data source? Switch between HyperSync, SQD, or a standard RPC node by changing one line of code.
- Plug-and-Play data transformations: Need to decode smart contract events or change data types? Use our built-in Rust-powered steps or write your own custom logic.
- Write Anywhere: Send your data to a local DuckDB file for prototyping, or a production-grade ClickHouse or PostgreSQL instance when you're ready to scale.
- Modular Reusable Pipelines: Protocols often reuse the same data structures. You don't need write modules from scratch every time. Since Tiders pipelines are regular Python objects, you can build functions around them, reuse across pipelines, or set input parameters to customize as needed.  

## Getting Started

See [getting started section](https://yulesa.github.io/tiders-docs/getting_started.html) of the docs.

## Two ways to use tiders

| Mode | How | When to use |
|---|---|---|
| **Python SDK** | Write a Python script, import `tiders` | Full control, custom logic, complex pipelines |
| **CLI (No-Code)** | Write a YAML config, run `tiders start` | Quick setup, no Python required, standard pipelines |

Both modes share the same pipeline engine.

You can also use `tiders codegen` to generate a Python script from a YAML config — a quick way to move from no-code to full Python control.

## Installation

For both CLI and SDK

```bash
pip install tiders
```

## Features

- Continuous Ingestion: Keep your datasets live and fresh. Tiders can poll the chain head to ensure your data is always up to date.
- Switch Providers: Move between HyperSync, SQD, or standard RPC nodes with a single config change.
- No Vendor Lock-in: Use the best data providers in the industry without being tied to their specific platforms or database formats.
- Custom Logic: Easily extend and customize your pipeline code in Python for complete flexibility.
- Advanced Analytics: Seamlessly works with industry-standard tools like Polars, Pandas, Data fusion and PyArrow as the data is fetched.
- Multiple Outputs: Send the same data to a local file and a production database simultaneously.
- Rust-Powered Speed: Core tasks like decoding and transforming data are handled in Rust, giving you massive performance without needing to learn a low-level language.
- Parallel Execution: Tiders doesn't wait around. While it's writing the last batch of data to your database, it’s already fetching and processing the next one in the background.

## Data Providers

Connect to the best data sources in the industry without vendor lock-in. Tiders decouples the provider from the destination, giving you a consistent way to fetch data.

| Provider | Ethereum (EVM) | Solana (SVM) |
|---|---|---|
| [HyperSync](https://docs.envio.dev/docs/HyperSync/overview) | ✅ | ❌ |
| [SQD](https://docs.sqd.ai/) | ✅ | ✅ |
| RPC | ✅ | ❌ |

## Transformations

Leverage the tools you already know. Tiders automatically convert data batch-by-batch into your engine's native format, allowing for seamless, custom transformations on every incoming increment immediately before it is written.


| Engine | Data format in your function | Best for |
|---|---|---|
| **Polars** | `Dict[str, pl.DataFrame]` | Fast columnar operations, expressive API |
| **Pandas** | `Dict[str, pd.DataFrame]` | Familiar API, complex row-level operations |
| **DataFusion** | `Dict[str, datafusion.DataFrame]` | SQL-based transformations, lazy evaluation |
| **PyArrow** | `Dict[str, pa.Table]` | Zero-copy, direct Arrow manipulation |

## Supported Output Formats

Whether local or a production-grade data lake, Tiders handles the schema mapping and batch-loading to your destination of choice.

| Writer | Format |
|---|---|
| **DuckDB** | DuckDB database |
| **ClickHouse** | ClickHouse |
| **Iceberg** | Apache Iceberg |
| **Delta Lake** | Delta Lake |
| **PyArrow Dataset** | Parquet files |
| **PostgreSQL** | PostgreSQL |
| **CSV** | CSV files |

## Usage examples

- [Examples](examples)

## Logging

Python code uses the standard `logging` module of python, so it can be configured according to [python docs](https://docs.python.org/3/library/logging.html).

Set `RUST_LOG` environment variable according to [env_logger docs](https://docs.rs/env_logger/latest/env_logger/#enabling-logging) in order to see logs from rust modules.

To run an example with trace level logging for rust modules:
```
RUST_LOG=trace uv run examples/path/to/my/example
```

## Development

Tiders is composed of some repositories. 3 owned ones.

<img src="/resources/tiders_architecture.png" alt="Tiders-architecture" width="1000">

This repo uses `uv` for development. Clone all three projects side by side:

```bash
git clone https://github.com/yulesa/tiders.git
git clone https://github.com/yulesa/tiders-core.git
git clone https://github.com/yulesa/tiders-rpc-client.git
```

### Local development with tiders-core

Configure `tiders` to use your local `tiders-core` Python package by adding to `pyproject.toml`:

```toml
[tool.uv.sources]
tiders-core = { path = "../tiders-core/python", editable = true }
```

Then sync the environment:

```bash
cd tiders
uv sync
```

For full instructions including building `tiders-core` and `tiders-rpc-client` from source, see the [Development Setup](https://yulesa.github.io/tiders-docs/getting_started/development_setup.html) docs.

Core libraries we use for ingesting/decoding/validating/transforming blockchain data are implemented in [tiders-core](https://github.com/yulesa/tiders-core) repo.

## Acknowledgements

Tiders is a fork of [Cherry](https://github.com/steelcake/cherry) and [cherry-core](https://github.com/steelcake/cherry-core), a blockchain data pipeline framework built by the [SteelCake](https://github.com/steelcake) team. Cherry laid the architectural foundation that Tiders builds upon, and we're grateful for their work and the open-source spirit that made this continuation possible.

## License

Licensed under either of

 * Apache License, Version 2.0
   ([LICENSE-APACHE](LICENSE-APACHE) or http://www.apache.org/licenses/LICENSE-2.0)
 * MIT license
   ([LICENSE-MIT](LICENSE-MIT) or http://opensource.org/licenses/MIT)

at your option.

## Contribution

Unless you explicitly state otherwise, any contribution intentionally submitted
for inclusion in the work by you, as defined in the Apache-2.0 license, shall be
dual licensed as above, without any additional terms or conditions.