Metadata-Version: 2.1
Name: cheesechaser
Version: 0.1.1
Summary: Swiftly get tons of images from indexed tars on Huggingface.
Home-page: https://github.com/deepghs/cheesechaser
Author: narugo1992
Author-email: narugo992@gmail.com
License: Apache License, Version 2.0
Keywords: Utilities of images.
Classifier: Development Status :: 5 - Production/Stable
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Programming Language :: Python
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.8
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Requires-Python: >=3.8
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: hfutils>=0.4.0
Requires-Dist: hbutils>=0.9.0
Requires-Dist: huggingface_hub>=0.22
Requires-Dist: tqdm
Requires-Dist: requests
Requires-Dist: click>=7
Requires-Dist: pillow
Requires-Dist: httpx[http2]
Requires-Dist: random_user_agent
Requires-Dist: pandas
Requires-Dist: pyrate_limiter
Requires-Dist: pyarrow
Provides-Extra: test
Requires-Dist: coverage>=5; extra == "test"
Requires-Dist: mock>=4.0.3; extra == "test"
Requires-Dist: flake8~=3.5; extra == "test"
Requires-Dist: testfixtures>=6.18.5; extra == "test"
Requires-Dist: pytest~=6.2.5; extra == "test"
Requires-Dist: pytest-cov~=3.0.0; extra == "test"
Requires-Dist: pytest-mock~=3.6.1; extra == "test"
Requires-Dist: pytest-xdist>=1.34.0; extra == "test"
Requires-Dist: pytest-rerunfailures~=10.2; extra == "test"
Requires-Dist: pytest-timeout~=2.0.2; extra == "test"
Requires-Dist: pytest-benchmark~=3.4.0; extra == "test"
Requires-Dist: easydict<2,>=1.7; extra == "test"
Requires-Dist: testtools>=2; extra == "test"
Requires-Dist: where>=1.0.2; extra == "test"
Requires-Dist: responses>=0.20.0; extra == "test"
Requires-Dist: natsort; extra == "test"
Requires-Dist: pytest-image-diff>=0.0.11; extra == "test"
Requires-Dist: dghs-imgutils>=0.4.8; extra == "test"
Provides-Extra: doc
Requires-Dist: Jinja2>=3.0.0; extra == "doc"
Requires-Dist: sphinx>=3.2.0; extra == "doc"
Requires-Dist: sphinx_rtd_theme>=0.4.3; extra == "doc"
Requires-Dist: enum_tools>=0.9.0; extra == "doc"
Requires-Dist: sphinx-toolbox; extra == "doc"
Requires-Dist: plantumlcli>=0.0.2; extra == "doc"
Requires-Dist: packaging; extra == "doc"
Requires-Dist: sphinx-multiversion>=0.2.4; extra == "doc"
Requires-Dist: where>=1.0.2; extra == "doc"
Requires-Dist: easydict<2,>=1.7; extra == "doc"
Requires-Dist: responses>=0.20.0; extra == "doc"
Requires-Dist: natsort; extra == "doc"
Requires-Dist: nbsphinx>=0.8.8; extra == "doc"
Requires-Dist: ipython>=7.16.3; extra == "doc"
Requires-Dist: psutil>=5.8.0; extra == "doc"
Requires-Dist: ipykernel>=6.15; extra == "doc"
Requires-Dist: py-cpuinfo>=8.0.0; extra == "doc"
Requires-Dist: click>=7.0.0; extra == "doc"
Requires-Dist: pandas; extra == "doc"
Provides-Extra: transfer
Requires-Dist: hf-transfer; extra == "transfer"

# cheesechaser

[![PyPI](https://img.shields.io/pypi/v/cheesechaser)](https://pypi.org/project/cheesechaser/)
![PyPI - Python Version](https://img.shields.io/pypi/pyversions/cheesechaser)
![Loc](https://img.shields.io/endpoint?url=https://gist.githubusercontent.com/narugo1992/eedf334ff9d7ff02e7ec9535e43a1faa/raw/loc.json)
![Comments](https://img.shields.io/endpoint?url=https://gist.githubusercontent.com/narugo1992/eedf334ff9d7ff02e7ec9535e43a1faa/raw/comments.json)

[![Code Test](https://github.com/deepghs/cheesechaser/workflows/Code%20Test/badge.svg)](https://github.com/deepghs/cheesechaser/actions?query=workflow%3A%22Code+Test%22)
[![Package Release](https://github.com/deepghs/cheesechaser/workflows/Package%20Release/badge.svg)](https://github.com/deepghs/cheesechaser/actions?query=workflow%3A%22Package+Release%22)
[![codecov](https://codecov.io/gh/deepghs/cheesechaser/branch/main/graph/badge.svg?token=XJVDP4EFAT)](https://codecov.io/gh/deepghs/cheesechaser)

[![Discord](https://img.shields.io/discord/1157587327879745558?style=social&logo=discord&link=https%3A%2F%2Fdiscord.gg%2FTwdHJ42N72)](https://discord.gg/TwdHJ42N72)
![GitHub Org's stars](https://img.shields.io/github/stars/deepghs)
[![GitHub stars](https://img.shields.io/github/stars/deepghs/cheesechaser)](https://github.com/deepghs/cheesechaser/stargazers)
[![GitHub forks](https://img.shields.io/github/forks/deepghs/cheesechaser)](https://github.com/deepghs/cheesechaser/network)
![GitHub commit activity](https://img.shields.io/github/commit-activity/m/deepghs/cheesechaser)
[![GitHub issues](https://img.shields.io/github/issues/deepghs/cheesechaser)](https://github.com/deepghs/cheesechaser/issues)
[![GitHub pulls](https://img.shields.io/github/issues-pr/deepghs/cheesechaser)](https://github.com/deepghs/cheesechaser/pulls)
[![Contributors](https://img.shields.io/github/contributors/deepghs/cheesechaser)](https://github.com/deepghs/cheesechaser/graphs/contributors)
[![GitHub license](https://img.shields.io/github/license/deepghs/cheesechaser)](https://github.com/deepghs/cheesechaser/blob/master/LICENSE)

Swiftly get tons of images from indexed tars on Huggingface

## Installation

```shell
pip install cheesechaser
```

## How this library works

This library is based on the mirror datasets on huggingface.

For the Gelbooru mirror dataset repository, such
as [deepghs/gelbooru_full](https://huggingface.co/datasets/deepghs/gelbooru_full), each data packet includes a tar
archive file and a corresponding JSON index file. The JSON index file contains detailed information about the files
within the tar archive, including file size, offset, and file fingerprint.

The files in this dataset repository are organized according to a fixed pattern based on their IDs. For example, a file
with the ID 114514 will have a modulus result of 4514 when divided by 10000. Consequently, it is stored
in `images/4/0514.tar`.

Utilizing the quick download feature
from [hfutils.index](https://deepghs.github.io/hfutils/main/api_doc/index/index.html), users can instantly access
individual files. Since the download service is provided through Huggingface's LFS service and not the original website
or an image CDN, there is no risk of IP or account blocking. **The only limitations to your download speed are your
network bandwidth and disk read/write speeds.**

This efficient system ensures a seamless and reliable access to the dataset without any restrictions.

## Batch Download Images

* Danbooru

```python
from cheesechaser.datapool import DanbooruNewestDataPool

pool = DanbooruNewestDataPool()

# download danbooru #2010000-2010300, to directory /data/exp2
pool.batch_download_to_directory(
    resource_ids=range(2010000, 2010300),
    dst_dir='/data/exp2',
    max_workers=12,
)
```

* Danbooru With Tags Query

```python
from cheesechaser.datapool import DanbooruNewestDataPool
from cheesechaser.query import DanbooruIdQuery

pool = DanbooruNewestDataPool()
my_waifu_ids = DanbooruIdQuery(['surtr_(arknights)', 'solo'])

# download danbooru images with surtr+solo, to directory /data/exp2_surtr
pool.batch_download_to_directory(
    resource_ids=my_waifu_ids,
    dst_dir='/data/exp2_surtr',
    max_workers=12,
)
```

* Konachan (Gated dataset, you should be granted first and set `HF_TOKEN` environment variable)

```python
from cheesechaser.datapool import KonachanDataPool

pool = KonachanDataPool()

# download konachan #210000-210300, to directory /data/exp2
pool.batch_download_to_directory(
    resource_ids=range(210000, 210300),
    dst_dir='/data/exp2',
    max_workers=12,
)
```

* Civitai (this mirror repository on hf is private for now, you have to use hf token of an authorized account)

```python
from cheesechaser.datapool import CivitaiDataPool

pool = CivitaiDataPool()

# download civitai #7810000-7810300, to directory /data/exp2
# should contain one image and one json metadata file
pool.batch_download_to_directory(
    resource_ids=range(7810000, 7810300),
    dst_dir='/data/exp2',
    max_workers=12,
)
```

More supported:

* `RealbooruDataPool` (Gated Dataset)
* `ThreedbooruDataPool` (Gated Dataset)
* `FancapsDataPool` (Gated Dataset)
* `BangumiBaseDataPool` (Gated Dataset)
* `AnimePicturesDataPool` (Gated Dataset)
* `KonachanDataPool` (Gated Dataset)
* `YandeDataPool` (Gated Dataset)
* `ZerochanDataPool` (Gated Dataset)
* `GelbooruDataPool` and `GelbooruWebpDataPool` (Gated Dataset)
* `DanbooruNewestDataPool` and `DanbooruNewestWebpDataPool`

## Batch Retrieving Images

```python
from itertools import islice

from cheesechaser.datapool import DanbooruNewestDataPool
from cheesechaser.pipe import SimpleImagePipe, PipeItem

pool = DanbooruNewestDataPool()
pipe = SimpleImagePipe(pool)

# select from danbooru 7349990-7359990
ids = range(7349990, 7359990)
with pipe.batch_retrieve(ids) as session:
    # only need 20 images
    for i, item in enumerate(islice(session, 20)):
        item: PipeItem
        print(i, item)

```
