Metadata-Version: 2.2
Name: wids
Version: 0.1.7
Summary: A Python library for managing and processing dataset archives.
Author-email: Thomas Breuel <tmbdev@gmail.com>
License: BSD-3-Clause
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: braceexpand
Requires-Dist: numpy
Requires-Dist: pyyaml
Provides-Extra: dev
Requires-Dist: autoflake; extra == "dev"
Requires-Dist: bandit; extra == "dev"
Requires-Dist: black[jupyter]; extra == "dev"
Requires-Dist: build; extra == "dev"
Requires-Dist: bump2version; extra == "dev"
Requires-Dist: flake8; extra == "dev"
Requires-Dist: imageio; extra == "dev"
Requires-Dist: isort; extra == "dev"
Requires-Dist: jupyter; extra == "dev"
Requires-Dist: jupyterlab; extra == "dev"
Requires-Dist: lmdb; extra == "dev"
Requires-Dist: matplotlib; extra == "dev"
Requires-Dist: mkdocs; extra == "dev"
Requires-Dist: mkdocs-autorefs; extra == "dev"
Requires-Dist: mkdocs-jupyter; extra == "dev"
Requires-Dist: mkdocs-material; extra == "dev"
Requires-Dist: mkdocs-material-extensions; extra == "dev"
Requires-Dist: mkdocs-minify-plugin; extra == "dev"
Requires-Dist: mkdocstrings; extra == "dev"
Requires-Dist: mkdocstrings-python; extra == "dev"
Requires-Dist: msgpack; extra == "dev"
Requires-Dist: mypy; extra == "dev"
Requires-Dist: nbconvert; extra == "dev"
Requires-Dist: notebook; extra == "dev"
Requires-Dist: papermill; extra == "dev"
Requires-Dist: pdm; extra == "dev"
Requires-Dist: Pillow; extra == "dev"
Requires-Dist: pre-commit; extra == "dev"
Requires-Dist: pydocstyle; extra == "dev"
Requires-Dist: pytest; extra == "dev"
Requires-Dist: pytest-cov; extra == "dev"
Requires-Dist: pytorch_lightning; extra == "dev"
Requires-Dist: ray[default,tune]; extra == "dev"
Requires-Dist: ruff; extra == "dev"
Requires-Dist: scipy; extra == "dev"
Requires-Dist: setuptools; extra == "dev"
Requires-Dist: torch; extra == "dev"
Requires-Dist: torchvision; extra == "dev"
Requires-Dist: transformers; extra == "dev"
Requires-Dist: twine; extra == "dev"
Requires-Dist: typer; extra == "dev"
Requires-Dist: wheel; extra == "dev"
Requires-Dist: icecream; extra == "dev"

# The Web Indexed Datasets Library

The Web Indexed Datasets Library (`wids`) is a library for reading
WebDataset datasets in PyTorch with indexed (random) access.  WebDataset
datasets are simply collections of standard POSIX tar files containing
training samples.

The `wids` library is a drop-in replacement for other indexed datasets
in Python.  Despite being indexed and random access, it can work
with datasets larger than local disk storage and performs incremental
downloads.  Under the covers, it uses a very fast memory mapped tar
file reader.
