This file is a merged representation of the entire codebase, combining all repository files into a single document.
Generated by Repomix on: 2025-02-20 17:59:38

# File Summary

## Purpose:

This file contains a packed representation of the entire repository's contents.
It is designed to be easily consumable by AI systems for analysis, code review,
or other automated processes.

## File Format:

The content is organized as follows:
1. This summary section
2. Repository information
3. Repository structure
4. Multiple file entries, each consisting of:
   a. A header with the file path (## File: path/to/file)
   b. The full contents of the file in a code block

## Usage Guidelines:

- This file should be treated as read-only. Any changes should be made to the
  original repository files, not this packed version.
- When processing this file, use the file path to distinguish
  between different files in the repository.
- Be aware that this file may contain sensitive information. Handle it with
  the same level of security as you would the original repository.

## Notes:

- Some files may have been excluded based on .gitignore rules and Repomix's
  configuration.
- Binary files are not included in this packed representation. Please refer to
  the Repository Structure section for a complete list of file paths, including
  binary files.

## Additional Information:

For more information about Repomix, visit: https://github.com/andersonby/python-repomix


# Repository Structure

```
LICENSE
pyproject.toml
CODE_OF_CONDUCT.md
scripts
  gen_credits.py
  make
  get_version.py
  gen_ref_nav.py
  make.py
src
  unibox
    unibox.py
    cli.py
    nb_helpers
      ipython_utils.py
      uni_peeker.py
      __init__.py
    __main__.py
    loaders
      base_loader.py
      toml_loader.py
      csv_loader.py
      parquet_loader.py
      loader_router.py
      txt_loader.py
      README.md
      jsonl_loader.py
      image_loder.py
      yaml_loader.py
      __init__.py
      json_loader.py
    debug.py
    py.typed
    backends
      hf_api_backend.py
      backend_router.py
      README.md
      local_backend.py
      __init__.py
      hf_datasets_backend.py
      s3_backend.py
      base_backend.py
      hf_router_backend.py
    utils
      utils.py
      llm_api.py
      s3_client.py
      README.md
      globals.py
      logger.py
      __init__.py
      constants.py
    __init__.py
mkdocs.yml
.github
  ISSUE_TEMPLATE
    3-docs.md
    1-bug.md
    config.yml
    4-change.md
    2-feature.md
  FUNDING.yml
  workflows
    release.yml
    ci.yml
.copier-answers.yml
Makefile
README.md
CONTRIBUTING.md
duties.py
.gitignore
config
  coverage.ini
  mypy.ini
  vscode
    launch.json
    tasks.json
    settings.json
  pytest.ini
  git-changelog.toml
  ruff.toml
tests
  test_cli.py
  README.md
  conftest.py
  test_files
    sample.toml
    sample.parquet
    sample.json
    sample.txt
    sample.yaml
    sample.jsonl
  backends
    test_local_backend.py
    test_s3_backend.py
    __init__.py
  test_loaders.py
  __init__.py
.envrc
```

# Repository Files


## LICENSE

- Characters: 1064
- Tokens: 0

```text
MIT License

Copyright (c) 2025 trojblue

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
```

## pyproject.toml

- Characters: 3015
- Tokens: 0

```text
[build-system]
requires = ["pdm-backend"]
build-backend = "pdm.backend"

[project]
name = "unibox"
description = "unibox provides unified interface for common file operations"
authors = [{name = "trojblue", email = "trojblue@gmail.com"}]
license = {text = "MIT"}
readme = "README.md"
requires-python = ">=3.10"
keywords = []
# version="0.5.2"  # Remove local suffix
dynamic = ["version"]
classifiers = [
    "Development Status :: 4 - Beta",
    "Intended Audience :: Developers",
    "Programming Language :: Python",
    "Programming Language :: Python :: 3",
    "Programming Language :: Python :: 3 :: Only",
    "Programming Language :: Python :: 3.10",
    "Programming Language :: Python :: 3.11",
    "Programming Language :: Python :: 3.12",
    "Programming Language :: Python :: 3.13",
    "Topic :: Documentation",
    "Topic :: Software Development",
    "Topic :: Utilities",
    "Typing :: Typed",
]
dependencies = [
    "boto3>=1.35.91",
    "colorama>=0.4.6",
    "colorlog>=6.9.0",
    "datasets>=3.2.0",
    "orjson>=3.10.13",
    "pandas[parquet]>=2.2.3",
    "pillow>=11.1.0",
    "tomli-w>=1.2.0",
    "tqdm>=4.67.1",
]

[project.urls]
Homepage = "https://trojblue.github.io/unibox"
Documentation = "https://trojblue.github.io/unibox"
Changelog = "https://trojblue.github.io/unibox/changelog"
Repository = "https://github.com/trojblue/unibox"
Issues = "https://github.com/trojblue/unibox/issues"
Discussions = "https://github.com/trojblue/unibox/discussions"
Gitter = "https://gitter.im/unibox/community"
Funding = "https://github.com/sponsors/trojblue"

[project.scripts]
unibox = "unibox.cli:main"

[tool.pdm.version]
source = "call"
getter = "scripts.get_version:get_version"

[tool.pdm.build]
# Include as much as possible in the source distribution, to help redistributors.
excludes = ["**/.pytest_cache"]
source-includes = [
    "config",
    "docs",
    "scripts",
    "share",
    "tests",
    "duties.py",
    "mkdocs.yml",
    "*.md",
    "LICENSE",
]

[tool.pdm.build.wheel-data]
# Manual pages can be included in the wheel.
# Depending on the installation tool, they will be accessible to users.
# pipx supports it, uv does not yet, see https://github.com/astral-sh/uv/issues/4731.
data = [
    {path = "share/**/*", relative-to = "."},
]

[dependency-groups]
dev = [
    # maintenance
    "build>=1.2",
    "git-changelog>=2.5",
    "twine>=5.1",

    # ci
    "duty>=1.4",
    "ruff>=0.4",
    "pytest>=8.2",
    "pytest-cov>=5.0",
    "pytest-randomly>=3.15",
    "pytest-xdist>=3.6",
    "mypy>=1.10",
    "types-markdown>=3.6",
    "types-pyyaml>=6.0",

    # docs
    "black>=24.4",
    "markdown-callouts>=0.4",
    "markdown-exec>=1.8",
    "mkdocs>=1.6",
    "mkdocs-coverage>=1.0",
    "mkdocs-gen-files>=0.5",
    "mkdocs-git-revision-date-localized-plugin>=1.2",
    "mkdocs-literate-nav>=0.6",
    "mkdocs-material>=9.5",
    "mkdocs-minify-plugin>=0.8",
    "mkdocstrings[python]>=0.25",
    # YORE: EOL 3.10: Remove line.
    "tomli>=2.0; python_version < '3.11'",
]
```

## CODE_OF_CONDUCT.md

- Characters: 5482
- Tokens: 0

```markdown
# Contributor Covenant Code of Conduct

## Our Pledge

We as members, contributors, and leaders pledge to make participation in our
community a harassment-free experience for everyone, regardless of age, body
size, visible or invisible disability, ethnicity, sex characteristics, gender
identity and expression, level of experience, education, socio-economic status,
nationality, personal appearance, race, caste, color, religion, or sexual
identity and orientation.

We pledge to act and interact in ways that contribute to an open, welcoming,
diverse, inclusive, and healthy community.

## Our Standards

Examples of behavior that contributes to a positive environment for our
community include:

* Demonstrating empathy and kindness toward other people
* Being respectful of differing opinions, viewpoints, and experiences
* Giving and gracefully accepting constructive feedback
* Accepting responsibility and apologizing to those affected by our mistakes,
  and learning from the experience
* Focusing on what is best not just for us as individuals, but for the overall
  community

Examples of unacceptable behavior include:

* The use of sexualized language or imagery, and sexual attention or advances of
  any kind
* Trolling, insulting or derogatory comments, and personal or political attacks
* Public or private harassment
* Publishing others' private information, such as a physical or email address,
  without their explicit permission
* Other conduct which could reasonably be considered inappropriate in a
  professional setting

## Enforcement Responsibilities

Community leaders are responsible for clarifying and enforcing our standards of
acceptable behavior and will take appropriate and fair corrective action in
response to any behavior that they deem inappropriate, threatening, offensive,
or harmful.

Community leaders have the right and responsibility to remove, edit, or reject
comments, commits, code, wiki edits, issues, and other contributions that are
not aligned to this Code of Conduct, and will communicate reasons for moderation
decisions when appropriate.

## Scope

This Code of Conduct applies within all community spaces, and also applies when
an individual is officially representing the community in public spaces.
Examples of representing our community include using an official e-mail address,
posting via an official social media account, or acting as an appointed
representative at an online or offline event.

## Enforcement

Instances of abusive, harassing, or otherwise unacceptable behavior may be
reported to the community leaders responsible for enforcement at
trojblue@gmail.com.
All complaints will be reviewed and investigated promptly and fairly.

All community leaders are obligated to respect the privacy and security of the
reporter of any incident.

## Enforcement Guidelines

Community leaders will follow these Community Impact Guidelines in determining
the consequences for any action they deem in violation of this Code of Conduct:

### 1. Correction

**Community Impact**: Use of inappropriate language or other behavior deemed
unprofessional or unwelcome in the community.

**Consequence**: A private, written warning from community leaders, providing
clarity around the nature of the violation and an explanation of why the
behavior was inappropriate. A public apology may be requested.

### 2. Warning

**Community Impact**: A violation through a single incident or series of
actions.

**Consequence**: A warning with consequences for continued behavior. No
interaction with the people involved, including unsolicited interaction with
those enforcing the Code of Conduct, for a specified period of time. This
includes avoiding interactions in community spaces as well as external channels
like social media. Violating these terms may lead to a temporary or permanent
ban.

### 3. Temporary Ban

**Community Impact**: A serious violation of community standards, including
sustained inappropriate behavior.

**Consequence**: A temporary ban from any sort of interaction or public
communication with the community for a specified period of time. No public or
private interaction with the people involved, including unsolicited interaction
with those enforcing the Code of Conduct, is allowed during this period.
Violating these terms may lead to a permanent ban.

### 4. Permanent Ban

**Community Impact**: Demonstrating a pattern of violation of community
standards, including sustained inappropriate behavior, harassment of an
individual, or aggression toward or disparagement of classes of individuals.

**Consequence**: A permanent ban from any sort of public interaction within the
community.

## Attribution

This Code of Conduct is adapted from the [Contributor Covenant][homepage],
version 2.1, available at
[https://www.contributor-covenant.org/version/2/1/code_of_conduct.html][v2.1].

Community Impact Guidelines were inspired by
[Mozilla's code of conduct enforcement ladder][Mozilla CoC].

For answers to common questions about this code of conduct, see the FAQ at
[https://www.contributor-covenant.org/faq][FAQ]. Translations are available at
[https://www.contributor-covenant.org/translations][translations].

[homepage]: https://www.contributor-covenant.org
[v2.1]: https://www.contributor-covenant.org/version/2/1/code_of_conduct.html
[Mozilla CoC]: https://github.com/mozilla/diversity
[FAQ]: https://www.contributor-covenant.org/faq
[translations]: https://www.contributor-covenant.org/translations
```

## scripts/gen_credits.py

- Characters: 6592
- Tokens: 0

```python
"""Script to generate the project's credits."""

from __future__ import annotations

import os
import sys
from collections import defaultdict
from collections.abc import Iterable
from importlib.metadata import distributions
from itertools import chain
from pathlib import Path
from textwrap import dedent
from typing import Union

from jinja2 import StrictUndefined
from jinja2.sandbox import SandboxedEnvironment
from packaging.requirements import Requirement

# YORE: EOL 3.10: Replace block with line 2.
if sys.version_info >= (3, 11):
    import tomllib
else:
    import tomli as tomllib

project_dir = Path(os.getenv("MKDOCS_CONFIG_DIR", "."))
with project_dir.joinpath("pyproject.toml").open("rb") as pyproject_file:
    pyproject = tomllib.load(pyproject_file)
project = pyproject["project"]
project_name = project["name"]
devdeps = [dep for dep in pyproject["dependency-groups"]["dev"] if not dep.startswith("-e")]

PackageMetadata = dict[str, Union[str, Iterable[str]]]
Metadata = dict[str, PackageMetadata]


def _merge_fields(metadata: dict) -> PackageMetadata:
    fields = defaultdict(list)
    for header, value in metadata.items():
        fields[header.lower()].append(value.strip())
    return {
        field: value if len(value) > 1 or field in ("classifier", "requires-dist") else value[0]
        for field, value in fields.items()
    }


def _norm_name(name: str) -> str:
    return name.replace("_", "-").replace(".", "-").lower()


def _requirements(deps: list[str]) -> dict[str, Requirement]:
    return {_norm_name((req := Requirement(dep)).name): req for dep in deps}


def _extra_marker(req: Requirement) -> str | None:
    if not req.marker:
        return None
    try:
        return next(marker[2].value for marker in req.marker._markers if getattr(marker[0], "value", None) == "extra")
    except StopIteration:
        return None


def _get_metadata() -> Metadata:
    metadata = {}
    for pkg in distributions():
        name = _norm_name(pkg.name)  # type: ignore[attr-defined,unused-ignore]
        metadata[name] = _merge_fields(pkg.metadata)  # type: ignore[arg-type]
        metadata[name]["spec"] = set()
        metadata[name]["extras"] = set()
        metadata[name].setdefault("summary", "")
        _set_license(metadata[name])
    return metadata


def _set_license(metadata: PackageMetadata) -> None:
    license_field = metadata.get("license-expression", metadata.get("license", ""))
    license_name = license_field if isinstance(license_field, str) else " + ".join(license_field)
    check_classifiers = license_name in ("UNKNOWN", "Dual License", "") or license_name.count("\n")
    if check_classifiers:
        license_names = []
        for classifier in metadata["classifier"]:
            if classifier.startswith("License ::"):
                license_names.append(classifier.rsplit("::", 1)[1].strip())
        license_name = " + ".join(license_names)
    metadata["license"] = license_name or "?"


def _get_deps(base_deps: dict[str, Requirement], metadata: Metadata) -> Metadata:
    deps = {}
    for dep_name, dep_req in base_deps.items():
        if dep_name not in metadata or dep_name == "unibox":
            continue
        metadata[dep_name]["spec"] |= {str(spec) for spec in dep_req.specifier}  # type: ignore[operator]
        metadata[dep_name]["extras"] |= dep_req.extras  # type: ignore[operator]
        deps[dep_name] = metadata[dep_name]

    again = True
    while again:
        again = False
        for pkg_name in metadata:
            if pkg_name in deps:
                for pkg_dependency in metadata[pkg_name].get("requires-dist", []):
                    requirement = Requirement(pkg_dependency)
                    dep_name = _norm_name(requirement.name)
                    extra_marker = _extra_marker(requirement)
                    if (
                        dep_name in metadata
                        and dep_name not in deps
                        and dep_name != project["name"]
                        and (not extra_marker or extra_marker in deps[pkg_name]["extras"])
                    ):
                        metadata[dep_name]["spec"] |= {str(spec) for spec in requirement.specifier}  # type: ignore[operator]
                        deps[dep_name] = metadata[dep_name]
                        again = True

    return deps


def _render_credits() -> str:
    metadata = _get_metadata()
    dev_dependencies = _get_deps(_requirements(devdeps), metadata)
    prod_dependencies = _get_deps(
        _requirements(
            chain(  # type: ignore[arg-type]
                project.get("dependencies", []),
                chain(*project.get("optional-dependencies", {}).values()),
            ),
        ),
        metadata,
    )

    template_data = {
        "project_name": project_name,
        "prod_dependencies": sorted(prod_dependencies.values(), key=lambda dep: str(dep["name"]).lower()),
        "dev_dependencies": sorted(dev_dependencies.values(), key=lambda dep: str(dep["name"]).lower()),
        "more_credits": "",
    }
    template_text = dedent(
        """
        # Credits

        These projects were used to build *{{ project_name }}*. **Thank you!**

        [Python](https://www.python.org/) |
        [uv](https://github.com/astral-sh/uv) |
        [copier-uv](https://github.com/pawamoy/copier-uv)

        {% macro dep_line(dep) -%}
        [{{ dep.name }}](https://pypi.org/project/{{ dep.name }}/) | {{ dep.summary }} | {{ ("`" ~ dep.spec|sort(reverse=True)|join(", ") ~ "`") if dep.spec else "" }} | `{{ dep.version }}` | {{ dep.license }}
        {%- endmacro %}

        {% if prod_dependencies -%}
        ### Runtime dependencies

        Project | Summary | Version (accepted) | Version (last resolved) | License
        ------- | ------- | ------------------ | ----------------------- | -------
        {% for dep in prod_dependencies -%}
        {{ dep_line(dep) }}
        {% endfor %}

        {% endif -%}
        {% if dev_dependencies -%}
        ### Development dependencies

        Project | Summary | Version (accepted) | Version (last resolved) | License
        ------- | ------- | ------------------ | ----------------------- | -------
        {% for dep in dev_dependencies -%}
        {{ dep_line(dep) }}
        {% endfor %}

        {% endif -%}
        {% if more_credits %}**[More credits from the author]({{ more_credits }})**{% endif %}
        """,
    )
    jinja_env = SandboxedEnvironment(undefined=StrictUndefined)
    return jinja_env.from_string(template_text).render(**template_data)


print(_render_credits())
```

## scripts/make

- Characters: 6284
- Tokens: 0

```text
#!/usr/bin/env python3
"""Management commands."""

from __future__ import annotations

import os
import shutil
import subprocess
import sys
from contextlib import contextmanager
from pathlib import Path
from textwrap import dedent
from typing import TYPE_CHECKING, Any

if TYPE_CHECKING:
    from collections.abc import Iterator


PYTHON_VERSIONS = os.getenv("PYTHON_VERSIONS", "3.10 3.11 3.12 3.13").split()


def shell(cmd: str, *, capture_output: bool = False, **kwargs: Any) -> str | None:
    """Run a shell command."""
    if capture_output:
        return subprocess.check_output(cmd, shell=True, text=True, **kwargs)  # noqa: S602
    subprocess.run(cmd, shell=True, check=True, stderr=subprocess.STDOUT, **kwargs)  # noqa: S602
    return None


@contextmanager
def environ(**kwargs: str) -> Iterator[None]:
    """Temporarily set environment variables."""
    original = dict(os.environ)
    os.environ.update(kwargs)
    try:
        yield
    finally:
        os.environ.clear()
        os.environ.update(original)


def uv_install(venv: Path) -> None:
    """Install dependencies using uv."""
    with environ(UV_PROJECT_ENVIRONMENT=str(venv), PYO3_USE_ABI3_FORWARD_COMPATIBILITY="1"):
        if "CI" in os.environ:
            shell("uv sync --no-editable")
        else:
            shell("uv sync")


def setup() -> None:
    """Setup the project."""
    if not shutil.which("uv"):
        raise ValueError("make: setup: uv must be installed, see https://github.com/astral-sh/uv")

    print("Installing dependencies (default environment)")
    default_venv = Path(".venv")
    if not default_venv.exists():
        shell("uv venv")
    uv_install(default_venv)

    if PYTHON_VERSIONS:
        for version in PYTHON_VERSIONS:
            print(f"\nInstalling dependencies (python{version})")
            venv_path = Path(f".venvs/{version}")
            if not venv_path.exists():
                shell(f"uv venv --python {version} {venv_path}")
            with environ(UV_PROJECT_ENVIRONMENT=str(venv_path.resolve())):
                uv_install(venv_path)


def run(version: str, cmd: str, *args: str, **kwargs: Any) -> None:
    """Run a command in a virtual environment."""
    kwargs = {"check": True, **kwargs}
    uv_run = ["uv", "run", "--no-sync"]
    if version == "default":
        with environ(UV_PROJECT_ENVIRONMENT=".venv"):
            subprocess.run([*uv_run, cmd, *args], **kwargs)  # noqa: S603, PLW1510
    else:
        with environ(UV_PROJECT_ENVIRONMENT=f".venvs/{version}", MULTIRUN="1"):
            subprocess.run([*uv_run, cmd, *args], **kwargs)  # noqa: S603, PLW1510


def multirun(cmd: str, *args: str, **kwargs: Any) -> None:
    """Run a command for all configured Python versions."""
    if PYTHON_VERSIONS:
        for version in PYTHON_VERSIONS:
            run(version, cmd, *args, **kwargs)
    else:
        run("default", cmd, *args, **kwargs)


def allrun(cmd: str, *args: str, **kwargs: Any) -> None:
    """Run a command in all virtual environments."""
    run("default", cmd, *args, **kwargs)
    if PYTHON_VERSIONS:
        multirun(cmd, *args, **kwargs)


def clean() -> None:
    """Delete build artifacts and cache files."""
    paths_to_clean = ["build", "dist", "htmlcov", "site", ".coverage*", ".pdm-build"]
    for path in paths_to_clean:
        shutil.rmtree(path, ignore_errors=True)

    cache_dirs = {".cache", ".pytest_cache", ".mypy_cache", ".ruff_cache", "__pycache__"}
    for dirpath in Path(".").rglob("*/"):
        if dirpath.parts[0] not in (".venv", ".venvs") and dirpath.name in cache_dirs:
            shutil.rmtree(dirpath, ignore_errors=True)


def vscode() -> None:
    """Configure VSCode to work on this project."""
    shutil.copytree("config/vscode", ".vscode", dirs_exist_ok=True)


def main() -> int:
    """Main entry point."""
    args = list(sys.argv[1:])
    if not args or args[0] == "help":
        if len(args) > 1:
            run("default", "duty", "--help", args[1])
        else:
            print(
                dedent(
                    """
                    Available commands
                      help                  Print this help. Add task name to print help.
                      setup                 Setup all virtual environments (install dependencies).
                      run                   Run a command in the default virtual environment.
                      multirun              Run a command for all configured Python versions.
                      allrun                Run a command in all virtual environments.
                      3.x                   Run a command in the virtual environment for Python 3.x.
                      clean                 Delete build artifacts and cache files.
                      vscode                Configure VSCode to work on this project.
                    """,
                ),
                flush=True,
            )
            if os.path.exists(".venv"):
                print("\nAvailable tasks", flush=True)
                run("default", "duty", "--list")
        return 0

    while args:
        cmd = args.pop(0)

        if cmd == "run":
            run("default", *args)
            return 0

        if cmd == "multirun":
            multirun(*args)
            return 0

        if cmd == "allrun":
            allrun(*args)
            return 0

        if cmd.startswith("3."):
            run(cmd, *args)
            return 0

        opts = []
        while args and (args[0].startswith("-") or "=" in args[0]):
            opts.append(args.pop(0))

        if cmd == "clean":
            clean()
        elif cmd == "setup":
            setup()
        elif cmd == "vscode":
            vscode()
        elif cmd == "check":
            multirun("duty", "check-quality", "check-types", "check-docs")
            run("default", "duty", "check-api")
        elif cmd in {"check-quality", "check-docs", "check-types", "test"}:
            multirun("duty", cmd, *opts)
        else:
            run("default", "duty", cmd, *opts)

    return 0


if __name__ == "__main__":
    try:
        sys.exit(main())
    except subprocess.CalledProcessError as process:
        if process.output:
            print(process.output, file=sys.stderr)
        sys.exit(process.returncode)
```

## scripts/get_version.py

- Characters: 1099
- Tokens: 0

```python
"""Get current project version from Git tags or changelog."""

import re
from contextlib import suppress
from pathlib import Path

from pdm.backend.hooks.version import SCMVersion, Version, default_version_formatter, get_version_from_scm

_root = Path(__file__).parent.parent
_changelog = _root / "CHANGELOG.md"
_changelog_version_re = re.compile(r"^## \[(\d+\.\d+\.\d+)\].*$")
_default_scm_version = SCMVersion(Version("0.0.0"), None, False, None, None)  # noqa: FBT003


def get_version() -> str:
    """Get current project version from Git tags or changelog."""
    scm_version = get_version_from_scm(_root) or _default_scm_version
    if scm_version.version <= Version("0.1"):  # Missing Git tags?
        with suppress(OSError, StopIteration):  # noqa: SIM117
            with _changelog.open("r", encoding="utf8") as file:
                match = next(filter(None, map(_changelog_version_re.match, file)))
                scm_version = scm_version._replace(version=Version(match.group(1)))
    return default_version_formatter(scm_version)


if __name__ == "__main__":
    print(get_version())
```

## scripts/gen_ref_nav.py

- Characters: 1196
- Tokens: 0

```python
"""Generate the code reference pages and navigation."""

from pathlib import Path

import mkdocs_gen_files

nav = mkdocs_gen_files.Nav()
mod_symbol = '<code class="doc-symbol doc-symbol-nav doc-symbol-module"></code>'

root = Path(__file__).parent.parent
src = root / "src"

for path in sorted(src.rglob("*.py")):
    module_path = path.relative_to(src).with_suffix("")
    doc_path = path.relative_to(src).with_suffix(".md")
    full_doc_path = Path("reference", doc_path)

    parts = tuple(module_path.parts)

    if parts[-1] == "__init__":
        parts = parts[:-1]
        doc_path = doc_path.with_name("index.md")
        full_doc_path = full_doc_path.with_name("index.md")
    elif parts[-1].startswith("_"):
        continue

    nav_parts = [f"{mod_symbol} {part}" for part in parts]
    nav[tuple(nav_parts)] = doc_path.as_posix()

    with mkdocs_gen_files.open(full_doc_path, "w") as fd:
        ident = ".".join(parts)
        fd.write(f"---\ntitle: {ident}\n---\n\n::: {ident}")

    mkdocs_gen_files.set_edit_path(full_doc_path, ".." / path.relative_to(root))

with mkdocs_gen_files.open("reference/SUMMARY.md", "w") as nav_file:
    nav_file.writelines(nav.build_literate_nav())
```

## scripts/make.py

- Characters: 6284
- Tokens: 0

```python
#!/usr/bin/env python3
"""Management commands."""

from __future__ import annotations

import os
import shutil
import subprocess
import sys
from contextlib import contextmanager
from pathlib import Path
from textwrap import dedent
from typing import TYPE_CHECKING, Any

if TYPE_CHECKING:
    from collections.abc import Iterator


PYTHON_VERSIONS = os.getenv("PYTHON_VERSIONS", "3.10 3.11 3.12 3.13").split()


def shell(cmd: str, *, capture_output: bool = False, **kwargs: Any) -> str | None:
    """Run a shell command."""
    if capture_output:
        return subprocess.check_output(cmd, shell=True, text=True, **kwargs)  # noqa: S602
    subprocess.run(cmd, shell=True, check=True, stderr=subprocess.STDOUT, **kwargs)  # noqa: S602
    return None


@contextmanager
def environ(**kwargs: str) -> Iterator[None]:
    """Temporarily set environment variables."""
    original = dict(os.environ)
    os.environ.update(kwargs)
    try:
        yield
    finally:
        os.environ.clear()
        os.environ.update(original)


def uv_install(venv: Path) -> None:
    """Install dependencies using uv."""
    with environ(UV_PROJECT_ENVIRONMENT=str(venv), PYO3_USE_ABI3_FORWARD_COMPATIBILITY="1"):
        if "CI" in os.environ:
            shell("uv sync --no-editable")
        else:
            shell("uv sync")


def setup() -> None:
    """Setup the project."""
    if not shutil.which("uv"):
        raise ValueError("make: setup: uv must be installed, see https://github.com/astral-sh/uv")

    print("Installing dependencies (default environment)")
    default_venv = Path(".venv")
    if not default_venv.exists():
        shell("uv venv")
    uv_install(default_venv)

    if PYTHON_VERSIONS:
        for version in PYTHON_VERSIONS:
            print(f"\nInstalling dependencies (python{version})")
            venv_path = Path(f".venvs/{version}")
            if not venv_path.exists():
                shell(f"uv venv --python {version} {venv_path}")
            with environ(UV_PROJECT_ENVIRONMENT=str(venv_path.resolve())):
                uv_install(venv_path)


def run(version: str, cmd: str, *args: str, **kwargs: Any) -> None:
    """Run a command in a virtual environment."""
    kwargs = {"check": True, **kwargs}
    uv_run = ["uv", "run", "--no-sync"]
    if version == "default":
        with environ(UV_PROJECT_ENVIRONMENT=".venv"):
            subprocess.run([*uv_run, cmd, *args], **kwargs)  # noqa: S603, PLW1510
    else:
        with environ(UV_PROJECT_ENVIRONMENT=f".venvs/{version}", MULTIRUN="1"):
            subprocess.run([*uv_run, cmd, *args], **kwargs)  # noqa: S603, PLW1510


def multirun(cmd: str, *args: str, **kwargs: Any) -> None:
    """Run a command for all configured Python versions."""
    if PYTHON_VERSIONS:
        for version in PYTHON_VERSIONS:
            run(version, cmd, *args, **kwargs)
    else:
        run("default", cmd, *args, **kwargs)


def allrun(cmd: str, *args: str, **kwargs: Any) -> None:
    """Run a command in all virtual environments."""
    run("default", cmd, *args, **kwargs)
    if PYTHON_VERSIONS:
        multirun(cmd, *args, **kwargs)


def clean() -> None:
    """Delete build artifacts and cache files."""
    paths_to_clean = ["build", "dist", "htmlcov", "site", ".coverage*", ".pdm-build"]
    for path in paths_to_clean:
        shutil.rmtree(path, ignore_errors=True)

    cache_dirs = {".cache", ".pytest_cache", ".mypy_cache", ".ruff_cache", "__pycache__"}
    for dirpath in Path(".").rglob("*/"):
        if dirpath.parts[0] not in (".venv", ".venvs") and dirpath.name in cache_dirs:
            shutil.rmtree(dirpath, ignore_errors=True)


def vscode() -> None:
    """Configure VSCode to work on this project."""
    shutil.copytree("config/vscode", ".vscode", dirs_exist_ok=True)


def main() -> int:
    """Main entry point."""
    args = list(sys.argv[1:])
    if not args or args[0] == "help":
        if len(args) > 1:
            run("default", "duty", "--help", args[1])
        else:
            print(
                dedent(
                    """
                    Available commands
                      help                  Print this help. Add task name to print help.
                      setup                 Setup all virtual environments (install dependencies).
                      run                   Run a command in the default virtual environment.
                      multirun              Run a command for all configured Python versions.
                      allrun                Run a command in all virtual environments.
                      3.x                   Run a command in the virtual environment for Python 3.x.
                      clean                 Delete build artifacts and cache files.
                      vscode                Configure VSCode to work on this project.
                    """,
                ),
                flush=True,
            )
            if os.path.exists(".venv"):
                print("\nAvailable tasks", flush=True)
                run("default", "duty", "--list")
        return 0

    while args:
        cmd = args.pop(0)

        if cmd == "run":
            run("default", *args)
            return 0

        if cmd == "multirun":
            multirun(*args)
            return 0

        if cmd == "allrun":
            allrun(*args)
            return 0

        if cmd.startswith("3."):
            run(cmd, *args)
            return 0

        opts = []
        while args and (args[0].startswith("-") or "=" in args[0]):
            opts.append(args.pop(0))

        if cmd == "clean":
            clean()
        elif cmd == "setup":
            setup()
        elif cmd == "vscode":
            vscode()
        elif cmd == "check":
            multirun("duty", "check-quality", "check-types", "check-docs")
            run("default", "duty", "check-api")
        elif cmd in {"check-quality", "check-docs", "check-types", "test"}:
            multirun("duty", cmd, *opts)
        else:
            run("default", "duty", cmd, *opts)

    return 0


if __name__ == "__main__":
    try:
        sys.exit(main())
    except subprocess.CalledProcessError as process:
        if process.output:
            print(process.output, file=sys.stderr)
        sys.exit(process.returncode)
```

## src/unibox/unibox.py

- Characters: 9773
- Tokens: 0

```python
# unibox.py
import os
import timeit
import warnings
from concurrent.futures import ProcessPoolExecutor, as_completed
from functools import partial
from pathlib import Path
from typing import Any, Dict, List, Optional, Union

from tqdm.auto import tqdm

from .backends.backend_router import get_backend_for_uri
from .backends.local_backend import LocalBackend
from .loaders.loader_router import get_loader_for_suffix
from .utils.globals import GLOBAL_TMP_DIR
from .utils.logger import UniLogger
from .utils.s3_client import S3Client

s3_client = S3Client()
logger = UniLogger()


def _get_type_info(obj: Any) -> str:
    obj_module, obj_name = type(obj).__module__, type(obj).__name__
    return str(obj_module), str(obj_name)


### CHANGED ###
def _parse_hf_uri(uri: str):
    # likely duplicate of hf_datasets_backend.py impl
    """Returns (repo_id, subpath) from 'hf://owner/repo/...'. If no subpath, returns ('owner/repo', '')."""
    trimmed = uri.replace("hf://", "", 1)
    parts = trimmed.split("/", 2)  # Split into at most three parts
    if len(parts) < 2:
        raise ValueError(f"Invalid Hugging Face URI format: {uri}")

    repo_id = f"{parts[0]}/{parts[1]}"  # First two parts make up the repo ID
    subpath = parts[2] if len(parts) > 2 else ""  # Remaining part is the subpath

    return repo_id, subpath


def load_file(uri: Union[str, Path], debug_print: bool = True, **kwargs) -> str:
    """Return a local file path (downloading if needed) without parsing."""
    import os
    import timeit

    start_time = timeit.default_timer()
    uri_str = str(uri)
    backend = get_backend_for_uri(uri_str)

    if isinstance(backend, LocalBackend):
        local_path = Path(uri_str).absolute()
    else:
        rand_dir = GLOBAL_TMP_DIR / str(os.getpid())
        rand_dir.mkdir(parents=True, exist_ok=True)
        local_path = backend.download(uri_str, target_dir=rand_dir)

    end_time = timeit.default_timer()
    if debug_print:
        logger.info(f'File available at "{local_path}" in {end_time - start_time:.2f}s')
    return str(local_path)


def loads(uri: Union[str, Path], file: bool = False, debug_print: bool = True, **kwargs) -> Any:
    """Loads data from a given URI.
    If file=True, just returns the local path, no parsing.
    Otherwise:
      - If HF with no subpath => treat as entire dataset => .load_dataset()
      - Else do normal "download then suffix-based loader."

    some kwargs:
        split: str = "train"  (for HF dataset load)
        streaming: bool = False  (for HF dataset load)
    """
    if file:
        return load_file(uri, debug_print=debug_print, **kwargs)

    start_time = timeit.default_timer()
    uri_str = str(uri)
    backend = get_backend_for_uri(uri_str)

    ### CHANGED - HuggingFace dataset logic ###
    if uri_str.startswith("hf://"):
        repo_id, subpath = _parse_hf_uri(uri_str)
        # If there's no subpath, that means "hf://owner/repo" => dataset
        if not subpath:
            # call backend.load_dataset
            # requires that the backend is our HF Router or old HFBackend
            # We'll assume it implements .load_dataset
            # here: split, streaming is passed into load_dataset
            res = (
                backend.ds_backend.load_dataset(repo_id, **kwargs)
                if hasattr(backend, "ds_backend")
                else backend.load_dataset(repo_id, **kwargs)
            )
            # done
            end_time = timeit.default_timer()
            if debug_print:
                mod, cls = _get_type_info(res)
                logger.info(f'{cls} LOADED from "{uri_str}" in {end_time - start_time:.2f}s')
            return res
        # else we fall back to the normal "download + suffix loader" below

    # Normal approach: download => loader => parse
    rand_dir = GLOBAL_TMP_DIR / str(os.getpid())
    rand_dir.mkdir(parents=True, exist_ok=True)
    if isinstance(backend, LocalBackend):
        local_path = Path(uri_str)
    else:
        local_path = backend.download(uri_str, target_dir=rand_dir)

    suffix = local_path.suffix.lower()
    loader = get_loader_for_suffix(suffix)
    if loader is None:
        raise ValueError(f"No loader found for suffix: {suffix}")

    res = loader.load(local_path, **kwargs)

    # if remote => remove temp
    if not isinstance(backend, LocalBackend):
        try:
            os.remove(local_path)
        except OSError:
            pass

    end_time = timeit.default_timer()
    if debug_print:
        mod, cls = _get_type_info(res)
        logger.info(f'{cls} LOADED from "{uri_str}" in {end_time - start_time:.2f}s')

    return res


def saves(data: Any, uri: Union[str, Path], debug_print: bool = True, **kwargs) -> None:
    """Saves data to local or remote. Special-case:
    - If HF with no extension => interpret as dataset push
    - Otherwise, do suffix-based local or single-file approach

    some kwargs:
        split: str = "train"  (for HF dataset push)
        private: bool = True  (for HF dataset push)
    """
    import tempfile

    start_time = timeit.default_timer()
    uri_str = str(uri)
    backend = get_backend_for_uri(uri_str)
    suffix = Path(uri_str).suffix.lower()

    ### CHANGED - HF dataset push if suffix=="" ###
    if uri_str.startswith("hf://") and suffix == "":
        # Means something like "hf://owner/repo" => push entire dataset
        if hasattr(backend, "ds_backend"):
            # If router-based, do "backend.ds_backend.data_to_hub(data, repo_id, ...)"
            repo_id, _ = _parse_hf_uri(uri_str)
            dataset_split = kwargs.get("split", "train")
            backend.ds_backend.data_to_hub(
                data,
                repo_id=repo_id,
                private=kwargs.get("private", True),
                split=dataset_split,
            )
        else:
            # old style: "backend.data_to_hub(...)"
            print("OLD STYLE HERE (canary print for debug)")
            dataset_split = kwargs.get("split", "train")
            backend.data_to_hub(data, uri_str, split=dataset_split, **kwargs)
        end_time = timeit.default_timer()
        if debug_print:
            _, data_cls = _get_type_info(data)
            logger.info(f'{data_cls} saved (HF dataset) to "{uri_str}" in {end_time - start_time:.2f}s')
        return

    # Otherwise normal suffix-based logic
    loader = get_loader_for_suffix(suffix)
    if loader is None:
        raise ValueError(f"No loader found for extension {suffix}")

    with tempfile.NamedTemporaryFile(suffix=suffix, delete=False) as tmpf:
        temp_path = Path(tmpf.name)

    try:
        loader.save(temp_path, data)
        backend.upload(temp_path, uri_str)
    finally:
        if temp_path.exists():
            temp_path.unlink()

    end_time = timeit.default_timer()
    if debug_print:
        _, data_cls = _get_type_info(data)
        logger.info(f'{data_cls} saved to "{uri_str}" in {end_time - start_time:.2f}s')


def ls(
    uri: Union[str, Path],
    exts: Optional[List[str]] = None,
    relative_unix: bool = False,
    debug_print: bool = True,
    **kwargs,
) -> list[str]:
    backend = get_backend_for_uri(str(uri))
    return backend.ls(str(uri), exts=exts, relative_unix=relative_unix, debug_print=debug_print, **kwargs)


def concurrent_loads(uris_list, num_workers=8, debug_print=True):
    results = [None] * len(uris_list)
    with ProcessPoolExecutor(max_workers=num_workers) as executor:
        partial_load = partial(loads, debug_print=False)
        future_to_idx = {executor.submit(partial_load, u): i for i, u in enumerate(uris_list)}

        if debug_print:
            futures_iter = tqdm(as_completed(future_to_idx), total=len(uris_list), desc="Loading concurrent")
        else:
            futures_iter = as_completed(future_to_idx)

        for future in futures_iter:
            idx = future_to_idx[future]
            try:
                results[idx] = future.result()
            except Exception as e:
                print(f"Exception reading {uris_list[idx]}: {e}")

    missing = sum(r is None for r in results)
    if missing > 0:
        logger.warning(f"{missing} loads returned None.")
    return results


def traverses(
    uri: Union[str, Path],
    exts: Optional[List[str]] = None,
    relative_unix: bool = False,
    debug_print: bool = True,
    **kwargs,
) -> list[str]:
    warnings.warn("`traverses()` is deprecated; use `ls()` instead.", DeprecationWarning, stacklevel=2)
    return ls(uri, exts=exts, relative_unix=relative_unix, debug_print=debug_print, **kwargs)


from .nb_helpers.uni_peeker import UniPeeker


def peeks(data: Any, n=3, console_print=False) -> Dict[str, Any]:
    peeker = UniPeeker(n, console_print)
    return peeker.peeks(data)


def gallery(
    paths: list[str],
    labels: list[str] = [],
    row_height="300px",
    num_workers=32,
    debug_print=True,
    thumbnail_size: int = 512,
):
    try:
        from .nb_helpers.ipython_utils import _gallery

        _gallery(paths, labels, row_height, num_workers, debug_print, thumbnail_size)
    except (ImportError, ModuleNotFoundError):
        print("IPython is not available. Gallery function cannot run.")


def label_gallery(
    paths: list[str],
    labels: list[str] = [],
    row_height="150px",
    num_workers=32,
    debug_print=True,
    thumbnail_size: int = 512,
):
    try:
        from .nb_helpers.ipython_utils import _label_gallery

        _label_gallery(paths, labels, row_height, num_workers, debug_print, thumbnail_size)
    except (ImportError, ModuleNotFoundError):
        print("IPython is not available. label_gallery function cannot run.")


def presigns(s3_uri: str, expiration: int = 604800) -> str:
    return s3_client.generate_presigned_uri(s3_uri, expiration=expiration)
```

## src/unibox/cli.py

- Characters: 1713
- Tokens: 0

```python
"""Module that contains the command line application."""

# Why does this file exist, and why not put this in `__main__`?
#
# You might be tempted to import things from `__main__` later,
# but that will cause problems: the code will get executed twice:
#
# - When you run `python -m unibox` python will execute
#   `__main__.py` as a script. That means there won't be any
#   `unibox.__main__` in `sys.modules`.
# - When you import `__main__` it will get executed again (as a module) because
#   there's no `unibox.__main__` in `sys.modules`.

from __future__ import annotations

import argparse
import sys
from typing import Any

from unibox import debug


class _DebugInfo(argparse.Action):
    def __init__(self, nargs: int | str | None = 0, **kwargs: Any) -> None:
        super().__init__(nargs=nargs, **kwargs)

    def __call__(self, *args: Any, **kwargs: Any) -> None:  # noqa: ARG002
        debug.print_debug_info()
        sys.exit(0)


def get_parser() -> argparse.ArgumentParser:
    """Return the CLI argument parser.

    Returns:
        An argparse parser.
    """
    parser = argparse.ArgumentParser(prog="unibox")
    parser.add_argument("-V", "--version", action="version", version=f"%(prog)s {debug.get_version()}")
    parser.add_argument("--debug-info", action=_DebugInfo, help="Print debug information.")
    return parser


def main(args: list[str] | None = None) -> int:
    """Run the main program.

    This function is executed when you type `unibox` or `python -m unibox`.

    Parameters:
        args: Arguments passed from the command line.

    Returns:
        An exit code.
    """
    parser = get_parser()
    opts = parser.parse_args(args=args)
    print(opts)
    return 0
```

## src/unibox/nb_helpers/ipython_utils.py

- Characters: 8763
- Tokens: 0

```python
from io import BytesIO

from IPython.display import HTML, Image, display

from ..unibox import concurrent_loads


def peek_df(df, n=1):
    print(df.shape)
    print(df.columns)
    display(df.head(n))


def _src_from_data(data):
    """Base64 encodes image bytes for inclusion in an HTML img element."""
    img_obj = Image(data=data)
    for bundle in img_obj._repr_mimebundle_():
        for mimetype, b64value in bundle.items():
            if mimetype.startswith("image/"):
                return f"data:{mimetype};base64,{b64value}"


def _gallery(
    paths: list[str],
    labels: list[str] = [],
    row_height="300px",
    num_workers=32,
    debug_print=True,
    thumbnail_size: int = 512,
):
    """Shows a set of images in a gallery that flexes with the width of the notebook.

    Parameters
    ----------
    paths: list of str
        Paths to images to display. Can be local paths, URLs, or S3 paths.

    row_height: str
        CSS height value to assign to all images. Set to 'auto' by default to show images
        with their native dimensions. Set to a value like '250px' to make all rows
        in the gallery equal height.

    num_workers: int
        Number of concurrent workers to load images.

    debug_print: bool
        Whether to print debug information or not.

    thumbnail_size: int (optional)
        If provided, resize images to this size using PIL's thumbnail method.
    """
    if len(paths) > 1000:
        raise ValueError("Too many images to display.")

    if len(labels) > 0 and len(labels) != len(paths):
        raise ValueError("Number of labels must match number of paths.")

    images = concurrent_loads(paths, num_workers=num_workers, debug_print=debug_print)

    figures = []
    for i, image in enumerate(images):
        try:
            if thumbnail_size > 0:
                image.thumbnail((thumbnail_size, thumbnail_size))

            # Ensure image is in RGB mode for saving as JPEG
            if image is not None and image.mode in ("RGBA", "P"):
                image = image.convert("RGB")

            buffered = BytesIO()
            image.save(buffered, format="JPEG")
            img_data = buffered.getvalue()
            src = _src_from_data(img_data)
            if len(labels) > 0:
                caption_str = labels[i]
            else:
                caption_str = paths[i].split("/")[-1]
            caption = f'<figcaption style="font-size: 0.6em">{caption_str}</figcaption>'
        except Exception as e:
            src = ""
            caption = f'<figcaption style="font-size: 0.6em; color: red;">Error loading {paths[i]}: {e}</figcaption>'

        figures.append(f"""
            <figure style="margin: 5px !important;">
              <img src="{src}" style="height: {row_height}">
              {caption}
            </figure>
        """)

    display(
        HTML(
            data=f"""
        <div style="display: flex; flex-flow: row wrap; text-align: center;">
        {"".join(figures)}
        </div>
    """,
        ),
    )


import base64


def _label_gallery(
    paths: list[str],
    labels: list[str] = [],
    row_height="150px",
    num_workers=32,
    debug_print=True,
    thumbnail_size: int = 512,
):
    """Displays images in a gallery with JavaScript-based selection.

    Parameters
    ----------
    paths: list of str
        Paths to images to display. Can be local paths or URLs.
    labels: list of str
        Labels for each image. Defaults to the filename if not provided.
    row_height: str
        CSS height value to assign to all images.
    num_workers: int
        Number of concurrent workers to load images.
    debug_print: bool
        Whether to print debug information or not.
    thumbnail_size: int
        If provided, resize images to this size using PIL's thumbnail method.
    """
    if len(paths) > 1000:
        raise ValueError("Too many images to display.")

    if len(labels) > 0 and len(labels) != len(paths):
        raise ValueError("Number of labels must match number of paths.")

    # Load images using ub.concurrent_loads()
    images = concurrent_loads(paths, num_workers=num_workers)  # list of PIL images

    # convert to rgb if necessary
    for i, img in enumerate(images):
        if img is not None and img.mode in ("RGBA", "P"):
            images[i] = img.convert("RGB")

    # Process images: resize and handle None
    processed_images = []
    for img in images:
        if img is not None:
            if thumbnail_size > 0:
                img.thumbnail((thumbnail_size, thumbnail_size))
            processed_images.append(img)
        else:
            processed_images.append(None)  # Keep None for images that failed to load

    figures = []
    for i, image in enumerate(processed_images):
        if image is not None:
            try:
                buffered = BytesIO()
                image.save(buffered, format="JPEG")
                img_data = base64.b64encode(buffered.getvalue()).decode()
                src = f"data:image/jpeg;base64,{img_data}"

                if len(labels) > 0:
                    caption_str = labels[i]
                else:
                    caption_str = paths[i].split("/")[-1]

                figures.append(f"""
                    <figure style="margin: 5px !important; width: auto; text-align: center;">
                      <img src="{src}" style="height: {row_height}; cursor: pointer; box-sizing: border-box;" onclick="selectImage({i}, this)">
                      <figcaption style="font-size: 0.6em">{caption_str}</figcaption>
                    </figure>
                """)
            except Exception as e:
                figures.append(f"""
                    <figure style="margin: 5px !important;">
                      <figcaption style="font-size: 0.6em; color: red;">Error processing image {i}: {e}</figcaption>
                    </figure>
                """)
        else:
            figures.append(f"""
                <figure style="margin: 5px !important;">
                  <figcaption style="font-size: 0.6em; color: red;">Error loading image {i}</figcaption>
                </figure>
            """)

    html_content = f"""
    <div id="gallery" style="display: flex; flex-wrap: wrap; text-align: center;">
    {"".join(figures)}
    </div>
    <button onclick="copySelection()" style="margin-top: 10px;">Copy Selection</button>

    <script>
    var selectedIndices = [];

    function updateOutput() {{
        var outputDiv = document.getElementById('output');
        if (selectedIndices.length > 0) {{
            var indicesList = '[' + selectedIndices.sort((a, b) => a - b).join(', ') + ']';
            outputDiv.innerHTML = 'Selected indices: ' + indicesList;
        }} else {{
            outputDiv.innerHTML = 'Selected indices: []';
        }}
    }}

    function selectImage(index, imgElement) {{
        var idx = selectedIndices.indexOf(index);
        if (idx > -1) {{
            // Deselect
            selectedIndices.splice(idx, 1);
            imgElement.style.outline = '';
        }} else {{
            // Select
            selectedIndices.push(index);
            imgElement.style.outline = '4px solid blue';
        }}
        updateOutput();
    }}

    function copySelection() {{
        if (selectedIndices.length > 0) {{
            var indicesList = '[' + selectedIndices.sort((a, b) => a - b).join(', ') + ']';
            if (navigator.clipboard && navigator.clipboard.writeText) {{
                navigator.clipboard.writeText(indicesList).then(function() {{
                    alert('Selected indices copied to clipboard.');
                }}, function(err) {{
                    // Fallback method
                    fallbackCopyTextToClipboard(indicesList);
                }});
            }} else {{
                // Fallback method
                fallbackCopyTextToClipboard(indicesList);
            }}
        }} else {{
            alert('No images selected.');
        }}
    }}

    function fallbackCopyTextToClipboard(text) {{
        var textArea = document.createElement("textarea");
        textArea.value = text;
        textArea.style.position = "fixed";  // Avoid scrolling to bottom
        document.body.appendChild(textArea);
        textArea.focus();
        textArea.select();
        try {{
            var successful = document.execCommand('copy');
            if (successful) {{
                alert('Selected indices copied to clipboard.');
            }} else {{
                alert('Could not copy text.');
            }}
        }} catch (err) {{
            alert('Could not copy text: ', err);
        }}
        document.body.removeChild(textArea);
    }}
    </script>
    """

    display(HTML(html_content))
```

## src/unibox/nb_helpers/uni_peeker.py

- Characters: 4845
- Tokens: 0

```python
import json  # Import the json module
from collections import Counter
from typing import Any

import pandas as pd


class CompactJSONEncoder(json.JSONEncoder):
    """Custom JSON Encoder for specific formatting of dictionaries and lists."""

    def iterencode(self, o, _one_shot=False):
        """Overridden method for custom JSON formatting."""
        if isinstance(o, (list, dict)):
            items = []
            if isinstance(o, dict):
                for key, value in o.items():
                    formatted_value = json.dumps(value, separators=(",", ": ")).replace('"', "'")
                    items.append(f"'{key}': {formatted_value}")
            else:  # For lists
                items = [json.dumps(item, separators=(",", ": ")).replace('"', "'") for item in o]
            yield from self._encode_line(items, isinstance(o, dict))
        else:
            yield from super().iterencode(o, _one_shot)

    def _encode_line(self, items, is_dict):
        """Helper method to yield formatted lines."""
        yield "{\n" if is_dict else "[\n"
        for i, item in enumerate(items):
            separator = "," if i < len(items) - 1 else ""
            yield f"    {item}{separator}\n"
        yield "}\n" if is_dict else "]\n"


class UniPeeker:
    """Utility class for peeking into data with efficient methods."""

    def __init__(self, n: int = 3, console_print: bool = False):
        self.n = n
        self.console_print = console_print

    def peeks(self, data: Any, n: int = None, console_print: bool = None) -> dict | None:
        """Peek into the data and return metadata and a preview of the data, with efficient handling for large data."""
        peek_n = n if n else self.n
        _print = console_print if console_print is not None else self.console_print

        data_type = type(data).__name__
        meta_dict = {}
        preview = None

        if data_type == "DataFrame":  # special handling for dataframes
            try:
                from .ipython_utils import peek_df

                return peek_df(data, n=3)

            except ModuleNotFoundError:  # If IPython is not available
                meta_dict, preview = self._peek_dataframe(data, peek_n)
                return {"metadata": meta_dict, "preview": preview}

        elif data_type == "list":
            meta_dict, preview = self._peek_list(data, peek_n)
        if data_type == "dict":
            meta_dict, preview = self._peek_dict(data, peek_n)
        elif data_type == "set" or data_type == "tuple":
            meta_dict, preview = self._peek_list(list(data), peek_n)

        # Handling for other data types...

        if _print:
            self._print_info(data_type, meta_dict, preview)

        return {"metadata": meta_dict, "preview": preview}

    @staticmethod
    def _peek_dict(data: dict, n: int) -> tuple:
        """Peek into a dictionary efficiently."""
        first_n = [(k, data[k]) for k in list(data)[:n]]
        value_types = Counter([type(v).__name__ for v in data.values()])
        is_nested = any(isinstance(v, dict) for v in data.values())

        meta_dict = {
            "len": len(data),
            "value_types": dict(value_types),
            "is_nested": is_nested,
        }
        return meta_dict, first_n

    @staticmethod
    def _peek_dataframe(data: pd.DataFrame, n: int) -> tuple:
        """Peek into a DataFrame."""
        meta_dict = {
            "len": len(data),
            "columns": list(data.columns),
            "dtypes": data.dtypes.to_dict(),
            "shape": data.shape,
        }
        preview = data.head(n)
        return meta_dict, preview

    @staticmethod
    def _peek_list(data: list, n: int) -> tuple:
        """Peek into a list."""
        first_n = data[:n]
        meta_dict = {
            "len": len(data),
            "item_type": type(data[0]).__name__ if data else "None",
        }
        return meta_dict, first_n

    @staticmethod
    def _print_info(data_type: str, meta_dict: dict, preview: Any) -> None:
        """Print information about the data using custom pretty print for the metadata."""
        data_len = meta_dict["len"]

        # Convert dtypes to string if the data is a DataFrame
        if data_type == "DataFrame":
            meta_dict["dtypes"] = {k: str(v) for k, v in meta_dict["dtypes"].items()}

        pretty_meta_dict = json.dumps(meta_dict, cls=CompactJSONEncoder, indent=4, sort_keys=True)
        print(f"[{data_type}] of size [{data_len}]:\n{pretty_meta_dict}\n")
        print(f"[Preview]\n{preview}")


if __name__ == "__main__":
    # Example usage:
    # some_dict_or_dataframe = {"name": "John", "age": 30}

    # df
    some_dict_or_dataframe = pd.DataFrame({"A": [1, 2], "B": [3, 4]})

    unipeeker = UniPeeker()
    result = unipeeker.peeks(some_dict_or_dataframe)

    print("D")
```

## src/unibox/nb_helpers/__init__.py

- Characters: 0
- Tokens: 0

```python

```

## src/unibox/__main__.py

- Characters: 336
- Tokens: 0

```python
"""Entry-point module, in case you use `python -m unibox`.

Why does this file exist, and why `__main__`? For more info, read:

- https://www.python.org/dev/peps/pep-0338/
- https://docs.python.org/3/using/cmdline.html#cmdoption-m
"""

import sys

from unibox.cli import main

if __name__ == "__main__":
    sys.exit(main(sys.argv[1:]))
```

## src/unibox/loaders/base_loader.py

- Characters: 281
- Tokens: 0

```python
"""basic loader class"""

# base_loader.py
from pathlib import Path
from typing import Any


class BaseLoader:
    def load(self, local_path: Path) -> Any:
        raise NotImplementedError

    def save(self, local_path: Path, data: Any) -> None:
        raise NotImplementedError
```

## src/unibox/loaders/toml_loader.py

- Characters: 432
- Tokens: 0

```python
from pathlib import Path
from typing import Any

import tomli
import tomli_w

from .base_loader import BaseLoader


class TOMLLoader(BaseLoader):
    """Load and save TOML files."""

    def load(self, file_path: Path) -> Any:
        with open(file_path, "rb") as f:
            return tomli.load(f)

    def save(self, file_path: Path, data: Any) -> None:
        with open(file_path, "wb") as f:
            tomli_w.dump(data, f)
```

## src/unibox/loaders/csv_loader.py

- Characters: 334
- Tokens: 0

```python
# csv_loader.py
from pathlib import Path

import pandas as pd

from .base_loader import BaseLoader


class CSVLoader(BaseLoader):
    def load(self, local_path: Path) -> pd.DataFrame:
        return pd.read_csv(local_path)

    def save(self, local_path: Path, data: pd.DataFrame) -> None:
        data.to_csv(local_path, index=False)
```

## src/unibox/loaders/parquet_loader.py

- Characters: 386
- Tokens: 0

```python
# parquet_loader.py
from pathlib import Path

import pandas as pd

from .base_loader import BaseLoader


class ParquetLoader(BaseLoader):
    """Load and save Parquet files using pandas."""

    def load(self, file_path: Path) -> pd.DataFrame:
        return pd.read_parquet(file_path)

    def save(self, file_path: Path, data: pd.DataFrame) -> None:
        data.to_parquet(file_path)
```

## src/unibox/loaders/loader_router.py

- Characters: 906
- Tokens: 0

```python
# loader_router.py
# ... etc
from ..utils.constants import IMG_FILES
from .csv_loader import CSVLoader
from .image_loder import ImageLoader
from .json_loader import JSONLoader
from .jsonl_loader import JSONLLoader
from .parquet_loader import ParquetLoader
from .toml_loader import TOMLLoader
from .txt_loader import TxtLoader
from .yaml_loader import YAMLLoader


def get_loader_for_suffix(suffix: str):
    suffix = suffix.lower()
    if suffix == ".csv":
        return CSVLoader()
    if suffix in IMG_FILES:
        return ImageLoader()
    if suffix == ".json":
        return JSONLoader()
    if suffix == ".jsonl":
        return JSONLLoader()
    if suffix == ".parquet":
        return ParquetLoader()
    if suffix == ".txt":
        return TxtLoader()
    if suffix == ".toml":
        return TOMLLoader()
    if suffix == ".yaml" or suffix == ".yml":
        return YAMLLoader()
    return None
```

## src/unibox/loaders/txt_loader.py

- Characters: 1151
- Tokens: 0

```python
from pathlib import Path
from typing import List

from .base_loader import BaseLoader


class TxtLoader(BaseLoader):
    """Load and (optionally) save text files."""

    def load(self, file_path: Path, encoding: str = "utf-8") -> List[str]:
        """Load a text file and return a list of stripped lines.

        Args:
            file_path (Path): The path to the text file.
            encoding (str): The file encoding. Defaults to 'utf-8'.

        Returns:
            List[str]: A list of lines with leading and trailing whitespace removed.
        """
        with open(file_path, encoding=encoding) as f:
            return [line.strip() for line in f.readlines()]

    def save(self, file_path: Path, data: List[str], encoding: str = "utf-8") -> None:
        """Save a list of strings to a text file.

        Args:
            file_path (Path): The path to the text file.
            data (List[str]): The list of strings to write to the file.
            encoding (str): The file encoding. Defaults to 'utf-8'.
        """
        with open(file_path, "w", encoding=encoding) as f:
            f.writelines(f"{line}\n" for line in data)
```

## src/unibox/loaders/README.md

- Characters: 558
- Tokens: 0

````markdown
# Loaders

naming convention: use `{extension}_loader.py`

each loader inherits from `base_loader.py` and implements:

```python
    def load(self, local_path: Path) -> Any:
        raise NotImplementedError

    def save(self, local_path: Path, data: Any) -> None:
        raise NotImplementedError
```


## Adding a new loader

1. Create a new file in `src/unibox/loaders/` with the name `{extension}_loader.py`
2. Implement the `load` and `save` methods
3. Add the loader to the `src/unibox/loaders/loader_router.py` file
4. test that the new loader works
````

## src/unibox/loaders/jsonl_loader.py

- Characters: 915
- Tokens: 0

```python
# jsonl_loader.py
import re
from pathlib import Path
from typing import Any, List

import orjson

from .base_loader import BaseLoader


class JSONLLoader(BaseLoader):
    """Load and save JSONL files."""

    def load(self, file_path: Path) -> List[Any]:
        data = []
        with open(file_path, "rb") as f:
            for line in f:
                line_str = line.decode("utf-8", errors="replace")
                if "NaN" in line_str:
                    line_str = re.sub(r"\bNaN\b", "null", line_str)
                try:
                    data.append(orjson.loads(line_str))
                except orjson.JSONDecodeError:
                    # handle errors if you want
                    pass
        return data

    def save(self, file_path: Path, data: List[Any]) -> None:
        with open(file_path, "wb") as f:
            for item in data:
                f.write(orjson.dumps(item) + b"\n")
```

## src/unibox/loaders/image_loder.py

- Characters: 459
- Tokens: 0

```python
from pathlib import Path

from PIL import Image

from .base_loader import BaseLoader


class ImageLoader(BaseLoader):
    """Load and (optionally) save images using PIL."""

    def load(self, file_path: Path) -> Image.Image:
        return Image.open(file_path)

    def save(self, file_path: Path, data: Image.Image) -> None:
        # Ensure the image is loaded in memory
        if data.fp is not None:
            data.load()
        data.save(file_path)
```

## src/unibox/loaders/yaml_loader.py

- Characters: 477
- Tokens: 0

```python
from pathlib import Path
from typing import Any

import yaml

from .base_loader import BaseLoader


class YAMLLoader(BaseLoader):
    """Load and save YAML files."""

    def load(self, file_path: Path) -> Any:
        with open(file_path, encoding="utf-8") as f:
            return yaml.safe_load(f)

    def save(self, file_path: Path, data: Any) -> None:
        with open(file_path, "w", encoding="utf-8") as f:
            yaml.safe_dump(data, f, default_flow_style=False)
```

## src/unibox/loaders/__init__.py

- Characters: 0
- Tokens: 0

```python

```

## src/unibox/loaders/json_loader.py

- Characters: 609
- Tokens: 0

```python
# json_loader.py
from pathlib import Path
from typing import Any

import orjson

from .base_loader import BaseLoader


class JSONLoader(BaseLoader):
    """Load and save JSON files."""

    def load(self, file_path: Path) -> Any:
        with open(file_path, "rb") as f:
            file_content = f.read()
            if not file_content:
                return None
            return orjson.loads(file_content)

    def save(self, file_path: Path, data: Any) -> None:
        # Decide how you want to handle e.g. dict vs list
        with open(file_path, "wb") as f:
            f.write(orjson.dumps(data))
```

## src/unibox/debug.py

- Characters: 2818
- Tokens: 0

```python
"""Debugging utilities."""

from __future__ import annotations

import os
import platform
import sys
from dataclasses import dataclass
from importlib import metadata


@dataclass
class Variable:
    """Dataclass describing an environment variable."""

    name: str
    """Variable name."""
    value: str
    """Variable value."""


@dataclass
class Package:
    """Dataclass describing a Python package."""

    name: str
    """Package name."""
    version: str
    """Package version."""


@dataclass
class Environment:
    """Dataclass to store environment information."""

    interpreter_name: str
    """Python interpreter name."""
    interpreter_version: str
    """Python interpreter version."""
    interpreter_path: str
    """Path to Python executable."""
    platform: str
    """Operating System."""
    packages: list[Package]
    """Installed packages."""
    variables: list[Variable]
    """Environment variables."""


def _interpreter_name_version() -> tuple[str, str]:
    if hasattr(sys, "implementation"):
        impl = sys.implementation.version
        version = f"{impl.major}.{impl.minor}.{impl.micro}"
        kind = impl.releaselevel
        if kind != "final":
            version += kind[0] + str(impl.serial)
        return sys.implementation.name, version
    return "", "0.0.0"


def get_version(dist: str = "unibox") -> str:
    """Get version of the given distribution.

    Parameters:
        dist: A distribution name.

    Returns:
        A version number.
    """
    try:
        return metadata.version(dist)
    except metadata.PackageNotFoundError:
        return "0.0.0"


def get_debug_info() -> Environment:
    """Get debug/environment information.

    Returns:
        Environment information.
    """
    py_name, py_version = _interpreter_name_version()
    packages = ["unibox"]
    variables = ["PYTHONPATH", *[var for var in os.environ if var.startswith("UNIBOX")]]
    return Environment(
        interpreter_name=py_name,
        interpreter_version=py_version,
        interpreter_path=sys.executable,
        platform=platform.platform(),
        variables=[Variable(var, val) for var in variables if (val := os.getenv(var))],
        packages=[Package(pkg, get_version(pkg)) for pkg in packages],
    )


def print_debug_info() -> None:
    """Print debug/environment information."""
    info = get_debug_info()
    print(f"- __System__: {info.platform}")
    print(f"- __Python__: {info.interpreter_name} {info.interpreter_version} ({info.interpreter_path})")
    print("- __Environment variables__:")
    for var in info.variables:
        print(f"  - `{var.name}`: `{var.value}`")
    print("- __Installed packages__:")
    for pkg in info.packages:
        print(f"  - `{pkg.name}` v{pkg.version}")


if __name__ == "__main__":
    print_debug_info()
```

## src/unibox/py.typed

- Characters: 0
- Tokens: 0

```text

```

## src/unibox/backends/hf_api_backend.py

- Characters: 6695
- Tokens: 0

```python
# unibox/backends/hf_api_backend.py

import os
import shutil
import tempfile
from pathlib import Path
from typing import List, Optional

from huggingface_hub import HfApi, hf_hub_download

from .base_backend import BaseBackend

HF_PREFIX = "hf://"


def parse_hf_uri(hf_uri: str):
    """Parse the Hugging Face URI in the format "hf://{owner}/{repo}/{path_in_repo}".
    Returns (repo_id, path_in_repo).
    """
    if not hf_uri.startswith(HF_PREFIX):
        raise ValueError(f"Invalid HF URI (no hf:// prefix): {hf_uri}")
    # Remove the "hf://" prefix.
    trimmed = hf_uri[len(HF_PREFIX) :]
    parts = trimmed.split("/", 2)
    if len(parts) < 2:
        # e.g. "hf://username/repo" with no trailing subpath -> path_in_repo=''
        owner, name = parts[0], ""
        repo_id = owner  # Not strictly valid, but let's handle carefully
        path_in_repo = ""
        return repo_id, path_in_repo
    if len(parts) == 2:
        # e.g. "hf://owner/repo"
        owner, name = parts
        if "/" not in name:
            return f"{owner}/{name}", ""
        # fallback if there's a slash
    # normal case: 3 parts
    owner, name, subpath = parts
    repo_id = f"{owner}/{name}"
    return repo_id, subpath


class HuggingFaceApiBackend(BaseBackend):
    """A backend that uses low-level HfApi to handle single-file or folder usage in HF repos.

    It can:
      - download a single file (download)
      - upload a single file (upload)
      - list files in a repo (ls)
    For dataset usage, see `HuggingFaceDatasetsBackend`.
    """

    def __init__(self):
        self.api = HfApi()

    def download(self, uri: str, target_dir: str = None) -> Path:
        """Download a single file from a HF repo to `target_dir`.
        If the path_in_repo is actually a folder or there's no final file, we raise NotImplemented.
        """
        if not uri.startswith(HF_PREFIX):
            raise ValueError(f"Invalid HF URI: {uri}")
        repo_id, path_in_repo = parse_hf_uri(uri)
        if not path_in_repo:
            raise ValueError(f"No subpath found in URI: {uri} (cannot do single-file download).")

        if not target_dir:
            target_dir = tempfile.gettempdir()
        os.makedirs(target_dir, exist_ok=True)

        # Download
        revision = "main"  # or read from kwargs if you prefer
        local_path = hf_hub_download(repo_id=repo_id, filename=path_in_repo, revision=revision)
        # Copy from HF cache to target_dir if needed
        filename_only = os.path.basename(path_in_repo)
        final_path = Path(target_dir) / filename_only
        shutil.copy(local_path, final_path)
        return final_path

    def upload(self, local_path: Path, uri: str) -> None:
        """Upload a single local file to HF at the given subpath in repo.
        If the subpath is empty => we treat that as 'folder'? Or raise error?
        """
        repo_id, path_in_repo = parse_hf_uri(uri)
        if not path_in_repo or path_in_repo.endswith("/"):
            # user wants a directory push
            path_in_repo = path_in_repo.rstrip("/") + "/" + local_path.name

        # Ensure repo exists
        self.api.create_repo(repo_id=repo_id, private=True, exist_ok=True)
        # Upload
        self.api.upload_file(
            path_or_fileobj=str(local_path),
            path_in_repo=path_in_repo,
            repo_id=repo_id,
        )

    def ls(
        self,
        uri: str,
        exts: Optional[List[str]] = None,
        relative_unix: bool = False,
        debug_print: bool = True,
        **kwargs,
    ) -> List[str]:
        """List all files in the HF repo. If path_in_repo is a subfolder prefix, we can filter.
        For extension filtering or subpath filtering, you'd manually do it. Here we do a simple approach.
        """
        repo_id, path_in_repo = parse_hf_uri(uri)
        files = self.api.list_repo_files(repo_id=repo_id)
        # If path_in_repo is not empty, we can filter by that prefix
        if path_in_repo:
            path_in_repo = path_in_repo.rstrip("/")
            files = [f for f in files if f.startswith(path_in_repo)]

        # If exts is given, filter
        if exts:
            exts = [e.lower() for e in exts]
            files = [f for f in files if any(f.lower().endswith(x) for x in exts)]

        # Possibly convert to relative or restore full "hf://..."
        results = []
        for f in files:
            if relative_unix:
                # show subpath relative
                sub = f[len(path_in_repo) :].lstrip("/") if path_in_repo else f
                sub = sub.replace("\\", "/")
                results.append(sub)
            else:
                # return a full HF URI
                results.append(f"hf://{repo_id}/{f}")
        return results

    # -------- Additional methods (from snippet) if needed. --------

    def load_file(self, hf_uri: str, revision: str = "main") -> str:
        """Download a single file from the HF repo and return the local path."""
        repo_id, path_in_repo = parse_hf_uri(hf_uri)
        local_path = hf_hub_download(repo_id=repo_id, filename=path_in_repo, revision=revision)
        print(f"load_file {hf_uri} -> {local_path}")
        return local_path

    def cp_to_hf(self, local_file_path: str, hf_uri: str, private: bool = True):
        """Copy (upload) a local file to a Hugging Face repository.
        (Provided for reference; 'upload()' is the simpler approach.)
        """
        repo_id, path_in_repo = parse_hf_uri(hf_uri)
        if hf_uri.endswith("/") or not os.path.basename(path_in_repo):
            path_in_repo = path_in_repo.rstrip("/") + "/" + os.path.basename(local_file_path)
        self.api.create_repo(repo_id=repo_id, private=private, exist_ok=True)
        self.api.upload_file(
            path_or_fileobj=local_file_path,
            path_in_repo=path_in_repo,
            repo_id=repo_id,
        )
        print(f"cp {local_file_path} hf://{repo_id}/{path_in_repo}")

    def cp_to_local(self, hf_uri: str, local_file_path: str, revision: str = "main"):
        """Copy (download) a file from a Hugging Face repository to a local path.
        (Provided for reference; 'download()' is the simpler approach.)
        """
        repo_id, path_in_repo = parse_hf_uri(hf_uri)
        if os.path.isdir(local_file_path):
            local_file_path = os.path.join(local_file_path, os.path.basename(path_in_repo))
        cached_file_path = hf_hub_download(repo_id=repo_id, filename=path_in_repo, revision=revision)
        shutil.copy(cached_file_path, local_file_path)
        print(f"cp {hf_uri} {local_file_path}")

    # You can include rm/mv/cp as well if you wish, but omitted here for brevity.
```

## src/unibox/backends/backend_router.py

- Characters: 665
- Tokens: 0

```python
# unibox/backends/backend_router.py

from unibox.utils.utils import is_hf_uri, is_s3_uri, is_url

from .base_backend import BaseBackend
from .hf_router_backend import HuggingFaceRouterBackend
from .local_backend import LocalBackend
from .s3_backend import S3Backend


def get_backend_for_uri(uri: str) -> BaseBackend:
    if is_s3_uri(uri):
        return S3Backend()
    if is_hf_uri(uri):
        # Our new router that delegates between dataset or single-file
        return HuggingFaceRouterBackend()
    if is_url(uri):
        # Possibly define an HTTPBackend or just treat as local after manual download
        return LocalBackend()
    return LocalBackend()
```

## src/unibox/backends/README.md

- Characters: 1097
- Tokens: 0

````markdown
# Backends

A `backend` interacts with the underlying file system to handle file operations. Currently it could be either local (no change), S3 (posix-like), or Huggingface 

Huggingface is designed to be duplex: 
- `hf:user/repo` as dataframe store
- or `hf:user/repo/file.ext` as posix-like (not implemented yet)

A backend implements a `BaseBackend`:

```python

class BaseBackend:
    """Interface for storage backends (local, S3, etc.)."""

    def download(self, uri: str, target_dir: str = None) -> Path:
        """Download the resource identified by `uri` to a local temp path. Return local Path."""
        raise NotImplementedError

    def upload(self, local_path: Path, uri: str) -> None:
        """Upload local_path to the specified `uri`."""
        raise NotImplementedError

    def ls(
        self,
        uri: str,
        exts: Optional[List[str]] = None,
        relative_unix: bool = False,
        debug_print: bool = True,
        **kwargs,
    ) -> List[str]:
        """List files under `uri` with optional extension filtering."""
        raise NotImplementedError
```
````

## src/unibox/backends/local_backend.py

- Characters: 3735
- Tokens: 0

```python
# local_backend.py
import os
import shutil
import warnings
from pathlib import Path
from typing import List, Optional

from tqdm.auto import tqdm

from .base_backend import BaseBackend


class LocalBackend(BaseBackend):
    def download(self, uri: str, target_dir: str = None) -> Path:
        # local path is the URI itself, so just return Path(uri)
        return Path(uri)

    def upload(self, local_path: Path, uri: str) -> None:
        """Ensure the final path has the saved file."""
        dest = Path(uri)
        if dest != local_path:
            # Copy the temp file to the final path
            dest.parent.mkdir(parents=True, exist_ok=True)
            shutil.copy2(str(local_path), str(dest))

    def _traverse_local_dir(
        self,
        root_dir: str,
        exts: Optional[List[str]] = None,
        relative_unix: bool = False,
        debug_print: bool = True,
    ) -> List[str]:
        """Traverses the local directory tree and returns a list of files matching the criteria.

        Args:
            root_dir (str): The root directory to traverse.
            exts (List[str], optional): List of file extensions to include. Defaults to None.
            relative_unix (bool, optional): Whether to return relative Unix-style paths. Defaults to False.
            debug_print (bool, optional): Whether to display a progress bar. Defaults to True.

        Returns:
            List[str]: List of file paths.
        """
        abs_root = Path(os.path.expanduser(os.path.expandvars(root_dir))).resolve()
        found_files = []

        # Precompute extensions as a tuple for faster filtering
        exts = tuple(exts) if exts else None

        # Initialize tqdm progress bar
        pbar = tqdm(desc="Listing local files", leave=False, unit="files", disable=not debug_print)

        for dirpath, _, filenames in os.walk(abs_root):
            for file_name in filenames:
                # Filter files based on extensions
                if exts is None or file_name.endswith(exts):
                    full_path = Path(dirpath) / file_name

                    if relative_unix:
                        rel_path = full_path.relative_to(abs_root).as_posix()
                        found_files.append(str(rel_path))
                    else:
                        found_files.append(str(full_path))

                    pbar.update(1)

        pbar.close()  # Close the progress bar
        return found_files

    def ls(
        self,
        uri: str,
        exts: Optional[List[str]] = None,
        relative_unix: bool = False,
        debug_print: bool = True,
        **kwargs,
    ) -> List[str]:
        """Lists files in the local directory with optional extension filtering.

        Args:
            uri (str): Directory URI to list.
            exts (List[str], optional): List of extensions to include. Defaults to None.
            relative_unix (bool, optional): Whether to return relative Unix-style paths. Defaults to False.
            debug_print (bool, optional): Whether to display a progress bar. Defaults to True.

        Returns:
            List[str]: List of file paths.
        """
        # Handle backward compatibility for `include_extensions`
        include_extensions = kwargs.pop("include_extensions", None)
        if include_extensions is not None:
            warnings.warn(
                "`include_extensions` is deprecated; use `exts` instead.",
                category=DeprecationWarning,
                stacklevel=2,
            )
            exts = include_extensions

        return self._traverse_local_dir(
            root_dir=uri,
            exts=exts,
            relative_unix=relative_unix,
            debug_print=debug_print,
        )
```

## src/unibox/backends/__init__.py

- Characters: 0
- Tokens: 0

```python

```

## src/unibox/backends/hf_datasets_backend.py

- Characters: 2943
- Tokens: 0

```python
# unibox/backends/hf_datasets_backend.py

from pathlib import Path
from typing import Any, List, Optional

import pandas as pd
from datasets import Dataset, load_dataset

from ..utils.logger import UniLogger
from .base_backend import BaseBackend

logger = UniLogger()

HF_PREFIX = "hf://"


def parse_hf_uri(uri: str):
    """Given 'hf://username/repo_name/subpath...' -> (repo_id='username/repo_name', subpath='subpath...')."""
    if not uri.startswith(HF_PREFIX):
        raise ValueError(f"Not an HF URI: {uri}")
    trimmed = uri[len(HF_PREFIX) :]
    parts = trimmed.split("/", maxsplit=1)
    repo_id = parts[0]
    subpath = parts[1] if len(parts) > 1 else ""
    return repo_id, subpath


class HuggingFaceDatasetsBackend(BaseBackend):
    """A backend that handles HF *datasets* usage:
    - load_dataset() from a repo
    - push entire Dataset to a repo
    """

    def download(self, uri: str, target_dir: str = None) -> Path:
        """For a dataset-based approach, we usually don't do single-file 'download()'.
        We'll raise NotImplemented if we ever get here for single-file usage.
        """
        raise NotImplementedError("HuggingFaceDatasetsBackend does not handle single-file download.")

    def upload(self, local_path: Path, uri: str) -> None:
        """Expects the user is pushing a Dataset => must handle separately. Use data_to_hub()."""
        raise NotImplementedError("HuggingFaceDatasetsBackend expects entire dataset push (see data_to_hub).")

    def ls(
        self,
        uri: str,
        exts: Optional[List[str]] = None,
        relative_unix: bool = False,
        debug_print: bool = True,
        **kwargs,
    ) -> List[str]:
        """List sub-files or splits in a HF dataset repo? Currently not implemented."""
        raise NotImplementedError("Listing HF dataset contents is not implemented here.")

    # -- Additional helper for dataset loading:
    def load_dataset(self, repo_id: str, split: Optional[str] = "train", revision: str = "main") -> Dataset:
        """Load a dataset from the HF Hub and return a `datasets.Dataset`."""
        return load_dataset(repo_id, split=split, revision=revision)

    # -- Additional helper for pushing data frames or Datasets:
    def data_to_hub(self, data: pd.DataFrame | Dataset | Any, repo_id: str, private: bool = True, **kwargs) -> None:
        """Upload a DataFrame or HF Dataset to HF as a dataset repo.

        kwargs:
            eg. `split` for Dataset.from_pandas()
        """
        logger.info(f"Uploading dataset to HF repo {repo_id}")
        if isinstance(data, Dataset):
            ds = data
        elif isinstance(data, pd.DataFrame):
            ds = Dataset.from_pandas(data, **kwargs)
        else:
            # attempt to convert
            ds = Dataset.from_pandas(pd.DataFrame(data), **kwargs)

        # create_repo(repo_id, private=private, exist_ok=True)
        ds.push_to_hub(repo_id, private=private)
```

## src/unibox/backends/s3_backend.py

- Characters: 2861
- Tokens: 0

```python
# s3_backend.py
import os
import warnings
from pathlib import Path
from typing import List, Optional

from unibox.utils.s3_client import S3Client

from .base_backend import BaseBackend


class S3Backend(BaseBackend):
    def __init__(self):
        self._client = S3Client()

    def _ensure_s3_uri(self, uri: str) -> str:
        uri = str(uri).strip()
        if not uri.startswith("s3://"):
            raise ValueError(f"Not an S3 URI: {uri}")
        return uri

    def download(self, uri: str, target_dir: str = None) -> Path:
        """Download the file from S3. If target_dir is given, place it there,
        else use your global or stable temp directory.
        """
        uri = self._ensure_s3_uri(uri)
        if not target_dir:
            from unibox.utils.globals import GLOBAL_TMP_DIR

            target_dir = GLOBAL_TMP_DIR

        os.makedirs(target_dir, exist_ok=True)
        local_path = self._client.download(uri, target_dir)
        return Path(local_path)

    def upload(self, local_path: Path, uri: str) -> None:
        """Upload the local file to S3."""
        uri = self._ensure_s3_uri(uri)
        self._client.upload(str(local_path), s3_uri=uri)

    def ls(
        self,
        uri: str,
        exts: Optional[List[str]] = None,
        relative_unix: bool = False,
        debug_print: bool = True,
        **kwargs,
    ) -> List[str]:
        """List files in the S3 "directory" with optional extension filtering.

        Args:
            uri (str): S3 directory URI to list.
            exts (Optional[List[str]]): List of file extensions to include. Defaults to None.
            relative_unix (bool): Whether to return relative Unix-style paths. Defaults to False.
            debug_print (bool): Whether to display a progress bar. Defaults to True.
            **kwargs: Additional arguments for backward compatibility.
                      Supports 'include_extensions' (deprecated) and 'exclude_extensions'.

        Returns:
            List[str]: List of file keys or full S3 URIs.
        """
        # Handle backward compatibility for include_extensions.
        include_extensions = kwargs.pop("include_extensions", None)
        if include_extensions is not None:
            warnings.warn(
                "`include_extensions` is deprecated; use `exts` instead.",
                category=DeprecationWarning,
                stacklevel=2,
            )
            exts = include_extensions

        # Also allow passing 'exclude_extensions' if needed.
        exclude_extensions = kwargs.pop("exclude_extensions", None)

        uri = self._ensure_s3_uri(uri)
        return self._client.traverse(
            s3_uri=uri,
            include_extensions=exts,
            exclude_extensions=exclude_extensions,
            relative_unix=relative_unix,
            debug_print=debug_print,
        )
```

## src/unibox/backends/base_backend.py

- Characters: 1831
- Tokens: 0

```python
# base_backend.py
import warnings
from pathlib import Path
from typing import List, Optional


class BaseBackend:
    """Interface for storage backends (local, S3, etc.)."""

    def download(self, uri: str, target_dir: str = None) -> Path:
        """Download the resource identified by `uri` to a local temp path. Return local Path."""
        raise NotImplementedError

    def upload(self, local_path: Path, uri: str) -> None:
        """Upload local_path to the specified `uri`."""
        raise NotImplementedError

    def ls(
        self,
        uri: str,
        exts: Optional[List[str]] = None,
        relative_unix: bool = False,
        debug_print: bool = True,
        **kwargs,
    ) -> List[str]:
        """List files under `uri` with optional extension filtering.

        :param uri: A string representing a directory path or location.
        :param exts: A list of file extensions to include (['.txt', '.csv']).
        :param include_extensions: (DEPRECATED) old name for exts, raises a warning if used.
        :param relative_unix: Return relative paths with forward slashes if True.
        :param debug_print: Show progress bar.
        :param kwargs: Additional arguments for compatibility (ignored by default).
        :return: A list of file paths.
        """
        include_extensions = kwargs.pop("include_extensions", None)
        if include_extensions is not None:
            warnings.warn(
                "`include_extensions` is deprecated; use `exts` instead.",
                category=DeprecationWarning,
                stacklevel=2,
            )
            exts = include_extensions

        # By default, raise NotImplementedError.
        # LocalBackend or other backends can override with real logic.
        raise NotImplementedError("ls() is not implemented in BaseBackend.")
```

## src/unibox/backends/hf_router_backend.py

- Characters: 4486
- Tokens: 0

```python
# unibox/backends/hf_router_backend.py

from pathlib import Path
from typing import List, Optional

from ..utils.logger import UniLogger
from .base_backend import BaseBackend
from .hf_api_backend import HuggingFaceApiBackend
from .hf_datasets_backend import HuggingFaceDatasetsBackend, parse_hf_uri

logger = UniLogger()


def has_dot_in_final_segment(subpath: str) -> bool:
    """Quick check: if the last path segment has a '.' in it, we treat that
    as a 'file-like' path. e.g. "folder/stuff.bin" or "myfile.parquet"
    """
    seg = subpath.split("/")[-1] if subpath else ""
    return "." in seg


class HuggingFaceRouterBackend(BaseBackend):
    """A meta-backend that tries either dataset or single-file approach:
      - If there's no subpath or the subpath doesn't have a dot => single-file first, fallback dataset
      - If there's a dot => dataset first, fallback single-file
    This matches the original “try one, fallback other” logic you requested.
    """

    def __init__(self):
        super().__init__()
        self.api_backend = HuggingFaceApiBackend()  # single-file
        self.ds_backend = HuggingFaceDatasetsBackend()  # dataset

    def download(self, uri: str, target_dir: str = None) -> Path:
        """We interpret “download” to mean “give me a local file”.
        So if we suspect dataset => not implemented. If we suspect single-file => do it.
        But we can do a fallback approach if that fails.
        """
        repo_id, subpath = parse_hf_uri(uri)
        # no subpath => definitely can't do single-file => or subpath has no dot => single-file first
        if not subpath:
            # no subpath => can't single-file => we do dataset approach? Actually dataset approach
            # doesn't produce a single local file. So “download a dataset” is a mismatch.
            raise NotImplementedError(
                "Cannot download entire dataset to single local file in this router. "
                "Try using unibox.py loads() to load HF dataset, or handle multiple files.",
            )
        # If there's a dot => dataset first, fallback single-file
        # OR if there's no dot => single-file first, fallback dataset
        if has_dot_in_final_segment(subpath):
            # dataset first
            try:
                # But dataset approach's download() is not implemented => we must do something else
                # We'll raise or fallback to single-file?
                raise NotImplementedError("Downloading HF dataset as a single file is not possible in ds_backend.")
            except Exception as e_ds:
                logger.debug(f"Dataset approach failed: {e_ds}, falling back to single-file download.")
                # fallback single-file
                return self.api_backend.download(uri, target_dir)
        else:
            # single-file first
            try:
                return self.api_backend.download(uri, target_dir)
            except Exception as e_file:
                logger.debug(f"Single-file approach failed: {e_file}, trying dataset approach.")
                raise NotImplementedError("No direct single-file => dataset fallback is feasible here.")

    def upload(self, local_path: Path, uri: str) -> None:
        """If subpath has a dot => dataset push first, fallback single-file?
        Or the opposite? Adjust the logic as you see fit.
        In practice, you might check if `local_path` is a single file or if you have
        a `datasets.Dataset` object.
        """
        repo_id, subpath = parse_hf_uri(uri)
        if not subpath:
            # Means “hf://owner/repo” no file => dataset approach
            # We actually do “data_to_hub” in the ds backend
            raise NotImplementedError("For dataset push, call ds_backend.data_to_hub(...) directly.")
        # If there's a dot => likely single-file => let's do the api backend
        self.api_backend.upload(local_path, uri)

    def ls(
        self,
        uri: str,
        exts: Optional[List[str]] = None,
        relative_unix: bool = False,
        debug_print: bool = True,
        **kwargs,
    ) -> List[str]:
        """We can attempt listing from the API backend (which uses list_repo_files).
        If you want to separate “dataset splits” listing from “files listing,”
        you might do that with ds_backend, but here let's just rely on api_backend.
        """
        return self.api_backend.ls(uri, exts=exts, relative_unix=relative_unix, debug_print=debug_print, **kwargs)
```

## src/unibox/utils/utils.py

- Characters: 1617
- Tokens: 0

```python
import logging
import os
from urllib.parse import urlparse

logger = logging.getLogger(__name__)


def to_relaive_unix_path(absolute_path: str, root_dir: str, convert_slash=True):
    relative_path = os.path.relpath(absolute_path, root_dir)
    if convert_slash:
        relative_path = relative_path.replace("\\", "/")
    return relative_path


def is_s3_uri(uri: str) -> bool:
    """Check if the URI is an S3 URI."""
    parsed = urlparse(uri)
    return parsed.scheme == "s3"


def is_url(path: str) -> bool:
    try:
        result = urlparse(path)
        return all([result.scheme, result.netloc])
    except:
        return False


def is_hf_uri(uri: str) -> bool:
    """Check if the URI is a Hugging Face URI."""
    return uri.startswith("hf://")


def merge_dicts(*dicts):
    """Merge dictionaries, raising warnings for overlapping keys and data type mismatches.

    Args:
        *dicts: Dictionaries to merge.
        logger: Logger instance for warnings (default: None).

    Returns:
        dict: Merged dictionary.
    """
    assert all(isinstance(d, dict) for d in dicts), "All inputs must be dictionaries."

    result = {}
    for d in dicts:
        for key, value in d.items():
            if key in result:
                logger.warning(f"Overlapping key '{key}' detected. Existing value: {result[key]}, New value: {value}")
                if type(result[key]) != type(value):
                    logger.warning(
                        f"Data type mismatch for key '{key}': {type(result[key])} vs {type(value)}.",
                    )
            result[key] = value

    return result
```

## src/unibox/utils/llm_api.py

- Characters: 2109
- Tokens: 0

```python
import requests


def generate_gemini(prompt, api_key, model="gemini-1.5-flash"):
    """Generates content using Google's Generative Language API

    Args:
        prompt: Input text/prompt for generation
        api_key: Your API key
        model: Model name (default: gemini-1.5-flash)

    Returns:
        Generated text as a string
    """
    url = f"https://generativelanguage.googleapis.com/v1beta/models/{model}:generateContent"
    headers = {"Content-Type": "application/json"}
    params = {"key": api_key}

    data = {
        "contents": [
            {
                "parts": [{"text": prompt}],
            },
        ],
    }

    response = requests.post(url, json=data, headers=headers, params=params)

    # Check for HTTP errors


def generate_openai(prompt, api_key, model="text-davinci-003", endpoint="https://api.openai.com/v1/engines"):
    """Generates content using OpenAI's API

    Args:
        prompt: Input text/prompt for generation
        api_key: Your API key
        model: Model name (default: text-davinci-003)
        endpoint: API endpoint (default: OpenAI's endpoint)

    Returns:
        Generated text as a string
    """
    url = f"{endpoint}/{model}/completions"
    headers = {
        "Content-Type": "application/json",
        "Authorization": f"Bearer {api_key}",
    }

    data = {
        "prompt": prompt,
        "max_tokens": 20000,
    }

    response = requests.post(url, json=data, headers=headers)

    # Check for HTTP errors
    if response.status_code != 200:
        raise Exception(f"API request failed: {response.status_code} - {response.text}")

    try:
        return response.json()["choices"][0]["text"]
    except (KeyError, IndexError) as e:
        raise ValueError("Unexpected API response format") from e

    if response.status_code != 200:
        raise Exception(f"API request failed: {response.status_code} - {response.text}")

    try:
        return response.json()["candidates"][0]["content"]["parts"][0]["text"]
    except (KeyError, IndexError) as e:
        raise ValueError("Unexpected API response format") from e
```

## src/unibox/utils/s3_client.py

- Characters: 5650
- Tokens: 0

```python
import logging
import os
from urllib.parse import urlparse

import boto3
from botocore.exceptions import ClientError
from tqdm.auto import tqdm


def parse_s3_url(url: str):
    parsed_url = urlparse(url)
    if parsed_url.scheme != "s3":
        raise ValueError(
            f"Expecting 's3' scheme, got: {parsed_url.scheme} in {url}.",
        )
    return parsed_url.netloc, parsed_url.path.lstrip("/")


class S3Client:
    def __init__(self) -> None:
        # Simple S3 client init; if you need custom credentials or region,
        # pass them directly via environment variables or create a custom session.
        self.s3 = boto3.client("s3")

    def download(self, s3_uri: str, target_dir: str) -> str:
        """Download a file from S3 to a local directory.
        :param s3_uri: S3 URI (e.g. s3://bucket/key)
        :param target_dir: Local directory path
        :return: Local file path
        """
        bucket, key = parse_s3_url(s3_uri)
        filename = os.path.basename(s3_uri)
        path = os.path.join(target_dir, filename)
        self.s3.download_file(bucket, key, path)
        return path

    def upload(self, file_path: str, s3_uri: str) -> None:
        """Upload a local file to S3.
        :param file_path: Local file path
        :param s3_uri: S3 URI (e.g. s3://bucket/key)
        """
        bucket, key = parse_s3_url(s3_uri)
        self.s3.upload_file(file_path, bucket, key)

    def exists(self, s3_uri: str) -> bool:
        """Check if a file exists in S3 at the given URI.
        :param s3_uri: S3 URI
        :return: True if object exists, False otherwise.
        """
        bucket, key = parse_s3_url(s3_uri)
        try:
            self.s3.head_object(Bucket=bucket, Key=key)
            return True
        except self.s3.exceptions.ClientError:
            return False

    def walk(self, s3_uri: str):
        """Generator that walks all objects under the given S3 URI.
        Yields metadata dictionaries for each object.
        """
        bucket, key = parse_s3_url(s3_uri)
        paginator = self.s3.get_paginator("list_objects_v2")
        pages = paginator.paginate(Bucket=bucket, Prefix=key)
        for page in pages:
            for obj in page["Contents"]:
                yield {
                    "key": obj["Key"],
                    "size": obj["Size"],
                    "last_modified": obj["LastModified"],
                    "etag": obj["ETag"],
                    "storage_class": obj["StorageClass"],
                }

    def traverse(
        self,
        s3_uri: str,
        include_extensions=None,
        exclude_extensions=None,
        relative_unix=False,
        debug_print=True,
    ):
        """Traverse through an S3 "directory" and return entries under it.

        :param include_extensions: list of file extensions to include (e.g. ['.jpg', '.png']).
        :param exclude_extensions: list of file extensions to exclude (e.g. ['.txt', '.json']).
        :param relative_unix: return relative paths or full s3:// URIs.
        :param debug_print: whether to show a tqdm progress bar.
        :return: list of keys or URIs.
        """
        bucket, prefix = parse_s3_url(s3_uri)

        if not prefix.endswith("/"):
            prefix += "/"

        paginator = self.s3.get_paginator("list_objects_v2")
        response_iterator = paginator.paginate(Bucket=bucket, Prefix=prefix, Delimiter="/")

        all_entries = []

        if debug_print:
            response_iterator = tqdm(response_iterator, desc="Traversing S3", unit="page")

        for page in response_iterator:
            # Subdirectories
            for d in page.get("CommonPrefixes", []):
                dir_key = d["Prefix"]
                dir_entry = dir_key if relative_unix else f"s3://{bucket}/{dir_key}"
                all_entries.append(dir_entry)

            # Files
            for obj in page.get("Contents", []):
                file_key = obj["Key"]
                if file_key == prefix:
                    continue  # skip the directory itself

                # Check include/exclude
                if (include_extensions is None or any(file_key.endswith(ext) for ext in include_extensions)) and (
                    exclude_extensions is None or not any(file_key.endswith(ext) for ext in exclude_extensions)
                ):
                    file_entry = file_key[len(prefix) :] if relative_unix else f"s3://{bucket}/{file_key}"
                    all_entries.append(file_entry)

        return all_entries

    def generate_presigned_uri(self, s3_uri: str, expiration: int = 604800) -> str:
        """Generate a presigned URL from a given S3 URI with a default expiration of 7 days.

        :param s3_uri: S3 URI (e.g., 's3://bucket-name/object-key')
        :param expiration: Time in seconds for the presigned URL to remain valid (default 7 days).
        :return: Presigned URL as a string. If error, returns None.
        """
        bucket, key = parse_s3_url(s3_uri)

        # Constrain expiration to AWS max if needed.
        expiration = min(expiration, 604800)

        try:
            response = self.s3.generate_presigned_url(
                "get_object",
                Params={"Bucket": bucket, "Key": key},
                ExpiresIn=expiration,
            )
            return response
        except ClientError as e:
            logging.exception(f"Failed to generate presigned URL for {s3_uri}: {e}")
            return None


if __name__ == "__main__":
    client = S3Client()
    s3_uri = "s3://dataset-ingested/gallery-dl/_todo_lists/"
    res = client.traverse(s3_uri)
    print(res)
    print("Done.")
```

## src/unibox/utils/README.md

- Characters: 541
- Tokens: 0

```markdown
# Utils

utility code that's often imported before everything else, so there wont't be a circular import:

- `constants`, `globals` are initialized at `__init__` before everything else

- `logger`: a pre-configured color logger with stack trace

- `s3_client`: s3 operations class used for s3 backend

- `utils`: other util functions

- `llm_api`: functions to interact with language model APIs like Google's Generative Language API and OpenAI's API
- `uni_peeker`: a "peeker" that provides previews for various datatypes, handy in notebooks
```

## src/unibox/utils/globals.py

- Characters: 899
- Tokens: 0

```python
"""create a global temporary directory that is cleaned up when the program exits.

When accessing files that's not local, they have to be downloaded first;

having a global temporary dir ensures that the files can be properly cleaned up when the program exits.

"""

import logging
import os
import tempfile
from pathlib import Path

logger = logging.getLogger(__name__)

# Define a fixed name for the temporary directory
_default_tmp_dir = Path(tempfile.gettempdir()) / "unibox_temp"
GLOBAL_TMP_DIR = Path(os.environ.get("UNIBOX_TEMP_DIR", _default_tmp_dir))
print(f"Using global temporary directory: {GLOBAL_TMP_DIR}")

# cleanup previous runs
# shutil.rmtree(GLOBAL_TMP_DIR, ignore_errors=True)

# Create the directory if it doesn't exist
GLOBAL_TMP_DIR.mkdir(parents=True, exist_ok=True)

# @atexit.register
# def cleanup_global_tmp_dir():
#     shutil.rmtree(GLOBAL_TMP_DIR, ignore_errors=True)
```

## src/unibox/utils/logger.py

- Characters: 5958
- Tokens: 0

```python
import inspect
import logging
import os
import sys
from datetime import datetime
from pathlib import Path

import colorama
import colorlog

NOTICE = 25  # Value between WARNING (30) and INFO (20)
logging.addLevelName(NOTICE, "NOTICE")


class UniLogger:
    """A logger that:
    1) Uses colorlog for console color.
    2) Writes an optional log file without color.
    3) Shows the caller's class and method.
    4) Conditionally includes file paths for specific log levels.
    5) Detects if console supports color and allows disabling colors.
    """

    def __init__(
        self,
        output_dir: str = "logs",
        file_suffix: str = "log",
        verbose: bool = False,
        logger_name: str = None,
        write_log: bool = True,
        shorten_levels: int = 2,  # how many path parts to show for debug logs
        use_color: bool = True,  # manually enable/disable color
    ):
        self.verbose = verbose
        self.write_log = write_log
        self.shorten_levels = shorten_levels

        # Determine if console supports color
        self.supports_color = self._detect_color_support() if use_color else False

        self.logger = logging.getLogger(logger_name if logger_name else self.__class__.__name__)
        self.logger.setLevel(logging.DEBUG if verbose else logging.INFO)

        # Prepare handlers
        self.handlers = []

        # Optional file handler
        if self.write_log:
            output_path = Path(output_dir)
            output_path.mkdir(parents=True, exist_ok=True)
            log_file = output_path / f"{file_suffix}_{datetime.now().strftime('%Y%m%d')}.log"
            fh = logging.FileHandler(log_file, mode="a", encoding="utf-8")
            self.handlers.append(fh)

        # Console handler
        ch = logging.StreamHandler(sys.stdout)
        self.handlers.append(ch)

        # Add handlers to logger
        if not self.logger.hasHandlers():
            for h in self.handlers:
                self.logger.addHandler(h)

        self._setup_formatters()

    def _setup_formatters(self):
        console_format = "%(asctime)s [%(levelname)s] %(my_func)s: %(message)s%(extra_path)s"
        date_format = "%Y-%m-%d %H:%M:%S"

        # 1) If console supports color, use colorlog for the console handler:
        if self.supports_color:
            console_formatter = colorlog.ColoredFormatter(
                "%(log_color)s" + console_format,
                datefmt=date_format,
                log_colors={
                    "DEBUG": "cyan",
                    "INFO": "white",
                    "NOTICE": "bold_green",
                    "WARNING": "yellow",
                    "ERROR": "red",
                    "CRITICAL": "bold_red",
                },
            )
        else:
            console_formatter = logging.Formatter(console_format, date_format)

        # 2) Plain-text formatter for the file handler:
        file_format = "%(asctime)s [%(levelname)s] %(my_func)s: %(message)s%(extra_path)s"
        file_formatter = logging.Formatter(file_format, date_format)

        # 3) Assign them:
        for handler in self.handlers:
            if isinstance(handler, logging.FileHandler):
                handler.setFormatter(file_formatter)
            else:
                handler.setFormatter(console_formatter)

    def _shorten_path(self, path_str: str, levels: int = 2) -> str:
        """Return a shortened path for display."""
        parts = path_str.strip("/").split("/")
        return "/" + "/".join(parts[-levels:]) if len(parts) > levels else "/" + "/".join(parts)

    def _detect_color_support(self) -> bool:
        """Detects if the console supports color."""
        # Check if running in Jupyter or IPython
        try:
            from IPython import get_ipython

            if get_ipython():
                return True
        except ImportError:
            pass

        # Check for TTY and colorama initialization
        if sys.stdout.isatty():
            try:
                colorama.init()  # Initialize colorama for Windows
                return True
            except ImportError:
                pass

        return False

    def log(self, level_name: str, message: str):
        """Log with custom formatting based on log level."""
        level = getattr(logging, level_name.upper(), logging.INFO)

        # Skip frames to find real caller
        caller_frame = inspect.currentframe().f_back.f_back
        method_name = caller_frame.f_code.co_name
        class_name = None

        if "self" in caller_frame.f_locals:
            class_name = caller_frame.f_locals["self"].__class__.__name__

        if class_name:
            full_func_name = f"{class_name}.{method_name}"
        else:
            full_func_name = method_name

        full_path = os.path.abspath(caller_frame.f_code.co_filename)
        short_path = self._shorten_path(full_path, self.shorten_levels)
        lineno = caller_frame.f_lineno

        # Conditionally include the path for debug, warning, error, critical levels
        if level in [logging.DEBUG, logging.WARNING, logging.ERROR, logging.CRITICAL]:
            extra_path = f" {short_path}:{lineno}"
        else:
            extra_path = ""

        # Provide custom fields in `extra`
        extra = {
            "my_func": full_func_name,
            "my_lineno": lineno,
            "extra_path": extra_path,
        }

        # Log using extra fields
        self.logger.log(level, message, extra=extra)

    def info(self, message: str):
        self.log("INFO", message)

    def debug(self, message: str):
        self.log("DEBUG", message)

    def warning(self, message: str):
        self.log("WARNING", f"⚠️ {message}")

    def error(self, message: str):
        self.log("ERROR", f"❌ {message}")

    def notice(self, message: str):
        self.log("NOTICE", f"✅ {message}")

    def critical(self, message: str):
        self.log("CRITICAL", f"🔥 {message}")
```

## src/unibox/utils/__init__.py

- Characters: 0
- Tokens: 0

```python

```

## src/unibox/utils/constants.py

- Characters: 563
- Tokens: 0

```python
"""commonly used constant variables"""

__all_imgs_raw = "jpg jpeg png bmp dds exif jp2 jpx pcx pnm ras gif tga tif tiff xbm xpm webp jpe"
IMG_FILES = ["." + i.strip() for i in __all_imgs_raw.split(" ")]

# new name
IMAGE_FILES = IMG_FILES

__all_videos_raw = "webm mkv flv vob ogv ogg avi mov qt wmv yuv rm rmvb asf m4v mpeg mp4 mpe mpg m2v 3gp 3g2"
VIDEO_FILES = ["." + i.strip() for i in __all_videos_raw.split(" ")]

__all_audio_raw = "mp3 wav aac flac ogg wma m4a aiff au opus alac amr ac3"
AUDIO_FILES = ["." + i.strip() for i in __all_audio_raw.split(" ")]
```

## src/unibox/__init__.py

- Characters: 628
- Tokens: 0

```python
"""unibox package.

unibox provides unified interface for common file operations
"""

# unibox/__init__.py
from __future__ import annotations

__all__: list[str] = [
    "GLOBAL_TMP_DIR",
    "IMAGE_FILES",
    "IMG_FILES",
    "VIDEO_FILES",
    "UniLogger",
    "concurrent_loads",
    "gallery",
    "label_gallery",
    "loads",
    "ls",
    "peeks",
    "saves",
    "traverses",
]

from .unibox import concurrent_loads, gallery, label_gallery, loads, ls, peeks, saves, traverses
from .utils.constants import IMAGE_FILES, IMG_FILES, VIDEO_FILES
from .utils.globals import GLOBAL_TMP_DIR
from .utils.logger import UniLogger
```

## mkdocs.yml

- Characters: 4030
- Tokens: 0

```yaml
site_name: "unibox"
site_description: "unibox provides unified interface for common file operations"
site_url: "https://trojblue.github.io/unibox"
repo_url: "https://github.com/trojblue/unibox"
repo_name: "trojblue/unibox"
site_dir: "site"
watch: [mkdocs.yml, README.md, CONTRIBUTING.md, CHANGELOG.md, src/unibox]
copyright: Copyright &copy; 2025 trojblue
edit_uri: edit/main/docs/

validation:
  omitted_files: warn
  absolute_links: warn
  unrecognized_links: warn

nav:
- Home:
  - Overview: index.md
  - Changelog: changelog.md
  - Credits: credits.md
  - License: license.md
# defer to gen-files + literate-nav
- API reference:
  - unibox: reference/
- Development:
  - Contributing: contributing.md
  - Code of Conduct: code_of_conduct.md
  - Coverage report: coverage.md

theme:
  name: material
  custom_dir: docs/.overrides
  features:
  - announce.dismiss
  - content.action.edit
  - content.action.view
  - content.code.annotate
  - content.code.copy
  - content.tooltips
  - navigation.footer
  - navigation.indexes
  - navigation.sections
  - navigation.tabs
  - navigation.tabs.sticky
  - navigation.top
  - search.highlight
  - search.suggest
  - toc.follow
  palette:
  - media: "(prefers-color-scheme)"
    toggle:
      icon: material/brightness-auto
      name: Switch to light mode
  - media: "(prefers-color-scheme: light)"
    scheme: default
    primary: teal
    accent: purple
    toggle:
      icon: material/weather-sunny
      name: Switch to dark mode
  - media: "(prefers-color-scheme: dark)"
    scheme: slate
    primary: black
    accent: lime
    toggle:
      icon: material/weather-night
      name: Switch to system preference

extra_css:
- css/material.css
- css/mkdocstrings.css

extra_javascript:
- js/feedback.js

markdown_extensions:
- attr_list
- admonition
- callouts
- footnotes
- pymdownx.emoji:
    emoji_index: !!python/name:material.extensions.emoji.twemoji
    emoji_generator: !!python/name:material.extensions.emoji.to_svg
- pymdownx.magiclink
- pymdownx.snippets:
    base_path: [!relative $config_dir]
    check_paths: true
- pymdownx.superfences
- pymdownx.tabbed:
    alternate_style: true
    slugify: !!python/object/apply:pymdownx.slugs.slugify
      kwds:
        case: lower
- pymdownx.tasklist:
    custom_checkbox: true
- toc:
    permalink: true

plugins:
- search
- markdown-exec
- gen-files:
    scripts:
    - scripts/gen_ref_nav.py
- literate-nav:
    nav_file: SUMMARY.md
- coverage
- mkdocstrings:
    handlers:
      python:
        import:
        - https://docs.python.org/3/objects.inv
        paths: [src]
        options:
          docstring_options:
            ignore_init_summary: true
          docstring_section_style: list
          filters: ["!^_"]
          heading_level: 1
          inherited_members: true
          merge_init_into_class: true
          separate_signature: true
          show_root_heading: true
          show_root_full_path: false
          show_signature_annotations: true
          show_source: true
          show_symbol_type_heading: true
          show_symbol_type_toc: true
          signature_crossrefs: true
          summary: true
- git-revision-date-localized:
    enabled: !ENV [DEPLOY, false]
    enable_creation_date: true
    type: timeago
- minify:
    minify_html: !ENV [DEPLOY, false]
- group:
    enabled: !ENV [MATERIAL_INSIDERS, false]
    plugins:
    - typeset

extra:
  social:
  - icon: fontawesome/brands/github
    link: https://github.com/trojblue
  - icon: fontawesome/brands/gitter
    link: https://gitter.im/unibox/community
  - icon: fontawesome/brands/python
    link: https://pypi.org/project/unibox/
  analytics:
    feedback:
      title: Was this page helpful?
      ratings:
        - icon: material/emoticon-happy-outline
          name: This page was helpful
          data: 1
          note: Thanks for your feedback!
        - icon: material/emoticon-sad-outline
          name: This page could be improved
          data: 0
          note: Let us know how we can improve this page.
```

## .github/ISSUE_TEMPLATE/3-docs.md

- Characters: 600
- Tokens: 0

```markdown
---
name: Documentation update
about: Point at unclear, missing or outdated documentation.
title: "docs: "
labels: docs
assignees: pawamoy
---

### Is something unclear, missing or outdated in our documentation?
<!-- A clear and concise description of what the documentation issue is. Ex. I can't find an explanation on feature [...]. -->

### Relevant code snippets
<!-- If the documentation issue is related to code, please provide relevant code snippets. -->

### Link to the relevant documentation section
<!-- Add a link to the relevant section of our documentation, or any addition context. -->
```

## .github/ISSUE_TEMPLATE/1-bug.md

- Characters: 1470
- Tokens: 0

````markdown
---
name: Bug report
about: Create a bug report to help us improve.
title: "bug: "
labels: unconfirmed
assignees: [pawamoy]
---

### Description of the bug
<!-- Please provide a clear and concise description of what the bug is. -->

### To Reproduce
<!-- Please provide a Minimal Reproducible Example (MRE) if possible.
     Try to boil down the problem to a few lines of code.
     Your code should run by simply copying and pasting it.

     Example:

     ```
     git clone https://github.com/username/repro
     cd repro
     python -m venv .venv
     . .venv/bin/activate
     pip install -r requirements.txt
     ...  # command or code showing the issue
     ```
-->

```
WRITE MRE / INSTRUCTIONS HERE
```

### Full traceback
<!-- Please provide the full error message / traceback if any, by pasting it in the code block below.
     No screenshots! -->

<details><summary>Full traceback</summary>

```python
PASTE TRACEBACK HERE
```

</details>

### Expected behavior
<!-- Please provide a clear and concise description of what you expected to happen. -->

### Environment information
<!-- Please run the following command in your repository and paste its output below it,
     redacting sensitive information. -->

```bash
unibox --debug-info  # | xclip -selection clipboard
```

PASTE MARKDOWN OUTPUT HERE

### Additional context
<!-- Add any other relevant context about the problem here,
     like links to other issues or pull requests, screenshots, etc.
-->
````

## .github/ISSUE_TEMPLATE/config.yml

- Characters: 211
- Tokens: 0

```yaml
blank_issues_enabled: false
contact_links:
- name: I have a question / I need help
  url: https://github.com/trojblue/unibox/discussions/new?category=q-a
  about: Ask and answer questions in the Discussions tab.
```

## .github/ISSUE_TEMPLATE/4-change.md

- Characters: 597
- Tokens: 0

```markdown
---
name: Change request
about: Suggest any other kind of change for this project.
title: "change: "
assignees: pawamoy
---

### Is your change request related to a problem? Please describe.
<!-- A clear and concise description of what the problem is. -->

### Describe the solution you'd like
<!-- A clear and concise description of what you want to happen. -->

### Describe alternatives you've considered
<!-- A clear and concise description of any alternative solutions you've considered. -->

### Additional context
<!-- Add any other context or screenshots about the change request here. -->
```

## .github/ISSUE_TEMPLATE/2-feature.md

- Characters: 650
- Tokens: 0

```markdown
---
name: Feature request
about: Suggest an idea for this project.
title: "feature: "
labels: feature
assignees: pawamoy
---

### Is your feature request related to a problem? Please describe.
<!-- A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]. -->

### Describe the solution you'd like
<!-- A clear and concise description of what you want to happen. -->

### Describe alternatives you've considered
<!-- A clear and concise description of any alternative solutions or features you've considered. -->

### Additional context
<!-- Add any other context or screenshots about the feature request here. -->
```

## .github/FUNDING.yml

- Characters: 97
- Tokens: 0

```yaml
github: trojblue
# ko_fi: trojblue
# polar: trojblue
# custom:
# - https://www.paypal.me/trojblue
```

## .github/workflows/release.yml

- Characters: 651
- Tokens: 0

```yaml
name: release

on: push
permissions:
  contents: write

jobs:
  release:
    runs-on: ubuntu-latest
    if: startsWith(github.ref, 'refs/tags/')
    steps:
    - name: Checkout
      uses: actions/checkout@v4
      with:
        fetch-depth: 0
        fetch-tags: true
    - name: Setup Python
      uses: actions/setup-python@v5
      with:
        python-version: "3.12"
    - name: Setup uv
      uses: astral-sh/setup-uv@v3
    - name: Prepare release notes
      run: uv tool run git-changelog --release-notes > release-notes.md
    - name: Create release
      uses: softprops/action-gh-release@v2
      with:
        body_path: release-notes.md
```

## .github/workflows/ci.yml

- Characters: 2305
- Tokens: 0

```yaml
# name: ci

# on:
#   push:
#   pull_request:
#     branches:
#     - main

# defaults:
#   run:
#     shell: bash

# env:
#   LANG: en_US.utf-8
#   LC_ALL: en_US.utf-8
#   PYTHONIOENCODING: UTF-8
#   PYTHON_VERSIONS: ""

# jobs:

#   quality:

#     runs-on: ubuntu-latest

#     steps:
#     - name: Checkout
#       uses: actions/checkout@v4
#       with:
#         fetch-depth: 0
#         fetch-tags: true

#     - name: Setup Python
#       uses: actions/setup-python@v5
#       with:
#         python-version: "3.12"

#     - name: Setup uv
#       uses: astral-sh/setup-uv@v3
#       with:
#         enable-cache: true
#         cache-dependency-glob: pyproject.toml

#     - name: Install dependencies
#       run: make setup

#     - name: Check if the documentation builds correctly
#       run: make check-docs

#     - name: Check the code quality
#       run: make check-quality

#     - name: Check if the code is correctly typed
#       run: make check-types

#     - name: Check for breaking changes in the API
#       run: make check-api

#   tests:

#     strategy:
#       matrix:
#         os:
#         - ubuntu-latest
#         - macos-latest
#         - windows-latest
#         python-version:
#         - "3.10"
#         - "3.11"
#         - "3.12"
#         - "3.13"
#         resolution:
#         - highest
#         - lowest-direct
#         exclude:
#         - os: macos-latest
#           resolution: lowest-direct
#         - os: windows-latest
#           resolution: lowest-direct
#     runs-on: ${{ matrix.os }}
#     continue-on-error: ${{ matrix.python-version == '3.14' }}

#     steps:
#     - name: Checkout
#       uses: actions/checkout@v4
#       with:
#         fetch-depth: 0
#         fetch-tags: true

#     - name: Setup Python
#       uses: actions/setup-python@v5
#       with:
#         python-version: ${{ matrix.python-version }}
#         allow-prereleases: true

#     - name: Setup uv
#       uses: astral-sh/setup-uv@v3
#       with:
#         enable-cache: true
#         cache-dependency-glob: pyproject.toml
#         cache-suffix: py${{ matrix.python-version }}

#     - name: Install dependencies
#       env:
#         UV_RESOLUTION: ${{ matrix.resolution }}
#       run: make setup

#     - name: Run the test suite
#       run: make test
```

## .copier-answers.yml

- Characters: 623
- Tokens: 0

```yaml
# Changes here will be overwritten by Copier
_commit: 1.5.6
_src_path: gh:pawamoy/copier-uv
author_email: trojblue@gmail.com
author_fullname: trojblue
author_username: trojblue
copyright_date: '2025'
copyright_holder: trojblue
copyright_holder_email: trojblue@gmail.com
copyright_license: MIT License
insiders: false
project_description: unibox provides unified interface for common file operations
project_name: unibox
python_package_command_line_name: unibox
python_package_distribution_name: unibox
python_package_import_name: unibox
repository_name: unibox
repository_namespace: trojblue
repository_provider: github.com
```

## Makefile

- Characters: 495
- Tokens: 0

```text
# If you have `direnv` loaded in your shell, and allow it in the repository,
# the `make` command will point at the `scripts/make` shell script.
# This Makefile is just here to allow auto-completion in the terminal.

actions = \
	allrun \
	changelog \
	check \
	check-api \
	check-docs \
	check-quality \
	check-types \
	clean \
	coverage \
	docs \
	docs-deploy \
	format \
	help \
	multirun \
	release \
	run \
	setup \
	test \
	vscode

.PHONY: $(actions)
$(actions):
	@python scripts/make "$@"
```

## README.md

- Characters: 2071
- Tokens: 0

````markdown
# unibox

[![ci](https://github.com/trojblue/unibox/workflows/ci/badge.svg)](https://github.com/trojblue/unibox/actions?query=workflow%3Aci)
[![documentation](https://img.shields.io/badge/docs-mkdocs-708FCC.svg?style=flat)](https://trojblue.github.io/unibox/)
[![pypi version](https://img.shields.io/pypi/v/unibox.svg)](https://pypi.org/project/unibox/)
[![gitter](https://badges.gitter.im/join%20chat.svg)](https://app.gitter.im/#/room/#unibox:gitter.im)

unibox provides unified interface for common file operations

## Installation

```bash
pip install unibox
```

With [`uv`](https://docs.astral.sh/uv/):

```bash
uv tool install unibox
```

If you're not using python 3.13, it's also recommended to install `pandas[performance]`:

```bash
pip install "pandas[performance]"
```


to update or remove project dependencies:

```bash

uv add requests

uv remove requests

# after adding new package: rerun
make setup
```


## Usage

import the lib:

```python
import unibox as ub
```


## Using Huggingface Backend

you can load and use a huggingface dataset directly with `hf://{username}/{daataset_repo}`:

```python
hf_dset = ub.loads("hf://incantor/aesthetic_eagle_5category_iter99")
df = hf_dset.to_pandas()
```

and upload a processed dataframe back to huggingface:

```python
df["new_col"] = "new changes"
ub.saves(df, "hf://datatmp/updated_repo")
```


## Dev notes

current concerns:

1. loads(): temp files could accumulate on global dir, and take up all of /tmp/; also concurrency issues
2. s3_backend: only one that takes a dir; should make others do the same

to get a coverage report, run:
```bash
pytest --cov=src/unibox --cov-report=term-missing tests
```

To build the docs:

```bash
make docs host=0.0.0.0

# or in debug mode:
make check-docs
```

to manual make a release:
```bash
# python -m pip install build twine
python -m build
twine check dist/*
twine upload dist/*
```


migrating from unibox 0.4

no longer supported:

- `ub.traverses()`: removed handlers and `exclude_extensions` (`include_extensions` still works but depreciated with `exts`)
````

## CONTRIBUTING.md

- Characters: 4222
- Tokens: 0

````markdown
# Contributing

Contributions are welcome, and they are greatly appreciated!
Every little bit helps, and credit will always be given.

## Environment setup

Nothing easier!

Fork and clone the repository, then:

```bash
cd unibox
make setup
```

> NOTE:
> If it fails for some reason,
> you'll need to install
> [uv](https://github.com/astral-sh/uv)
> manually.
>
> You can install it with:
>
> ```bash
> curl -LsSf https://astral.sh/uv/install.sh | sh
> ```
>
> Now you can try running `make setup` again,
> or simply `uv sync`.

You now have the dependencies installed.

You can run the application with `make run unibox [ARGS...]`.

Run `make help` to see all the available actions!

## Tasks

The entry-point to run commands and tasks is the `make` Python script,
located in the `scripts` directory. Try running `make` to show the available commands and tasks.
The *commands* do not need the Python dependencies to be installed,
while the *tasks* do.
The cross-platform tasks are written in Python, thanks to [duty](https://github.com/pawamoy/duty).

If you work in VSCode, we provide
[an action to configure VSCode](https://pawamoy.github.io/copier-uv/work/#vscode-setup)
for the project.

## Development

As usual:

1. create a new branch: `git switch -c feature-or-bugfix-name`
1. edit the code and/or the documentation

**Before committing:**

1. run `make format` to auto-format the code
1. run `make check` to check everything (fix any warning)
1. run `make test` to run the tests (fix any issue)
1. if you updated the documentation or the project dependencies:
    1. run `make docs`
    1. go to http://localhost:8000 and check that everything looks good
1. follow our [commit message convention](#commit-message-convention)

If you are unsure about how to fix or ignore a warning,
just let the continuous integration fail,
and we will help you during review.

Don't bother updating the changelog, we will take care of this.

## Commit message convention

Commit messages must follow our convention based on the
[Angular style](https://gist.github.com/stephenparish/9941e89d80e2bc58a153#format-of-the-commit-message)
or the [Karma convention](https://karma-runner.github.io/4.0/dev/git-commit-msg.html):

```
<type>[(scope)]: Subject

[Body]
```

**Subject and body must be valid Markdown.**
Subject must have proper casing (uppercase for first letter
if it makes sense), but no dot at the end, and no punctuation
in general.

Scope and body are optional. Type can be:

- `build`: About packaging, building wheels, etc.
- `chore`: About packaging or repo/files management.
- `ci`: About Continuous Integration.
- `deps`: Dependencies update.
- `docs`: About documentation.
- `feat`: New feature.
- `fix`: Bug fix.
- `perf`: About performance.
- `refactor`: Changes that are not features or bug fixes.
- `style`: A change in code style/format.
- `tests`: About tests.

If you write a body, please add trailers at the end
(for example issues and PR references, or co-authors),
without relying on GitHub's flavored Markdown:

```
Body.

Issue #10: https://github.com/namespace/project/issues/10
Related to PR namespace/other-project#15: https://github.com/namespace/other-project/pull/15
```

These "trailers" must appear at the end of the body,
without any blank lines between them. The trailer title
can contain any character except colons `:`.
We expect a full URI for each trailer, not just GitHub autolinks
(for example, full GitHub URLs for commits and issues,
not the hash or the #issue-number).

We do not enforce a line length on commit messages summary and body,
but please avoid very long summaries, and very long lines in the body,
unless they are part of code blocks that must not be wrapped.

## Pull requests guidelines

Link to any related issue in the Pull Request message.

During the review, we recommend using fixups:

```bash
# SHA is the SHA of the commit you want to fix
git commit --fixup=SHA
```

Once all the changes are approved, you can squash your commits:

```bash
git rebase -i --autosquash main
```

And force-push:

```bash
git push -f
```

If this seems all too complicated, you can push or force-push each new commit,
and we will squash them ourselves if needed, before merging.
````

## duties.py

- Characters: 6100
- Tokens: 0

```python
"""Development tasks."""

from __future__ import annotations

import os
import sys
from contextlib import contextmanager
from importlib.metadata import version as pkgversion
from pathlib import Path
from typing import TYPE_CHECKING

from duty import duty, tools

if TYPE_CHECKING:
    from collections.abc import Iterator

    from duty.context import Context


PY_SRC_PATHS = (Path(_) for _ in ("src", "tests", "duties.py", "scripts"))
PY_SRC_LIST = tuple(str(_) for _ in PY_SRC_PATHS)
PY_SRC = " ".join(PY_SRC_LIST)
CI = os.environ.get("CI", "0") in {"1", "true", "yes", ""}
WINDOWS = os.name == "nt"
PTY = not WINDOWS and not CI
MULTIRUN = os.environ.get("MULTIRUN", "0") == "1"


def pyprefix(title: str) -> str:  # noqa: D103
    if MULTIRUN:
        prefix = f"(python{sys.version_info.major}.{sys.version_info.minor})"
        return f"{prefix:14}{title}"
    return title


@contextmanager
def material_insiders() -> Iterator[bool]:  # noqa: D103
    if "+insiders" in pkgversion("mkdocs-material"):
        os.environ["MATERIAL_INSIDERS"] = "true"
        try:
            yield True
        finally:
            os.environ.pop("MATERIAL_INSIDERS")
    else:
        yield False


@duty
def changelog(ctx: Context, bump: str = "") -> None:
    """Update the changelog in-place with latest commits.

    Parameters:
        bump: Bump option passed to git-changelog.
    """
    ctx.run(tools.git_changelog(bump=bump or None), title="Updating changelog")


@duty(pre=["check-quality", "check-types", "check-docs", "check-api"])
def check(ctx: Context) -> None:
    """Check it all!"""


@duty
def check_quality(ctx: Context) -> None:
    """Check the code quality."""
    ctx.run(
        tools.ruff.check(*PY_SRC_LIST, config="config/ruff.toml"),
        title=pyprefix("Checking code quality"),
    )


@duty
def check_docs(ctx: Context) -> None:
    """Check if the documentation builds correctly."""
    Path("htmlcov").mkdir(parents=True, exist_ok=True)
    Path("htmlcov/index.html").touch(exist_ok=True)
    with material_insiders():
        ctx.run(
            tools.mkdocs.build(strict=True, verbose=True),
            title=pyprefix("Building documentation"),
        )


@duty
def check_types(ctx: Context) -> None:
    """Check that the code is correctly typed."""
    ctx.run(
        tools.mypy(*PY_SRC_LIST, config_file="config/mypy.ini"),
        title=pyprefix("Type-checking"),
    )


@duty
def check_api(ctx: Context, *cli_args: str) -> None:
    """Check for API breaking changes."""
    ctx.run(
        tools.griffe.check("unibox", search=["src"], color=True).add_args(*cli_args),
        title="Checking for API breaking changes",
        nofail=True,
    )


@duty
def docs(ctx: Context, *cli_args: str, host: str = "127.0.0.1", port: int = 8000) -> None:
    """Serve the documentation (localhost:8000).

    Parameters:
        host: The host to serve the docs from.
        port: The port to serve the docs on.
    """
    with material_insiders():
        ctx.run(
            tools.mkdocs.serve(dev_addr=f"{host}:{port}").add_args(*cli_args),
            title="Serving documentation",
            capture=False,
        )


@duty
def docs_deploy(ctx: Context) -> None:
    """Deploy the documentation to GitHub pages."""
    os.environ["DEPLOY"] = "true"
    with material_insiders() as insiders:
        if not insiders:
            ctx.run(lambda: False, title="Not deploying docs without Material for MkDocs Insiders!")
        ctx.run(tools.mkdocs.gh_deploy(), title="Deploying documentation")


@duty
def format(ctx: Context) -> None:
    """Run formatting tools on the code."""
    ctx.run(
        tools.ruff.check(*PY_SRC_LIST, config="config/ruff.toml", fix_only=True, exit_zero=True),
        title="Auto-fixing code",
    )
    ctx.run(tools.ruff.format(*PY_SRC_LIST, config="config/ruff.toml"), title="Formatting code")


@duty
def build(ctx: Context) -> None:
    """Build source and wheel distributions."""
    ctx.run(
        tools.build(),
        title="Building source and wheel distributions",
        pty=PTY,
    )


@duty
def publish(ctx: Context) -> None:
    """Publish source and wheel distributions to PyPI."""
    if not Path("dist").exists():
        ctx.run("false", title="No distribution files found")
    dists = [str(dist) for dist in Path("dist").iterdir()]
    ctx.run(
        tools.twine.upload(*dists, skip_existing=True),
        title="Publishing source and wheel distributions to PyPI",
        pty=PTY,
    )


@duty(post=["build", "publish", "docs-deploy"])
def release(ctx: Context, version: str = "") -> None:
    """Release a new Python package.

    Parameters:
        version: The new version number to use.
    """
    if not (version := (version or input("> Version to release: ")).strip()):
        ctx.run("false", title="A version must be provided")
    ctx.run("git add pyproject.toml CHANGELOG.md", title="Staging files", pty=PTY)
    ctx.run(["git", "commit", "-m", f"chore: Prepare release {version}"], title="Committing changes", pty=PTY)
    ctx.run(f"git tag {version}", title="Tagging commit", pty=PTY)
    ctx.run("git push", title="Pushing commits", pty=False)
    ctx.run("git push --tags", title="Pushing tags", pty=False)


@duty(silent=True, aliases=["cov"])
def coverage(ctx: Context) -> None:
    """Report coverage as text and HTML."""
    ctx.run(tools.coverage.combine(), nofail=True)
    ctx.run(tools.coverage.report(rcfile="config/coverage.ini"), capture=False)
    ctx.run(tools.coverage.html(rcfile="config/coverage.ini"))


@duty
def test(ctx: Context, *cli_args: str, match: str = "") -> None:
    """Run the test suite.

    Parameters:
        match: A pytest expression to filter selected tests.
    """
    py_version = f"{sys.version_info.major}{sys.version_info.minor}"
    os.environ["COVERAGE_FILE"] = f".coverage.{py_version}"
    ctx.run(
        tools.pytest(
            "tests",
            config_file="config/pytest.ini",
            select=match,
            color="yes",
        ).add_args("-n", "auto", *cli_args),
        title=pyprefix("Running tests"),
    )
```

## .gitignore

- Characters: 241
- Tokens: 0

```text
repomix-output.md
*.log

# editors
.idea/
.vscode/

# python
*.egg-info/
*.py[cod]
.venv/
.venvs/
/build/
/dist/

# tools
.coverage*
/.pdm-build/
/htmlcov/
/site/
uv.lock

# cache
.cache/
.pytest_cache/
.mypy_cache/
.ruff_cache/
__pycache__/
```

## config/coverage.ini

- Characters: 370
- Tokens: 0

```text
[coverage:run]
branch = true
parallel = true
source =
  src/
  tests/

[coverage:paths]
equivalent =
  src/
  .venv/lib/*/site-packages/
  .venvs/*/lib/*/site-packages/

[coverage:report]
precision = 2
omit =
  src/*/__init__.py
  src/*/__main__.py
  tests/__init__.py
exclude_lines =
  pragma: no cover
  if TYPE_CHECKING

[coverage:json]
output = htmlcov/coverage.json
```

## config/mypy.ini

- Characters: 113
- Tokens: 0

```text
[mypy]
ignore_missing_imports = true
exclude = tests/fixtures/
warn_unused_ignores = true
show_error_codes = true
```

## config/vscode/launch.json

- Characters: 1151
- Tokens: 0

```json
{
    "version": "0.2.0",
    "configurations": [
        {
            "name": "python (current file)",
            "type": "debugpy",
            "request": "launch",
            "program": "${file}",
            "console": "integratedTerminal",
            "justMyCode": false
        },
        {
            "name": "docs",
            "type": "debugpy",
            "request": "launch",
            "module": "mkdocs",
            "justMyCode": false,
            "args": [
                "serve",
                "-v"
            ]
        },
        {
            "name": "test",
            "type": "debugpy",
            "request": "launch",
            "module": "pytest",
            "justMyCode": false,
            "args": [
                "-c=config/pytest.ini",
                "-vvv",
                "--no-cov",
                "--dist=no",
                "tests",
                "-k=${input:tests_selection}"
            ]
        }
    ],
    "inputs": [
        {
            "id": "tests_selection",
            "type": "promptString",
            "description": "Tests selection",
            "default": ""
        }
    ]
}
```

## config/vscode/tasks.json

- Characters: 2437
- Tokens: 0

```json
{
    "version": "2.0.0",
    "tasks": [
        {
            "label": "changelog",
            "type": "process",
            "command": "scripts/make",
            "args": ["changelog"]
        },
        {
            "label": "check",
            "type": "process",
            "command": "scripts/make",
            "args": ["check"]
        },
        {
            "label": "check-quality",
            "type": "process",
            "command": "scripts/make",
            "args": ["check-quality"]
        },
        {
            "label": "check-types",
            "type": "process",
            "command": "scripts/make",
            "args": ["check-types"]
        },
        {
            "label": "check-docs",
            "type": "process",
            "command": "scripts/make",
            "args": ["check-docs"]
        },
        {
            "label": "check-api",
            "type": "process",
            "command": "scripts/make",
            "args": ["check-api"]
        },
        {
            "label": "clean",
            "type": "process",
            "command": "scripts/make",
            "args": ["clean"]
        },
        {
            "label": "docs",
            "type": "process",
            "command": "scripts/make",
            "args": ["docs"]
        },
        {
            "label": "docs-deploy",
            "type": "process",
            "command": "scripts/make",
            "args": ["docs-deploy"]
        },
        {
            "label": "format",
            "type": "process",
            "command": "scripts/make",
            "args": ["format"]
        },
        {
            "label": "release",
            "type": "process",
            "command": "scripts/make",
            "args": ["release", "${input:version}"]
        },
        {
            "label": "setup",
            "type": "process",
            "command": "scripts/make",
            "args": ["setup"]
        },
        {
            "label": "test",
            "type": "process",
            "command": "scripts/make",
            "args": ["test", "coverage"],
            "group": "test"
        },
        {
            "label": "vscode",
            "type": "process",
            "command": "scripts/make",
            "args": ["vscode"]
        }
    ],
    "inputs": [
        {
            "id": "version",
            "type": "promptString",
            "description": "Version"
        }
    ]
}
```

## config/vscode/settings.json

- Characters: 964
- Tokens: 0

```json
{
    "files.watcherExclude": {
        "**/.venv*/**": true,
        "**/.venvs*/**": true,
        "**/venv*/**": true
    },
    "mypy-type-checker.args": [
        "--config-file=config/mypy.ini"
    ],
    "python.testing.unittestEnabled": false,
    "python.testing.pytestEnabled": true,
    "python.testing.pytestArgs": [
        "--config-file=config/pytest.ini"
    ],
    "ruff.enable": true,
    "ruff.format.args": [
        "--config=config/ruff.toml"
    ],
    "ruff.lint.args": [
        "--config=config/ruff.toml"
    ],
    "yaml.schemas": {
        "https://squidfunk.github.io/mkdocs-material/schema.json": "mkdocs.yml"
    },
    "yaml.customTags": [
        "!ENV scalar",
        "!ENV sequence",
        "!relative scalar",
        "tag:yaml.org,2002:python/name:materialx.emoji.to_svg",
        "tag:yaml.org,2002:python/name:materialx.emoji.twemoji",
        "tag:yaml.org,2002:python/name:pymdownx.superfences.fence_code_format"
    ]
}
```

## config/pytest.ini

- Characters: 470
- Tokens: 0

```text
[pytest]
python_files =
  test_*.py
addopts =
  --cov
  --cov-config config/coverage.ini
  --cov-report=term-missing
testpaths =
  tests

# action:message_regex:warning_class:module_regex:line
filterwarnings =
  error
  # TODO: remove once pytest-xdist 4 is released
  ignore:.*rsyncdir:DeprecationWarning:xdist

  # ignore the deprecation about datetime.datetime.utcnow in botocore
  ignore:datetime\.datetime\.utcnow\(\) is deprecated.*:DeprecationWarning:botocore\..*
```

## config/git-changelog.toml

- Characters: 223
- Tokens: 0

```text
bump = "auto"
convention = "angular"
in-place = true
output = "CHANGELOG.md"
parse-refs = false
parse-trailers = true
sections = ["build", "deps", "feat", "fix", "refactor"]
template = "keepachangelog"
versioning = "pep440"
```

## config/ruff.toml

- Characters: 2121
- Tokens: 0

```text
target-version = "py39"
line-length = 120

[lint]
exclude = [
    "tests/fixtures/*.py",
]
select = [
    "A", "ANN", "ARG",
    "B", "BLE",
    "C", "C4",
    "COM",
    "D", "DTZ",
    "E", "ERA", "EXE",
    "F", "FBT",
    "G",
    "I", "ICN", "INP", "ISC",
    "N",
    "PGH", "PIE", "PL", "PLC", "PLE", "PLR", "PLW", "PT", "PYI",
    "Q",
    "RUF", "RSE", "RET",
    "S", "SIM", "SLF",
    "T", "T10", "T20", "TCH", "TID", "TRY",
    "UP",
    "W",
    "YTT",
]
ignore = [
    "A001",  # Variable is shadowing a Python builtin
    "ANN101",  # Missing type annotation for self
    "ANN102",  # Missing type annotation for cls
    "ANN204",  # Missing return type annotation for special method __str__
    "ANN401",  # Dynamically typed expressions (typing.Any) are disallowed
    "ARG005",  # Unused lambda argument
    "C901",  # Too complex
    "D105",  # Missing docstring in magic method
    "D417",  # Missing argument description in the docstring
    "E501",  # Line too long
    "ERA001",  # Commented out code
    "G004",  # Logging statement uses f-string
    "PLR0911",  # Too many return statements
    "PLR0912",  # Too many branches
    "PLR0913",  # Too many arguments to function call
    "PLR0915",  # Too many statements
    "SLF001", # Private member accessed
    "TRY003",  # Avoid specifying long messages outside the exception class
]

[lint.per-file-ignores]
"src/*/cli.py" = [
    "T201",  # Print statement
]
"src/*/debug.py" = [
    "T201",  # Print statement
]
"scripts/*.py" = [
    "INP001",  # File is part of an implicit namespace package
    "T201",  # Print statement
]
"tests/*.py" = [
    "ARG005",  # Unused lambda argument
    "FBT001",  # Boolean positional arg in function definition
    "PLR2004",  # Magic value used in comparison
    "S101",  # Use of assert detected
]

[lint.flake8-quotes]
docstring-quotes = "double"

[lint.flake8-tidy-imports]
ban-relative-imports = "all"

[lint.isort]
known-first-party = ["unibox"]

[lint.pydocstyle]
convention = "google"

[format]
exclude = [
    "tests/fixtures/*.py",
]
docstring-code-format = true
docstring-code-line-length = 80
```

## tests/test_cli.py

- Characters: 1212
- Tokens: 0

```python
"""Tests for the `cli` module."""

from __future__ import annotations

import pytest

from unibox import cli, debug


def test_main() -> None:
    """Basic CLI test."""
    assert cli.main([]) == 0


def test_show_help(capsys: pytest.CaptureFixture) -> None:
    """Show help.

    Parameters:
        capsys: Pytest fixture to capture output.
    """
    with pytest.raises(SystemExit):
        cli.main(["-h"])
    captured = capsys.readouterr()
    assert "unibox" in captured.out


def test_show_version(capsys: pytest.CaptureFixture) -> None:
    """Show version.

    Parameters:
        capsys: Pytest fixture to capture output.
    """
    with pytest.raises(SystemExit):
        cli.main(["-V"])
    captured = capsys.readouterr()
    assert debug.get_version() in captured.out


def test_show_debug_info(capsys: pytest.CaptureFixture) -> None:
    """Show debug information.

    Parameters:
        capsys: Pytest fixture to capture output.
    """
    with pytest.raises(SystemExit):
        cli.main(["--debug-info"])
    captured = capsys.readouterr().out.lower()
    assert "python" in captured
    assert "system" in captured
    assert "environment" in captured
    assert "packages" in captured
```

## tests/README.md

- Characters: 128
- Tokens: 0

````markdown
# Pytests


to run test with `pytest` directly:

(in root dir):

```bash
pytest tests
```

to add a new test see test_loaders.py
````

## tests/conftest.py

- Characters: 46
- Tokens: 0

```python
"""Configuration for the pytest test suite."""
```

## tests/test_files/sample.toml

- Characters: 640
- Tokens: 0

```text
name = "Anime Rating System"
version = 1.0

[[categories]]
name = "Proportions"
description = "Evaluates the correctness of body proportions."
weight = 0.4

[[categories]]
name = "Pose Accuracy"
description = "Checks if the pose follows realistic constraints."
weight = 0.3

[[categories]]
name = "Structural Consistency"
description = "Ensures body parts are correctly connected."
weight = 0.3

[[raters]]
id = 101
name = "RaterA"
availability = "high"
assigned_images = 2000

[[raters]]
id = 102
name = "RaterB"
availability = "medium"
assigned_images = 1000

[[raters]]
id = 103
name = "RaterC"
availability = "low"
assigned_images = 500
```

## tests/test_files/sample.json

- Characters: 947
- Tokens: 0

```json
{
    "quiz": {
        "sport": {
            "q1": {
                "question": "Which one is correct team name in NBA?",
                "options": [
                    "New York Bulls",
                    "Los Angeles Kings",
                    "Golden State Warriros",
                    "Huston Rocket"
                ],
                "answer": "Huston Rocket"
            }
        },
        "maths": {
            "q1": {
                "question": "5 + 7 = ?",
                "options": [
                    "10",
                    "11",
                    "12",
                    "13"
                ],
                "answer": "12"
            },
            "q2": {
                "question": "12 - 8 = ?",
                "options": [
                    "1",
                    "2",
                    "3",
                    "4"
                ],
                "answer": "4"
            }
        }
    }
}
```

## tests/test_files/sample.txt

- Characters: 39
- Tokens: 0

```text
This
is a text file
with multiple lines
```

## tests/test_files/sample.yaml

- Characters: 637
- Tokens: 0

```yaml
name: "Anime Rating System"
version: 1.0
categories:
  - name: "Proportions"
    description: "Evaluates the correctness of body proportions."
    weight: 0.4
  - name: "Pose Accuracy"
    description: "Checks if the pose follows realistic constraints."
    weight: 0.3
  - name: "Structural Consistency"
    description: "Ensures body parts are correctly connected."
    weight: 0.3
raters:
  - id: 101
    name: "RaterA"
    availability: "high"
    assigned_images: 2000
  - id: 102
    name: "RaterB"
    availability: "medium"
    assigned_images: 1000
  - id: 103
    name: "RaterC"
    availability: "low"
    assigned_images: 500
```

## tests/test_files/sample.jsonl

- Characters: 223
- Tokens: 0

```text
{"name": "Gilbert", "wins": [["straight", "7♣"], ["one pair", "10♥"]]}
{"name": "Alexa", "wins": [["two pair", "4♠"], ["two pair", "9♠"]]}
{"name": "May", "wins": []}
{"name": "Deloise", "wins": [["three of a kind", "5♣"]]}
```

## tests/backends/test_local_backend.py

- Characters: 3718
- Tokens: 0

```python
from pathlib import Path

import pytest

import unibox as ub


@pytest.fixture
def test_folder(tmp_path):
    """Creates a temporary test folder with sample files for testing."""
    # Create test folder structure
    test_dir = tmp_path / "test_files"
    test_dir.mkdir()

    # Add sample files in the root directory
    extensions = ["jpg", "json", "parquet"]
    file_counts = {"jpg": 6, "json": 3, "parquet": 1}
    for ext, count in file_counts.items():
        for i in range(count):
            (test_dir / f"file_{i}.{ext}").touch()

    # Add a nested directory with a sample file
    nested_dir = test_dir / "nested"
    nested_dir.mkdir()
    (nested_dir / "nested_file.jpg").touch()

    return test_dir


@pytest.mark.parametrize("relative_unix, expected_relative", [(False, False), (True, True)])
def test_ls_paths(test_folder, relative_unix, expected_relative):
    """Test that ls() returns the correct paths (absolute or relative based on `relative_unix`)."""
    files = ub.ls(test_folder, relative_unix=relative_unix)

    # Ensure all paths are valid and have the correct type (absolute/relative)
    for file in files:
        path = Path(file)
        if expected_relative:
            assert not path.is_absolute(), f"Path {file} should be relative"
            if "nested/" in file:
                assert "/" in file, f"Nested path {file} should use Unix-style slashes"
        else:
            assert path.is_absolute(), f"Path {file} should be absolute"


def test_ls_exts(test_folder):
    """Test that ls() filters files correctly by extensions."""
    # Filter for jpg and json files
    files = ub.ls(test_folder, exts=["jpg", "json"])

    # Ensure all files have the correct extensions
    allowed_exts = {".jpg", ".json"}
    for file in files:
        assert Path(file).suffix in allowed_exts, f"File {file} has an unexpected extension"

    # Ensure the total number of files is correct
    assert len(files) == 6 + 3 + 1, "Incorrect number of files returned for extensions jpg and json"


def test_ls_backward_compatibility(test_folder):
    """Test that traverses() raises a DeprecationWarning and returns the same result as ls()."""
    with pytest.warns(DeprecationWarning):
        files_old = ub.traverses(test_folder, relative_unix=True)

    files_new = ub.ls(test_folder, relative_unix=True)

    # Ensure the results are identical
    assert files_old == files_new, "Results from traverses() and ls() should match"


def test_ls_absolute_paths(test_folder):
    """Test that ls() returns absolute paths when relative_unix=False."""
    files = ub.ls(test_folder, relative_unix=False)

    # Ensure all paths are absolute
    for file in files:
        assert Path(file).is_absolute(), f"Path {file} is not absolute"


def test_ls_relative_unix(test_folder):
    """Test that ls() returns relative Unix-style paths when relative_unix=True."""
    files = ub.ls(test_folder, relative_unix=True)

    # Ensure all paths are relative and use Unix-style slashes
    for file in files:
        path = Path(file)
        assert not path.is_absolute(), f"Path {file} should be relative"
        if "nested/" in file:
            assert "/" in file, f"Nested path {file} should use Unix-style slashes"


def test_ls_with_exts(test_folder):
    """Test that ls() correctly filters by extensions."""
    # Filter only for jpg files
    files = ub.ls(test_folder, exts=["jpg"])

    # Ensure all returned files have the correct extension
    for file in files:
        assert file.endswith(".jpg"), f"File {file} does not have the expected extension"

    # Ensure the correct number of jpg files is returned
    assert len(files) == 6 + 1, "Incorrect number of jpg files returned"
```

## tests/backends/test_s3_backend.py

- Characters: 178
- Tokens: 0

```python
import pytest


# Leave placeholders for S3-based tests
@pytest.mark.skip(reason="S3 tests not implemented yet.")
def test_ls_s3():
    """Placeholder for S3-based ls() tests."""
```

## tests/backends/__init__.py

- Characters: 0
- Tokens: 0

```python

```

## tests/test_loaders.py

- Characters: 2765
- Tokens: 0

```python
import os

import pytest

import unibox as ub


@pytest.mark.parametrize(
    "uri, expected_non_empty, is_image",
    [
        # Local file tests
        ("tests/test_files/sample.csv", True, False),
        ("tests/test_files/sample.parquet", True, False),
        ("tests/test_files/sample.txt", True, False),
        pytest.param(
            "tests/test_files/sample.jpg",
            False,
            True,
            marks=pytest.mark.filterwarnings("ignore::ResourceWarning"),
        ),
        ("tests/test_files/sample.jsonl", True, False),
        ("tests/test_files/sample.json", True, False),
        # HuggingFace dataset scenario
        pytest.param(
            "hf://incantor/aesthetic_eagle_5category_iter99",
            True,
            False,
            id="HuggingFace Dataset",
        ),
        # S3 file scenario
        pytest.param(
            "s3://bucket-external/misc/yada_store/configs/clip_prompts_list_full_v2.txt",
            True,
            False,
            id="S3 Text File",
        ),
    ],
)
def test_loads_and_saves(uri: str, expected_non_empty, is_image, tmp_path):
    """Test loader functionality for various URIs, including saving and reloading.

    Parameters:
        uri: The URI to load the file or dataset.
        expected_non_empty: Whether the loaded result should be non-empty.
        is_image: Whether the file is an image.
        tmp_path: Temporary directory provided by pytest for saving test files.
    """
    # Load the URI
    loaded_data = ub.loads(uri)

    # Validate the loaded content
    if expected_non_empty:
        assert loaded_data is not None, f"Loaded data from {uri} is None"
        if not is_image:
            assert len(loaded_data) > 0, f"Loaded data from {uri} is empty"

    # Save the loaded content to a new location
    uri = str(uri)
    if uri.startswith("hf://"):
        save_uri = "hf://datatmp/unibox-debug-repo"
    elif uri.startswith("s3://"):
        save_uri = "s3://dataset-ingested/temp/unibox_debug/" + os.path.basename(uri)
    else:
        save_uri = tmp_path / f"saved_{os.path.basename(uri)}"

    save_uri = str(save_uri)
    ub.saves(loaded_data, save_uri)

    # Ensure the saved file or dataset exists
    if not save_uri.startswith("hf://") and not save_uri.startswith("s3://"):
        assert os.path.exists(save_uri), f"Saved file {save_uri} does not exist"
        assert os.path.getsize(save_uri) > 0, f"Saved file {save_uri} is empty"

    # Reload the saved file or dataset
    reloaded_data = ub.loads(save_uri)

    # Validate the reloaded content
    assert reloaded_data is not None, f"Reloaded data from {save_uri} is None"
    if not is_image:
        assert len(reloaded_data) > 0, f"Reloaded data from {save_uri} is empty"
```

## tests/__init__.py

- Characters: 158
- Tokens: 0

```python
"""Tests suite for `unibox`."""

from pathlib import Path

TESTS_DIR = Path(__file__).parent
TMP_DIR = TESTS_DIR / "tmp"
FIXTURES_DIR = TESTS_DIR / "fixtures"
```

## .envrc

- Characters: 16
- Tokens: 0

```text
PATH_add scripts
```

## Statistics

- Total Files: 83
- Total Characters: 150969
- Total Tokens: 0
