Metadata-Version: 2.2
Name: datamate
Version: 1.0.0
Summary: A data organization and compilation system.
License: MIT License
        
        Copyright (c) 2022 Janne Lappalainen
        Copyright (c) 2018 Mason McGill
        
        Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
        
        The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
        
        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
        
Project-URL: Homepage, https://github.com/flyvis/datamate
Project-URL: Bug Tracker, https://github.com/flyvis/datamate/issues
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: Topic :: Scientific/Engineering
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Development Status :: 3 - Alpha
Requires-Python: <3.13,>=3.9
Description-Content-Type: text/markdown
Requires-Dist: numpy
Requires-Dist: pandas
Requires-Dist: typing_extensions
Requires-Dist: h5py>=3.6.0
Requires-Dist: ruamel.yaml
Requires-Dist: PyYAML
Provides-Extra: dev
Requires-Dist: pre-commit; extra == "dev"
Requires-Dist: jupyter; extra == "dev"
Requires-Dist: setuptools_scm; extra == "dev"
Requires-Dist: pytest; extra == "dev"
Requires-Dist: ruff; extra == "dev"
Requires-Dist: pytest-cov; extra == "dev"
Provides-Extra: docs
Requires-Dist: mkdocs; extra == "docs"
Requires-Dist: mkdocs-material; extra == "docs"
Requires-Dist: markdown-include; extra == "docs"
Requires-Dist: mkdocs-redirects; extra == "docs"
Requires-Dist: mkdocstrings[python]>=0.18; extra == "docs"
Requires-Dist: nbconvert; extra == "docs"
Requires-Dist: mike; extra == "docs"
Requires-Dist: mkdocs-macros-plugin; extra == "docs"
Requires-Dist: jupyter; extra == "docs"
Requires-Dist: tabulate; extra == "docs"
Requires-Dist: tqdm; extra == "docs"
Requires-Dist: ipywidgets; extra == "docs"

# Datamate

Datamate is a data and configuration management framework in Python for machine-learning research. It uses the filesystem as memory through Directory objects, providing a programming interface to store and retrieve files in hierarchical structures using HDF5.

## Main Features

- Filesystem as memory through Directory objects
- Hierarchical data organization
- Automatic path handling and resolution with pathlib
- Array storage in HDF5 format
- Parallel read/write operations
- Configuration-based compilation and access of data
- Configuration management in YAML files
- Configuration comparison and diffing
- Pandas DataFrame integration
- Directory structure visualization (tree view)
- Experiment status tracking

## Example

```python
import datamate
import numpy as np

# Set up experiment directory
datamate.set_root_dir("./experiments")

# Set up experiment configuration
config = {
    "model": "01",
    "optimizer": "Adam",
    "learning_rate": 0.001,
    "n_epochs": 100
}

# Set up experiment directory and store configuration
exp = datamate.Directory("vision_study/model_01", config)

# Store arrays as HDF5 files
exp.images = np.random.rand(100, 64, 64)  # stored as images.h5
exp.responses = np.zeros((100, 1000))     # stored as responses.h5

# Access data
mean_response = exp.responses[:].mean()
```

More detailed examples in the [documentation](https://flyvis.github.io/datamate).

## Installation

Using pip:

```bash
pip install datamate
```

## Documentation

Full documentation is available at [flyvis.github.io/datamate](https://flyvis.github.io/datamate).

## Related Projects

- [flyvis](https://github.com/turagalab/flyvis) - Usage example of datamate
- [artisan](https://github.com/MasonMcGill/artisan) - The framework that inspired datamate

## Contributing

Contributions welcome! Please check our [Contributing Guide](https://flyvis.github.io/datamate/contribute) for guidelines.
