Metadata-Version: 2.1
Name: ossfs
Version: 2023.12.0
Summary: fsspec filesystem for OSS
Home-page: https://github.com/fsspec/ossfs
Download-URL: https://github.com/fsspec/ossfs
Maintainer-email: gao@iterive.ai
License: Apache-2.0
Platform: any
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.8
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Development Status :: 4 - Beta
Requires-Python: >=3.8
Description-Content-Type: text/x-rst
License-File: LICENSE
Requires-Dist: fsspec>=2023.9.2
Requires-Dist: oss2==2.18.1
Requires-Dist: aiooss2>=0.2.7
Provides-Extra: tests
Requires-Dist: pytest==7.2.0; extra == "tests"
Requires-Dist: pytest-sugar==0.9.5; extra == "tests"
Requires-Dist: pytest-cov==3.0.0; extra == "tests"
Requires-Dist: pytest-mock==3.8.2; extra == "tests"
Requires-Dist: pylint==2.15.0; extra == "tests"
Requires-Dist: pylint-pytest==1.1.2; extra == "tests"
Requires-Dist: mypy==0.971; extra == "tests"
Requires-Dist: requests==2.31.0; extra == "tests"
Requires-Dist: aliyun-python-sdk-sts==3.1.1; extra == "tests"
Provides-Extra: dev
Requires-Dist: pytest==7.2.0; extra == "dev"
Requires-Dist: pytest-sugar==0.9.5; extra == "dev"
Requires-Dist: pytest-cov==3.0.0; extra == "dev"
Requires-Dist: pytest-mock==3.8.2; extra == "dev"
Requires-Dist: pylint==2.15.0; extra == "dev"
Requires-Dist: pylint-pytest==1.1.2; extra == "dev"
Requires-Dist: mypy==0.971; extra == "dev"
Requires-Dist: requests==2.31.0; extra == "dev"
Requires-Dist: aliyun-python-sdk-sts==3.1.1; extra == "dev"

OSSFS
=====

|PyPI| |Status| |Python Version| |License|

|Tests| |Codecov| |pre-commit| |Black|

.. |PyPI| image:: https://img.shields.io/pypi/v/ossfs.svg
   :target: https://pypi.org/project/ossfs/
   :alt: PyPI
.. |Status| image:: https://img.shields.io/pypi/status/ossfs.svg
   :target: https://pypi.org/project/ossfs/
   :alt: Status
.. |Python Version| image:: https://img.shields.io/pypi/pyversions/ossfs
   :target: https://pypi.org/project/ossfs
   :alt: Python Version
.. |License| image:: https://img.shields.io/pypi/l/ossfs
   :target: https://opensource.org/licenses/Apache-2.0
   :alt: License
.. |Tests| image:: https://github.com/karajan1001/ossfs/workflows/Tests/badge.svg
   :target: https://github.com/karajan1001/ossfs/actions?workflow=Tests
   :alt: Tests
.. |Codecov| image:: https://codecov.io/gh/karajan1001/ossfs/branch/main/graph/badge.svg
   :target: https://app.codecov.io/gh/karajan1001/ossfs
   :alt: Codecov
.. |pre-commit| image:: https://img.shields.io/badge/pre--commit-enabled-brightgreen?logo=pre-commit&logoColor=white
   :target: https://github.com/pre-commit/pre-commit
   :alt: pre-commit
.. |Black| image:: https://img.shields.io/badge/code%20style-black-000000.svg
   :target: https://github.com/psf/black
   :alt: Black

**OSSFS** is a Python-based interface for file systems that enables interaction with
OSS (Object Storage Service). Through **OSSFS**, users can utilize fsspec's standard
API to operate on OSS objects

Installation
------------

You can install *OSSFS* via pip_ from PyPI_:

.. code:: console

   $ pip install ossfs

Up-to-date package also provided through conda-forge distribution:

.. code:: console

   $ conda install -c conda-forge ossfs

Quick Start
------------

Here is a simple example of locating and reading an object in OSS.

.. code:: python

   import ossfs
   fs = ossfs.OSSFileSystem(endpoint='http://oss-cn-hangzhou.aliyuncs.com')
   fs.ls('/dvc-test-anonymous/LICENSE')
   [{'name': '/dvc-test-anonymous/LICENSE',
     'Key': '/dvc-test-anonymous/LICENSE',
     'type': 'file',
     'size': 11357,
     'Size': 11357,
     'StorageClass': 'OBJECT',
     'LastModified': 1622761222}]
   with fs.open('/dvc-test-anonymous/LICENSE') as f:
   ...     print(f.readline())
   b'                                 Apache License\n'

For more use case and apis please refer to the documentation of `fsspec <https://filesystem-spec.readthedocs.io/en/latest/index.html>`_

Async OSSFS
------------

Async **OSSFS** is a variant of ossfs that utilizes the third-party async OSS
backend `aiooss2`_, rather than the official sync one, `oss2`_. Async OSSFS
allows for concurrent calls within bulk operations, such as *cat*, *put*, and
*get* etc even from normal code, and enables the direct use of fsspec in async
code without blocking. The usage of async **OSSFS** is similar to the synchronous
variant; one simply needs to replace **OSSFileSystem** with **AioOSSFileSystem**
need to do is replacing the **OSSFileSystem** with the **AioOSSFileSystem**

.. code:: python

   import ossfs
   fs = ossfs.AioOSSFileSystem(endpoint='http://oss-cn-hangzhou.aliyuncs.com')
   print(fs.cat('/dvc-test-anonymous/LICENSE'))
   b'                                 Apache License\n'
   ...

Although `aiooss2`_ is not officially supported, there are still some
features that are currently lacking. However, in tests involving the
*put*/*get* of 1200 small files, the async version of ossfs ran ten times
faster than the synchronous variant (depending on the pool size of the
concurrency).

+-------------------------------------------+------------------------+
| Task                                      | time cost in (seconds) |
+===========================================+========================+
| put 1200 small files via OSSFileSystem    | 35.2688 (13.53)        |
+-------------------------------------------+------------------------+
| put 1200 small files via AioOSSFileSystem | 2.6060 (1.0)           |
+-------------------------------------------+------------------------+
| get 1200 small files via OSSFileSystem    | 32.9096 (12.63)        |
+-------------------------------------------+------------------------+
| get 1200 small files via AioOSSFileSystem | 3.3497 (1.29)          |
+-------------------------------------------+------------------------+

Contributing
------------

Contributions are very welcome.
To learn more, see the `Contributor Guide`_.


License
-------

Distributed under the terms of the `Apache 2.0 license`_,
*Ossfs* is free and open source software.


Issues
------

If you encounter any problems,
please `file an issue`_ along with a detailed description.


.. _Apache 2.0 license: https://opensource.org/licenses/Apache-2.0
.. _PyPI: https://pypi.org/
.. _file an issue: https://github.com/fsspec/ossfs/issues
.. _aiooss2: https://github.com/karajan1001/aiooss2/
.. _oss2: https://pypi.org/project/oss2/
.. _pip: https://pip.pypa.io/
.. github-only
.. _Contributor Guide: CONTRIBUTING.rst
