.. _blob-storage-reference:

==========================
Blob Storage API Reference
==========================


.. currentmodule:: neuromation.api


Blob Storage
============

.. note::
   Be careful with using trailing slashes in Blob Storage URL's, keys and prefixes.
   ``URL("blob:my_bucket/folder/")`` is not the same as ``URL("blob:my_bucket/folder")``.
   Try to always use trailing slash when working with folder prefixes or keys.
   Also note, that keys and prefixes are specified **without** a leading slash.


.. class:: BlobStorage

   Blob Storage interaction subsystem, available as :attr:`Client.blob_storage`.

   The subsystem helps take advantage of many basic functionality of Blob Storage
   solutions different cloud providers support. For AWS it would be S3, for GCP -
   Cloud Storage, etc.


   .. comethod:: list_buckets() -> List[BucketListing]

      List all buckets available to the current user. The availability is determined
      by Neuromation ACL definitions, so the bucket list may be different for each user.

      Permissions required:

      - ``blob://{cluster_name}/`` *READ* - access to list all buckets
      - ``blob://{cluster_name}/{bucket_name}/`` *READ* - access to list single
        bucket

      :return: a :class:`list` of :class:`BucketListing` objects available to user.

   .. comethod:: list_blobs(bucket_name: str, prefix: str = "", \
                              recursive: bool = False, max_keys: int = 10000 \
                  ) -> Tuple[Sequence[BlobListing], Sequence[PrefixListing]]

      List blobs in the bucket. You can filter by prefix and return results similar
      to a folder structure if ``recursive=False`` is provided ::

         blobs, prefixes = await client.blob_storage.list_blobs(
            bucket_name="my_bucket",
            recursive=False,
            prefix="parent/"
         )
         for blob in blobs:
            print("File ", blob.key)
         for folder in prefixes:
            print("Folder ", folder.prefix)

      :param str bucket_name: Name of the bucket.
      :param str prefix: Filter results by a prefix of it's key.
      :param recursive bool: If ``True`` listing will contain *all* keys filtered by
          prefix, while with ``False`` only ones up to next ``/`` will be returned.
          To indicate missing keys, all that were listed will be combined under a
          common prefix and returned as :class:`PrefixListing`.
      :param max_keys int: Maximum number of :class:`BlobListing` objects returned.

      :return: a :class:`list` of either :class:`BlobListing` or
          :class:`PrefixListing` objects.

   .. comethod:: glob_blobs(bucket_name: str, pattern: str) -> List[BlobListing]

      Glob search the given key pattern *pattern* in the bucket *bucket_name*::

          for blob in await client.blob_storage.glob_blobs(
              bucket_name="my_bucket",
              pattern="folder1/**/*.txt"
          ):
              print(blob.key, blob.size, blob.modification_time)

      Similar to :meth:`Storage.glob` the ``“**”`` pattern means “this directory and
      all sub-directories, recursively”.

      :param str bucket_name: Name of the bucket.
      :param str pattern: key pattern according to the rules used by the Unix shell,
          similar to Python's :meth:`~glob.glob`.

      :return: a :class:`list` of either :class:`BlobListing` objects.

   .. comethod:: head_blob(bucket_name: str, key: str) -> BlobListing

      Look up the blob and return it's metadata.

      :param str bucket_name: Name of the bucket.
      :param str key: Key of the blob.

      :return: :class:`BlobListing` object.

      :raises: :exc:`FileNotFound` if key does not exist *or* you don't have access
          to it.

   .. comethod:: get_blob(bucket_name: str, key: str, offset: int = 0, size: Optional[int] = None) -> Blob
      :async-with:

      Look up the blob and return it's metadata with body content.

      :param str bucket_name: Name of the bucket.
      :param str key: Key of the blob.
      :param int offset: Position in blob from which to read.
      :param int size: Maximal size of the read data.  If ``None`` read to the end of the blob.

      :return: :class:`Blob` object. Please note, that ``body_stream``'s lifetime is
         bounded to the asynchronous context manager. Accessing the attribute outside
         of the context manager will result in an error.

      :raises: :exc:`FileNotFound` if key does not exist *or* you don't have access
          to it.

   .. comethod:: fetch_blob(bucket_name: str, key: str, offset: int = 0, size: Optional[int] = None) -> AsyncIterator[bytes]
      :async-for:

      Look up the blob and return it's body content only. The content will be streamed
      using an asynchronous iterator, e.g.::

         async for data in client.blob_storage.fetch_blob("my_bucket", key: "file.txt"):
             print("Next chunk of data:", data)

      :param str bucket_name: Name of the bucket.
      :param str key: Key of the blob.
      :param int offset: Position in blob from which to read.
      :param int size: Maximal size of the read data.  If ``None`` read to the end of the blob.

      :return: asynchronous iterator used for retrieving the file content.

      :raises: :exc:`FileNotFound` if key does not exist *or* you don't have access
          to it.

   .. comethod:: put_blob(bucket_name: str, key: str, \
            body: Union[AsyncIterator[bytes], bytes], \
            size: int, content_md5: str) -> str

      Create or replace blob identified by ``key`` in the bucket, e.g::

         large_file = Path("large_file.dat")
         size = large_file.stat().st_size
         file_md5 = await calc_md5(large_file)

         async def body_stream():
             with large_file.open("r") as f:
                 for line in f:
                     yield f

         await client.blob_storage.put_blob(
             bucket_name="my_bucket", key="large_file.dat",
             body=body_stream, size=size, content_md5=file_md5
         )

      ``md5`` should be a Base64 encoding of the 128 bit digest, e.g::

            body = b"My file body"
            md5_digest = hashlib.md5(body).digest()
            encoded_md5 = base64.b64encode(md5_digest).decode()

      :param str bucket_name: Name of the bucket.
      :param str key: Key of the blob.
      :param bytes body: Body of the blob. Can be passed as either :class:`bytes`
         or as an ``AsyncIterator[bytes]``.
      :param int size: Size of body in bytes. Only required if body is passed as an
         ``AsyncIterator[bytes]``.
      :param str content_md5: Base64 encoded 128 bit MD5 digest of the body.

      :raises: :exc:`FileNotFound` if key does not exist *or* you don't have access
         to it.

   .. rubric:: Data transfer operations

   .. comethod:: make_url(bucket_name: str, key: str) -> URL:

      Minor helper function to make URL creation easier when using High Level
      operations like :meth:`BlobStorage.download_dir`::

         src = URL("file:///usr/data")
         dst = client.blob_storage.make_url(
            bucket_name="my_bucket",
            key="folder1/"
         )
         await client.blob_storage.upload_dir(src, dst)

      :param ~yarl.URL src: path on remote storage to download a directory from
                            e.g. ``yarl.URL("storage:folder")``.

      :param ~yarl.URL dst: local path to save downloaded directory,
                            e.g. ``yarl.URL("file:///home/andrew/folder")``.

      :param AbstractRecursiveFileProgress progress:

         a callback interface for reporting downloading progress, ``None`` for no
         progress report (default).

   .. comethod:: download_dir(src: URL, dst: URL, \
                              *, update: bool = False, \
                              continue_: bool = False, \
                              filter: Optional[Callable[[str], Awaitable[bool]]] = None, \
                              progress: Optional[AbstractRecursiveFileProgress] = None \
                 ) -> None:

      Similarly to :meth:`Storage.download_dir`, allows to recursively download
      remote directory *src* to local path *dst*.

      :param ~yarl.URL src: path on Blob Storage to download a directory from
                            e.g. ``yarl.URL("blob:my_bucket/folder/")``.

      :param ~yarl.URL dst: local path to save downloaded directory,
                            e.g. ``yarl.URL("file:///home/andrew/folder")``.

      :param bool update: if true, download only when the source file is newer
                          than the destination file or when the destination
                          file is missing.

      :param bool continue_: if true, download only the part of the source file
                             past the end of the destination file and append it
                             to the destination file if the destination file is
                             newer and not longer than the source file.
                             Otherwise download and overwrite the whole file.

      :param Callable[[str], Awaitable[bool]] filter:

         a callback function for determining which files and subdirectories
         be downloaded. It is called with a relative path of file or directory
         and if the result is false the file or directory will be skipped.

      :param AbstractRecursiveFileProgress progress:

         a callback interface for reporting downloading progress, ``None`` for no
         progress report (default).

   .. comethod:: download_file(src: URL, dst: URL, \
                               *, update: bool = False, \
                               continue_: bool = False, \
                               progress: Optional[AbstractFileProgress] = None \
                 ) -> None:

      Similarly to :meth:`Storage.download_file`, allows to download remote file
      *src* to local path *dst*.

      :param ~yarl.URL src: path on Blob Storage to download a file from
                            e.g. ``yarl.URL("blob:my_bucket/folder/file.bin")``.

      :param ~yarl.URL dst: local path to save downloaded file,
                            e.g. ``yarl.URL("file:///home/andrew/folder/file.bin")``.

      :param bool update: if true, download only when the source file is newer
                          than the destination file or when the destination
                          file is missing.

      :param bool continue_: if true, download only the part of the source file
                             past the end of the destination file and append it
                             to the destination file if the destination file is
                             newer and not longer than the source file.
                             Otherwise download and overwrite the whole file.

      :param AbstractFileProgress progress:

         a callback interface for reporting downloading progress, ``None`` for
         no progress report (default).

   .. comethod:: upload_dir(src: URL, dst: URL, \
                            *, update: bool = False, \
                            filter: Optional[Callable[[str], Awaitable[bool]]] = None, \
                            ignore_file_names: AbstractSet[str] = frozenset(), \
                            progress: Optional[AbstractRecursiveFileProgress] = None \
                 ) -> None:

      Similarly to :meth:`Storage.upload_dir`, allows to recursively upload local
      directory *src* to Blob Storage URL *dst*.


      :param ~yarl.URL src: path to uploaded directory on local disk,
                            e.g. ``yarl.URL("file:///home/andrew/folder")``.

      :param ~yarl.URL dst: path on Blob Storage for saving uploading directory
                            e.g. ``yarl.URL("blob:my_folder/folder/")``.


      :param bool update: if true, download only when the source file is newer
                          than the destination file or when the destination
                          file is missing.

      :param Callable[[str], Awaitable[bool]] filter:

         a callback function for determining which files and subdirectories
         be uploaded. It is called with a relative path of file or directory
         and if the result is false the file or directory will be skipped.

      :param AbstractSet[str] ignore_file_names:

         a set of names of files which specify filters for skipping files and
         subdirectories. The format of ignore files is the same as
         ``.gitignore``.

      :param AbstractRecursiveFileProgress progress:

         a callback interface for reporting uploading progress, ``None`` for no progress
         report (default).

   .. comethod:: upload_file(src: URL, dst: URL, \
                             *, update: bool = False, \
                             progress: Optional[AbstractFileProgress] = None \
                 ) -> None:

      Similarly to :meth:`Storage.upload_file`, allows to upload local file *src* to
      storage URL *dst*.

      :param ~yarl.URL src: path to uploaded file on local disk,
                            e.g. ``yarl.URL("file:///home/andrew/folder/file.txt")``.

      :param ~yarl.URL dst: path on remote storage for saving uploading file
                            e.g. ``yarl.URL("storage:folder/file.txt")``.

      :param bool update: if true, download only when the source file is newer
                          than the destination file or when the destination
                          file is missing.

      :param AbstractFileProgress progress:

         a callback interface for reporting uploading progress, ``None`` for no progress
         report (default).


BucketListing
=============

.. class:: BucketListing

   *Read-only* :class:`~dataclasses.dataclass` for describing bucket.

   .. attribute:: name

      Name of the bucket, :class:`str`.

   .. attribute:: creation_time

      Creation time in seconds since the :ref:`epoch`, like the value returned from
      :func:`time.time()`, :class:`int`.

   .. attribute:: permission

      Permission (*read*, *write* or *manage*), :class:`Action`. Derived from
      ACL permission action on ``blob://{cluster_name}/{bucket_name}/`` resource.

   .. attribute:: uri

      Relative URI identifying the bucket, :class:`~yarl.URL`, e.g. ``blob:my_bucket``


BlobListing
===========

.. class:: BlobListing

   *Read-only* :class:`~dataclasses.dataclass` for describing blobs.

   .. attribute:: bucket_name

      Name of the bucket, :class:`str`.

   .. attribute:: key

      Key of the blob, :class:`str`.

   .. attribute:: modification_time

      Modification time in seconds since the :ref:`epoch`, like the value returned from
      :func:`time.time()`, :class:`int`.

   .. attribute:: size

      Size of the data in *bytes*, :class:`int`.

   .. attribute:: uri

      Relative URI identifying the blob, :class:`~yarl.URL`, e.g.
      ``blob:my_bucket/file.txt``.

PrefixListing
=============

.. class:: PrefixListing

   *Read-only* :class:`~dataclasses.dataclass` for describing common prefixes for
   blobs in non-recursive listing. You can treat it as a kind of *folder* on Blob
   Storage.

   .. attribute:: bucket_name

      Name of the bucket, :class:`str`.

   .. attribute:: prefix

      Prefix path, :class:`str`.

   .. attribute:: uri

      Relative URI identifying the folder, :class:`~yarl.URL`, e.g.
      ``blob:my_bucket/my_folder/``.

Blob
====

.. class:: Blob

   *Read-only* :class:`~dataclasses.dataclass` for describing common prefixes for
   blobs in non-recursive listing. You can treat it as a kind of *folder* on Blob
   Storage.

   .. attribute:: stats

      :class:`BlobListing` related to this blob.

   .. attribute:: body_stream

      :class:`aiohttp.StreamReader` content body.
