GCSFS
=====

A Python filesystem abstraction of Google Cloud Storage (GCS) implemented as a `PyFilesystem2 <https://github.com/PyFilesystem/pyfilesystem2>`__ extension.


.. image:: https://img.shields.io/pypi/v/fs-gcsfs.svg
    :target: https://pypi.org/project/fs-gcsfs/

.. image:: https://img.shields.io/pypi/pyversions/fs-gcsfs.svg
    :target: https://pypi.org/project/fs-gcsfs/

.. image:: https://travis-ci.org/Othoz/gcsfs.svg?branch=master
    :target: https://travis-ci.org/Othoz/gcsfs

.. image:: https://readthedocs.org/projects/fs-gcsfs/badge/?version=latest
    :target: https://fs-gcsfs.readthedocs.io/en/latest/?badge=latest


With GCSFS, you can interact with `Google Cloud Storage <https://cloud.google.com/storage/>`__ as if it was a regular filesystem.

Apart from the nicer interface, this will highly decouple your code from the underlying storage mechanism: Exchanging the storage backend with an
`in-memory filesystem <https://pyfilesystem2.readthedocs.io/en/latest/reference/memoryfs.html>`__ for testing or any other
filesystem like `S3FS <https://github.com/pyfilesystem/s3fs>`__ becomes as easy as replacing ``gs://bucket_name`` with ``mem://`` or ``s3://bucket_name``.

For a full reference on all the PyFilesystem possibilities, take a look at the
`PyFilesystem Docs <https://pyfilesystem2.readthedocs.io/en/latest/index.html>`__!


Documentation
-------------

-  `GCSFS Documentation <http://fs-gcsfs.readthedocs.io/en/latest/>`__
-  `PyFilesystem Wiki <https://www.pyfilesystem.org>`__
-  `PyFilesystem Reference <https://docs.pyfilesystem.org/en/latest/reference/base.html>`__


Installing
----------

Install the latest GCSFS version by running::

    $ pip install fs-gcsfs

Or in case you are using conda::

    $ conda install -c conda-forge fs-gcsfs


Examples
--------

Instantiating a filesystem on Google Cloud Storage (for a full reference visit the
`Documentation <http://fs-gcsfs.readthedocs.io/en/latest/index.html#reference>`__):

.. code-block:: python

    from fs_gcsfs import GCSFS
    gcsfs = GCSFS(bucket_name="mybucket")


Alternatively you can use a `FS URL <https://pyfilesystem2.readthedocs.io/en/latest/openers.html>`__ to open up a filesystem:

.. code-block:: python

    from fs import open_fs
    gcsfs = open_fs("gs://mybucket/root_path?project=test&api_endpoint=http%3A//localhost%3A8888&strict=False")


Supported query parameters are:

- `project` (str): Google Cloud project to use
- `api_endpoint` (str): URL-encoded endpoint that will be passed to the GCS client's `client_options <https://googleapis.dev/python/google-api-core/latest/client_options.html#google.api_core.client_options.ClientOptions>`__
- `strict` ("True" or "False"): Whether GCSFS will be opened in strict mode


You can use GCSFS like your local filesystem:

.. code-block:: python

    >>> from fs_gcsfs import GCSFS
    >>> gcsfs = GCSFS(bucket_name="mybucket")
    >>> gcsfs.tree()
    ├── foo
    │   ├── bar
    │   │   ├── file1.txt
    │   │   └── file2.csv
    │   └── baz
    │       └── file3.txt
    └── file4.json
    >>> gcsfs.listdir("foo")
    ["bar", "baz"]
    >>> gcsfs.isdir("foo/bar")
    True


Uploading a file is as easy as:

.. code-block:: python

    from fs_gcsfs import GCSFS
    gcsfs = GCSFS(bucket_name="mybucket")
    with open("local/path/image.jpg", "rb") as local_file:
        with gcsfs.open("path/on/bucket/image.jpg", "wb") as gcs_file:
            gcs_file.write(local_file.read())


You can even sync an entire bucket on your local filesystem by using PyFilesystem's utility methods:

.. code-block:: python

    from fs_gcsfs import GCSFS
    from fs.osfs import OSFS
    from fs.copy import copy_fs

    gcsfs = GCSFS(bucket_name="mybucket")
    local_fs = OSFS("local/path")

    copy_fs(gcsfs, local_fs)


For exploring all the possibilities of GCSFS and other filesystems implementing the PyFilesystem interface, we recommend visiting the official
`PyFilesystem Docs <https://pyfilesystem2.readthedocs.io/en/latest/index.html>`__!


Development
-----------

To develop on this project make sure you have `pipenv <https://pipenv.readthedocs.io/en/latest/>`__ installed
and run the following from the root directory of the project::

    $ pipenv install --dev --three

This will create a virtualenv with all packages and dev-packages installed.


Tests
-----
All CI tests run against an actual GCS bucket provided by `Othoz <http://othoz.com/>`__.

In order to run the tests against your own bucket,
make sure to set up a `Service Account <https://cloud.google.com/iam/docs/service-accounts>`__ with all necessary permissions:

- storage.objects.get
- storage.objects.list
- storage.objects.create
- storage.objects.update
- storage.objects.delete

All five permissions listed above are e.g. included in the `predefined Cloud Storage IAM Role <https://cloud.google.com/storage/docs/access-control/iam-roles>`__ ``roles/storage.objectAdmin``.

Expose your bucket name as an environment variable ``$TEST_BUCKET`` and run the tests via::

    $ pipenv run pytest

Note that the tests mostly wait for I/O, therefore it makes sense to highly parallelize them with `xdist <https://github.com/pytest-dev/pytest-xdist>`__, e.g. by running the tests with::

    $ pipenv run pytest -n 10


Credits
-------

Credits go to `S3FS <https://github.com/PyFilesystem/s3fs>`__ which was the main source of inspiration and shares a lot of code with GCSFS.