GCSFS
=====

A pythonic file-system interface to `Google Cloud Storage`_.

This software is alpha, use at your own risk.

Please file issues and requests on github_ and we welcome pull requests.

.. _github: https://github.com/martindurant/gcsfs/issues

Installation
------------

The code is pure-python. The only requirements are:

   - oauth2client

   - requests

It can be installed using ``pip``

.. code-block:: bash

   pip install git+https://github.com/martindurant/gcsfs.git

or by cloning the repository

.. code-block:: bash

   git clone https://github.com/martindurant/gcsfs/
   cd gcsfs
   python setup.py install

or simply by copying the ``gcsfs`` directory in one of the entries in your PYTHONPATH.

A conda package will become available in the near future.

Examples
--------

Simple locate and read a file:

.. code-block:: python

   >>> import gcsfs
   >>> fs = gcsfs.GCSFileSystem(project='my-google-project', token='/path/to/token')
   >>> fs.ls('my-bucket')
   ['my-file.txt']
   >>> with fs.open('my-bucket/my-file.txt', 'rb') as f:
   ...     print(f.read())
   b'Hello, world'

(see also ``walk`` and ``glob``)

Reading with delimited blocks:

.. code-block:: python

   >>> fs.read_block(path, offset=1000, length=10, delimiter=b'\n')
   b'A whole line of text\n'

Writing with blocked caching:

.. code-block:: python

   >>> with fs.open('mybucket/new-file', 'wb') as f:
   ...     f.write(2*2**20 * b'a')
   ...     f.write(2*2**20 * b'a') # data is flushed and file closed
   >>> fs.du('mybucket/new-file')
   {'mybucket/new-file': 4194304}

Because GCSFS faithfully copies the Python file interface it can be used
smoothly with other projects that consume the file interface like ``gzip`` or
``pandas``.

.. code-block:: python

   >>> with fs.open('mybucket/my-file.csv.gz', 'rb') as f:
   ...     g = gzip.GzipFile(fileobj=f)  # Decompress data with gzip
   ...     df = pd.read_csv(g)           # Read CSV file with Pandas

Credentials
-----------

Two modes of authentication are supported:

    - if ``token=None``, you will be given a "device code", which you must
      enter into a browser where you are logged in with your Google identity.

    - if ``token='cloud'``, we assume we are running within google (compute
      or container engine) and fetch the credentials automatically from the
      metadata service.

    - you may supply a token generated by the
      gcloud_ utility; this is either a python dictionary, or the name of a file
      containing the JSON returned by logging in with the gcloud CLI tool. On
      a posix system this may be at
      ``~/.config/gcloud/application_default_credentials.json``

Authorizations are cached in a local file, for a given project/access level, so
you should not need to authorize again.

The acquired session tokens are *not* preserved when serializing the instances, so
it is safe to pass them to worker processes on other machines if using in a
distributed computation context. Credentials must be given by a file path, however,
and this file must exist on every machine.

Connection with Dask and Zarr
-----------------------------

Importing gcsfs will make this file-system backend available to dask_ for
parallel data ingestion using URLs
something like ``gcs://mybucket/myfiles/*.csv``; both ``gcs:`` and ``gs:`` work.

Similarly, ``GCSMap`` is a valid mutable-mapping, which can be used with zarr_.

Contents
========

.. toctree::
   api
   developer
   :maxdepth: 2


.. _Google Cloud Storage: https://cloud.google.com/storage/docs/

.. _gcloud: https://cloud.google.com/sdk/docs/

.. _dask: http://dask.pydata.org/en/latest/remote-data-services.html

.. _zarr: http://zarr.readthedocs.io/en/latest/tutorial.html#storage-alternatives

Indices and tables
==================

* :ref:`genindex`
* :ref:`modindex`
* :ref:`search`