Setting up a database and API server
====================================

This page describes the process of setting up your own database /
storage for the models. This puts you in full control of who has access
to the experiment data. For the moment, Studio only supports Firebase
(https://firebase.google.com/) as a database backend, and 
firebase / google cloud storage (GCS) / Amazon S3 as storage
backends.

Introduction
------------
Firebase and Firebase Storage provide fairly simple rules to control use access 
to experiments. Additionally, GCS and S3 don't work directly with Firebase 
authentication tokens, so one cannot create access rules for the storage. 
In order to provide more rigorous rules, we are employing
an API server that proxies database requests and can provide arbitrarily complex
access rules (at the moment the access rules are still very simple - anyone can
read any experiment, and only user who created the experiment can delete / overwrite
it). Also, API server should allow one to swap database backends 
(i.e. from Firebase to DynamoDB) 
completely seamlessly for the users, without even updating the users' config
files. Yet another reason to use the API server is that GCS and S3 are 
much cheaper than Firebase Storage for large amounts of data. 

Generally, the outline of the API server / database / storage interaction is 
as follows: 

1. API server has read/write access to database and storage
2. When getting/writing the data about experiment, user signs the HTTP request 
   with firebase authentication token. The API server then validates that user 
   indeed has permissions to do so, and either 
   returns data about experiment (for /api/get_experiment method) or 
   writes the expeirment data
3. Artifacts are being read and written via communicating with storage
   directly using signed urls, generated by API server

The detailed instructions on setting up the API server (we'll use
google app engine, GAE, but these steps can be trivially adapted 
for heroku or just running API server on a dedicated instance)

Prerequisites
~~~~~~~~~~~~~
If deploying onto google app engine, you'll need to have Google Cloud SDK
installed (https://cloud.google.com/sdk/downloads)

In what follows, deployment machine means either the local machine 
(when deploying on GAE) or the instance on which you are 
planning to run the API server

Deploying the API server 
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

1. Create a new Firebase project: go to https://firebase.google.com,
   sign in, click add project, specify project name
2. Enable authentication using google accounts (in the left-hand pane 
   select Authentication, then tab "Sign-in method", click on 
   "Google", select "Enabled")
3. Go to project settings (little cogwheel next to "Overview" on the
   left-hand pane), tab "General"
4. Copy the Web API key and paste it in apiKey of the database section of
   studio/apiserver\_config.yaml 
5. Copy the project ID and paste it in projectId of the database section of
   config yaml file. 
6. Go to Service Accounts tab and generate a new key for the firebase
   service account. This key is a json file that will give API server admin 
   access to the database. Save it to the deployment machine. 
7. Modify other entries of the apiserver_config.yaml file to your specs 
   (e.g. storage type and bucket)
8. On the deployment machine in the folder studio/studio, run
      
      ::
      
      ./deploy_apiserver.sh gae
      
   for GAE and 
   
      ::
       
      ./deploy_apiserver.sh local <port>
      
   when running on a dedicated instance (where port is the port on which 
   the server will be listening). When prompted, input path to 
   the firebase admin credentials json file generated in step 6.

   
       
Configuring studio to work with the API server
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

For clients to work with the API server, you'll
need to modify their config.yaml files as follows:

1. Remove storage section
2. In the database section, set type: http, 
   serverUrl: <url of your deployed server>. 
   When deploying to GAE, the url will have format
   https://<project_name>.appspot.com. When deploying
   on a dedicated instance, don't forget to specify the
   port.
       

