***************************
Administrator Documentation
***************************

When databroker is imported, it discovers catalogs available on the system.
User can list the discovered catalogs by importing a special global
``databroker.catalog`` object and listing its entries.

.. code:: python

   from databroker import catalog
   list(catalog)  # a list of strings, names of sub-catalogs

which can be accessed like

.. code:: python

   catalog['SOME_SUB_CATALOG']

DataBroker assembles this list of catalogs by looking for:

1. Old-style "databroker v0.x" YAML configuration files, for backward-compatibility
2. Intake-style catalog YAML files, which have different fields
3. Python packages that advertise catalogs via the ``intake.catalogs``
   entrypoint

Old-style databroker configuration files
========================================

DataBroker v0.x used a custom YAML-based configuration file. See
:ref:`v0_configuration`. For backward-compatibility, configuration files
specifying MongoDB storage will be discovered and included in
``databroker.catalog``.

Migrating sqlite or HDF5 storage
--------------------------------

The implementation in ``databroker.v0`` interfaces with storage in MongoDB,
sqlite, or HDF5.  The implementations in ``databroker.v1`` and
``databroker.v2`` drop support for sqlite and HDF5 and add support for JSONL_
(newline-delimited JSON) and msgpack_. For binary file-based storage, we
recommend using msgpack. Data can be migrated from sqlite or HDF5 to msgpack
like so:

.. code-block:: python

   from databroker import Broker
   import suitcase.msgpack

   # If the config file associated with YOUR_BROKER_NAME specifies sqlite or
   # HDF5 storage, then this will return a databroker.v0.Broker instance.
   db = Broker.named(YOUR_BROKER_NAME)
   # Loop through every run in the old Broker.
   for run in db():
       # Load all the documents out of this run from their existing format and
       # write them into one file located at
       # `<DESTINATION_DIRECTORY>/<uid>.msgpack`.
       suitcase.msgpack.export(run.documents(), DESTINATION_DIRECTORY)

In the next section, we'll create a "catalog YAML file" to make this data
discoverable by databroker.

Intake-style Catalog YAML Files
===============================

Search Path
-----------

Use the convenience function :func:`catalog_search_path`. Place catalog YAML
files in one of these locations to make them discoverable by intake and, in
turn, by databroker.

.. code:: python

   from databroker import catalog_search_path
   catalog_search_path()  # result will vary depending on OS and environment

Structure
---------

The general structure of a catalog YAML file is a nested dictionary of
data "sources". Each source name is mapped to information for accessing that
data, which includes a type of "driver" and some keyword arguments to pass to
it. A "driver" is generally associated with a particular storage format.

.. code:: yaml

   sources:
     SOME_NAME:
       driver: SOME_DRIVER
       args:
         SOME_PARAMETER: VALUE
         ANOTHER_PARAMETER: VALUE
     ANOTHER_NAME:
       driver: SOME_DRIVER
       args:
         SOME_PARAMETER: VALUE
         ANOTHER_PARAMETER: VALUE

As shown, multiple sources can be specified in one file. All sources found in
all the YAML files in the search path will be included as top-level entries in
``databroker.catalog``.

Msgpack Example
---------------

Msgpack_ is a binary file format.

.. code:: yaml

   sources:
     ENTRY_NAME:
       driver: bluesky-msgpack-catalog
       args:
         paths:
           - "DESTINATION_DIRECTORY/*.msgpack"

where ``ENTRY_NAME`` is a name of the entry that will appear in
``databroker.catalog``, and ``DESTINATION_DIRECTORY`` is a directory of msgpack
files generated by suitcase-msgpack_, as illustrated in the previous section.

Note that the value of ``paths`` is a list. Multiple directories can be grouped
into one "source".

JSONL (Newline-delimited JSON) Example
--------------------------------------

JSONL_ is a text-based format in which each line is a
valid JSON. Unlike ordinary JSON, it is suitable for streaming. This storage is
much slower than msgpack, but the format is human-readable.

.. code:: yaml

   sources:
     ENTRY_NAME:
       driver: bluesky-jsonl-catalog
       args:
         paths:
           - "DESTINATION_DIRECTORY/*.jsonl"

where ``ENTRY_NAME`` is a name of the entry that will appear in
``databroker.catalog`` and ``DESTINATION_DIRECTORY`` is a directory of
newline-delimited JSON files generated by suitcase-jsonl_.

Note that the value of ``paths`` is a list. Multiple directories can be grouped
into one "source".

MongoDB Example
---------------

MongoDB_ is the recommended storage format for
large-scale deployments because it supports fast search.

.. code:: yaml

   sources:
     ENTRY_NAME:
       driver: bluesky-mongo-normalized-catalog
       args:
         metadatastore_db: mongodb://HOST:PORT/MDS_DATABASE_NAME
         asset_registry_db: mongodb://HOST:PORT/ASSETS_DATABASE_NAME

where ``ENTRY_NAME`` is a name of the entry that will appear in
``databroker.catalog``, and the ``mongodb://...`` URIs point to MongoDB
databases with documents inserted by suitcase-mongo_.

The driver's name, ``bluesky-mongo-normalized-catalog``, differentiates it from
the ``bluesky-mongo-embedded-catalog``, an experimental alternative way of
original bluesky documents into MongoDB documents and collections. It is still
under evaluation and not yet recommended for use in production.

Python packages
===============

To distribute catalogs to users, it may be more convenient to provide an
installable Python package, rather than placing YAML files in specific
locations on the user's machine.  To achieve this, a Python package can
advertise catalog objects using the ``'intake.catalogs'`` entrypoint. Here is a
minimal example:

.. code:: python

   # setup.py
   from setuptools import setup

   setup(name='example',
         entry_points={'intake.catalogs':
             ['ENTRY_NAME = example:catalog_instance']},
         py_modules=['example'])

.. code:: python

   # example.py

   # Create an object named `catalog_instance` which is referenced in the
   # setup.py, and will be discovered by databroker. How the instance is
   # created, and what type of catalog it is, is completely up to the
   # implementation. This is just one possible example.

   import intake

   # Look up a driver class by its name in the registry.
   catalog_class = intake.registry['bluesky-mongo-normalized-catalog']

   catalog_instance = catalog_class(
       metadatastore_db='mongodb://...', asset_registry_db='mongodb://...')

The ``entry_points`` parameter in the ``setup(...)`` is a feature supported by
Python packaging. When this package is installed, a special file inside the
distribution, ``entry_points.txt``, will advertise that is has catalogs.
DataBroker will discover these and add them to ``databroker.catalog``. Note
that databroker does *not* need to actually *import* the package to discover
its catalogs. The package will only be imported if and when the catalog is
accessed. Thus, the overhead of this discovery process is low.

.. important::

   Some critical details of Python's entrypoints feature:

   * Note the unusual syntax of the entrypoints. Each item is given as one long
     string, with the ``=`` as part of the string. Modules are separated by
     ``.``, and the final object name is preceded by ``:``.
   * The right hand side of the equals sign must point to where the object is
     *actually defined*. If ``catalog_instance`` is defined in
     ``foo/bar.py`` and imported into ``foo/__init__.py`` you might expect
     ``foo:catalog_instance`` to work, but it does not. You must spell out
     ``foo.bar:catalog_instance``.


.. _jsonl: http://jsonlines.org/
.. _msgpack: https://msgpack.org/index.html
.. _suitcase-mongo: https://github.com/bluesky/suitcase-mongo
.. _suitcase-jsonl: https://github.com/bluesky/suitcase-jsonl
.. _suitcase-msgpack: https://github.com/bluesky/suitcase-msgpack
.. _MongoDB: https://www.mongodb.com/
