Configuration — MARV Robotics 3.3.5 documentation

# Configuration

The configuration file `marv.conf` is in Python config parser / ini-syntax and consists of at least one marv section and one or more collection section. See some Examples below.

If you make changes to your configuration, keep in mind that you have to stop `uwsgi`, run `marv init`, and start `uwsgi` again.

## Relative paths

The location of `marv.conf` is the site directory and relative paths are relative to that directory.

## marv section

    [marv]

### acl

Use access control lists to control who can perform which actions. See `marv_webapi.acls` for more information.

Example:

    acl = marv_webapi.acls:public

Default:

    acl = marv_webapi.acls:authenticated

### collections

Name of one or more collections, corresponding to a collection section.

Example:

    collections = bags

### dburi

Location of sqlite database. Despite the generic name, only sqlite is supported.

Example:

    dburi = sqlite:////var/local/lib/marv/db/db.sqlite

Default:

    dburi = sqlite:///path/to/sitedir/db/db.sqlite

Note

Keep the `db/db.sqlite` suffix for ease of migrations.

### reverse_proxy

When marv is running behind a reverse proxy, serving of files can be offloaded for greatly improved performance. Currently, the only supported reverse proxy is `nginx`.

Example:

    reverse_proxy = nginx

See uWSGI behind NGINX for the corresponding nginx configuration.

### storedir

Example:

    storedir = /var/local/lib/marv/store

Default:

    storedir = ./store

## collection section

Configuration for a collection of datasets.

### scanner

A `marv.scanner` is responsible to group files into named datasets.

Example:

    scanner = marv_robotics.bag:scan

See `marv_robotics.bag.scan()`

### scanroots

One or more directories to scan for datasets.

Example:

    scanroos =
        ./foo
        ./bar

### nodes

List of nodes made available within this collection under the name following the column, which is also the name of the function the node is created from.

Example:

    nodes =
        # pkg.module:func_name
        marv_nodes:dataset
        marv_robotics.bag:bagmeta

For a list of nodes see Nodes.

### filters

Listings of datasets for the web frontend and API responses can be filtered.

Nodes extract and process data from datasets. Node output persisted in the store is available via a node’s name. For this to happen the node needs to define a message type and be listed in nodes. Filters and listing_columns use S-Expressions to extract values from node output. Via API or web frontend a user supplies filter input to be compared with the extracted value using a selected operator. A filter’s name is displayed in the web frontend and its ID is used via API.

At least one operator has to be configured per filter and valid operators depend on the field type. The `tags` and `comments` filter are special and have to be defined exactly as shown below.

See S-Expressions on how to create functions to extract values from node output.

Example:

    filters =
        # id     | Display Name | operators         | field type | extractor function
        name     | Name         | substring         | string     | (get "dataset.name")
        setid    | Set Id       | startswith        | string     | (get "dataset.id")
        size     | Size         | lt le eq ne ge gt | filesize   | (sum (get "dataset.files[:].size"))
        tags     | Tags         | any all           | subset     | (tags )
        comments | Comments     | substring         | string     | (comments )
        fulltext | Fulltext     | words             | words      | (get "fulltext.words")
        files    | File paths   | substring_any     | string[]   | (get "dataset.files[:].path")
        end_time | End time     | lt le eq ne ge gt | datetime   | (get "bagmeta.end_time")
        duration | Duration     | lt le eq ne ge gt | timedelta  | (get "bagmeta.duration")
        topics   | Topics       | any all           | subset     | (get "bagmeta.topics[:].name")

In case you use emacs, it’s easy to align these: `C-u M-x align-regexp | RET RET y`.

#### field type

The field type determines what python type the extractor function is expected to return, how this is interpreted and displayed, and what is expected as filter input.

##### `datetime`

extract: int, nanoseconds since epoch
input: int, millisecons since epoch
operators: `lt` `le` `eq` `ne` `ge` `gt`

##### `filesize`

extract: int, bytes
input: int, bytes
operators: `lt` `le` `eq` `ne` `ge` `gt`

##### `string`

extract: unicode
input: utf-8
operators: `substring`, `startswith`

##### `string[]`

extract: list of unicode
input: utf-8
operators: `substring_any`

##### `subset`

extract: list of unicode
input: list of utf-8
operators: `any`, `all`

##### `timedelta`

extract: int, nanoseconds
input: int, millisecons
operators: `lt` `le` `eq` `ne` `ge` `gt`

##### `words`

extract: list of unicode
input: list of utf-8
operators: `words`

#### operators

##### `lt` `le` `eq` `ne` `ge` `gt`

Comparison of numeric input with numeric stored value.

##### `substring`

Match input as substring anywhere in stored string.

##### `startswith`

Stored string starts with input string.

##### `substring_any`

The input string is a substring of any string in a stored list of strings.

##### `any`

The set of input strings intersects with the set of stored strings.

##### `all`

The set of input strings is a subset of the set of stored strings.

### listing_columns

Columns displayed for the collection’s listing.

For certain colums the id is important, so keep the ids used in the Default configuration. The heading is used as column heading, formatters are explained below and see S-Expressions on how to write functions to extract values from node output.

Example:

    listing_columns =
        # id | Heading | formatter | extractor function
        name | Name    | route     | (detail_route (get "dataset.id") (get "dataset.name"))
        size | Size    | filesize  | (sum (get "dataset.files[:].size"))

#### formatter

Marv ships with a set of formatters. See Custom on how to override these and supply your own.

##### `date`

extract: int, nanoseconds since epoch

##### `datetime`

extract: int, nanoseconds since epoch

##### `float`

Renders float with two decimal places.

extract: float

##### `icon`

Render a glyphicon by name (`glyphicon-<name>`) with optional additional space-separated css classes and a title rendered in a tooltip for the icon.

extract: `{'icon': name, 'classes': css_classes, 'title': title}`

##### `int`

extract: int

##### `link`

extract: `{'href': '', 'title': ''}`

##### `pill`

extract: int, float, unicode

##### `route`

Used only for the detail route so far in conjunction with detail_route.

##### `string`

extract: int, float, unicode

##### `timedelta`

extract: int, nanoseconds since epoch

### listing_sort

Column and sort order for listing.

Example:

    listing_sort = start_time | descending

The first field corresponds to an id in listing_columns, the second is one of `ascending` (default) or `descending`.

### listing_summary

Summary calculated for the filtered rows of the listing.

Example:

    listing_summary =
        # id     | Title    | formatter | extractor
        datasets | datasets | int       | (len (rows ))
        size     | size     | filesize  | (sum (rows "size" 0))
        duration | duration | timedelta | (sum (rows "duration" 0))

A unique id, a title to be displayed bewlo the value, a formatter explained in formatter and extractor function explained in S-Expressions.

### detail_summary_widgets

List of widgets to be rendered on the first tab of the detail view, aka the summary section.

Example:

    detail_summary_widgets =
        summary_keyval
        bagmeta_table

You can write your own and use some of the already existing Widget nodes.

### detail_sections

List of detail section to be rendered beyond the summary section. For any given dataset those sections will be rendered only if the dataset contains the necessary data. In absence of meaningful data, sections will be omitted from the web frontend detail view.

Example:

    detail_sections =
        connections_section
        video_section

You can write your own and use some of the already existing Section nodes.

## S-Expressions

S-Expressions are used in the config file to create small functions that extract values from output of stored nodes. S-Expressions are (nested) lists in parentheses, with list elements being separated by spaces.

    (get "dataset.name")
    (get "dataset.files[:].path")
    (sum (get "dataset.files[:].size"))
    (tags )
    (comments )

The first element of a list is the name of a function. Any additional arguments are passed as arguments to the function and the list defining a function is replaced with its return value.

Valid arguments are:

- strings enclosed in `""`
- functions enclosed in `()`
- `0` number, literal zero

### Functions

Functions in S-expressions get and process data from store node output. Some may be used in all scopes filters, listing_columns, and listing_summary; some only in some (see below).

#### `comments`

Return list of unicode objects with text of comments.

scope: filters, listing_columns

#### `detail_route`

Return dictionary rendering link to detail route of dataset. First argument is the dataset’s setid, second optional name is displayed instead of setid.

scope: listing_columns

#### `format`

Wrapper for `fmt.format(*args)`. First argument is the format string `fmt`, remaining arguments are passed on.

scope: filters, listing_columns

#### `get`

Get a value from a nodes output. First argument defines node and traversal into its output, second optional argument is used as default value instead of `None`.

Examples:

    (get "bagmeta.start_time")
    (get "dataset.files[:].size")

The specifier starts with the nodes name. A `.` performs dictionary key lookup. Lists can be traversed into in part or full using slicing and further dictionary lookup is performed on each element of the list.

scope: filters, listing_columns

#### `join`

Wrapper for `joinstr.join(args)`. First argument is the join string remaining arguments are joined with.

scope: filters, listing_columns

#### `len`

Return length of first argument.

scope: filters, listing_columns, listing_summary

#### `link`

Render link with first argument as href and second argument displayed.

scope: listing_columns

#### `list`

Return list consisting of arguments with arguments being None removed.

scope: filters, listing_columns, listing_summary

#### `max`

Return maximum element of first argument.

scope: filters, listing_columns, listing_summary

#### `min`

Return minimum element of first argument.

scope: filters, listing_columns, listing_summary

#### `rows`

Return all rows matching current filter criteria. The optional second argument extracts a specific column defined in listing_columns instead of the full row and the third provides a default value instead of None.

Examples:

    (sum (rows "size" 0))

scope: listing_summary

#### `sum`

Return sum of arguments.

scope: filters, listing_columns, listing_summary

#### `tags`

Return list of tags associated with dataset.

scope: filters, listing_columns

#### `trace`

Print trace messages.

scope: filters, listing_columns, listing_summary

## Examples

### Default configuration

    #
    # MARV Robotics default configuration file
    #

    [marv]
    collections = bags

    [collection bags]
    scanner = marv_robotics.bag:scan

    scanroots =
        scanroot

    nodes =
        marv_nodes:dataset
        marv_robotics.bag:bagmeta
        marv_robotics.cam:ffmpeg
        marv_robotics.cam:images
        marv_robotics.fulltext:fulltext
        marv_robotics.gnss:gnss_plots
        marv_robotics.trajectory:trajectory
        # summary widgets
        marv_robotics.detail:summary_keyval
        marv_robotics.detail:bagmeta_table
        # detail sections
        marv_robotics.detail:connections_section
        marv_robotics.detail:images_section
        marv_robotics.detail:video_section
        marv_robotics.detail:gnss_section
        marv_robotics.detail:trajectory_section

    filters =
        # id       | Display Name  | operators         | value type | value function
        name       | Name          | substring         | string     | (get "dataset.name")
        setid      | Set Id        | startswith        | string     | (get "dataset.id")
        size       | Size          | lt le eq ne ge gt | filesize   | (sum (get "dataset.files[:].size"))
        tags       | Tags          | any all           | subset     | (tags )
        comments   | Comments      | substring         | string     | (comments )
        fulltext   | Fulltext      | words             | words      | (get "fulltext.words")
        files      | File paths    | substring_any     | string[]   | (get "dataset.files[:].path")
        added_time | Added         | lt le eq ne ge gt | datetime   | (get "dataset.time_added")
        start_time | Start time    | lt le eq ne ge gt | datetime   | (get "bagmeta.start_time")
        end_time   | End time      | lt le eq ne ge gt | datetime   | (get "bagmeta.end_time")
        duration   | Duration      | lt le eq ne ge gt | timedelta  | (get "bagmeta.duration")
        topics     | Topics        | any all           | subset     | (get "bagmeta.topics")
        msg_types  | Message types | any all           | subset     | (get "bagmeta.msg_types")

    listing_columns =
        # id       | Heading    | formatter | value function
        name       | Name       | route     | (detail_route (get "dataset.id") (get "dataset.name"))
        size       | Size       | filesize  | (sum (get "dataset.files[:].size"))
        tags       | Tags       | pill[]    | (tags )
        added      | Added      | datetime  | (get "dataset.time_added")
        start_time | Start time | datetime  | (get "bagmeta.start_time")
        end_time   | End time   | datetime  | (get "bagmeta.end_time")
        duration   | Duration   | timedelta | (get "bagmeta.duration")

    listing_sort = start_time | descending

    listing_summary =
        # id     | Title    | formatter | extractor
        datasets | datasets | int       | (len (rows ))
        size     | size     | filesize  | (sum (rows "size" 0))
        duration | duration | timedelta | (sum (rows "duration" 0))

    detail_summary_widgets =
        summary_keyval
        bagmeta_table

    detail_sections =
        connections_section
        images_section
        video_section
        gnss_section
        trajectory_section

### System-wide configuration

`/etc/marv/marv.conf`

    [marv]
    collections = bags

    # keep db/db.sqlite as the suffix!
    dburi = sqlite:////var/local/lib/marv/db/db.sqlite

    # store could also be somewhere else
    store = /var/local/lib/marv/store

    ...

### Multiple collections

    [marv]
    collections = bags bags2 videos

    [collection bags]
    scanner = marv_robotics.bag:scan

    ...

    [collection bags2]
    scanner = marv_robotics.bag:scan

    ...

    [collection videos]
    scanner = my_own_scanner:scan

    ...