Rucio demo — Rucio 1.2 documentation

# Rucio demo

## Starting a Rucio demo instance

The instructions can be found in Setting up a Rucio demo environment

## Boostrap the Rucio demo

Once everything is ready you can log into the container and start playing around with rucio:

    $ docker exec -i -t demo_rucio_1 /bin/bash

The bash tab completion is by default enabled for the Rucio CLIs. The rucio configuration file is located in /opt/rucio/etc/rucio.cfg. The clients are configured to talk to the local server instance:

    $ cat /opt/rucio/etc/rucio.cfg
    ...
    [client]
    rucio_host = https://localhost:443
    auth_host = https://localhost:443
    ...

In this demo, an apache server runs and the log files are located in ´/var/log/rucio/´ To query the Rucio server, you can ping it with:

    $ curl -k https://localhost/ping
     {"version": "1.14.9.post1"}

In the corresponding apache log file, you can see the access log entry:

    $ tail -f  /var/log/rucio/httpd_access_log
      localhost - - [27/Feb/2018:11:57:59 +0000] "GET /ping HTTP/1.1" 200 27

The equivalent command exists with the rucio CLI:

    $ rucio ping
    cannot get auth_token
    2018-02-27 13:22:54,297       ERROR   Cannot authenticate.
    Details: userpass authentication failed

But it fails since the command also authenticates with Rucio and this Rucio instance is not configured.

## Configuring Rucio

A bootstrap script is provided at /setup_data.py. This script creates the database tables and creates 2 accounts: root and jdoe and 2 local Rucio storage elements (RSEs): SITE1_DISK, SITE2_DISK mounted on the /tmp partition.

This example script uses the Rucio python client module.To execute it:

    $ /setup_data.py

By default, it will create a MySQL database as specified in the configuration:

    $ cat /opt/rucio/etc/rucio.cfg
     ...
     [database]
     default = mysql://rucio:rucio@mysql/rucio
     ...

To create a different one like sqlite, you need to change this section, e.g.,:

    $ cat /opt/rucio/etc/rucio.cfg
     ...
     [database]
     default = sqlite:////tmp/rucio.db
     ...

Execute the script and restart apache:

    $ /setup_data.py
    $  httpd -k restart

Rucio supports mysql, mariadb, oracle and postgresql.

rucio ping now works:

    $ rucio ping
    1.14.9.post1

The token is stored in /tmp/root/.rucio_root/auth_token_root.

The equivalent can be done with the Rucio python clients:

    $ python
     ...
     >>> from rucio.client import Client
     >>> rucio_client = Client()
     >>> rucio_client.ping()
     {u'version': u'1.14.9.post1'}

You can also check your account:

    [root@3a6d4527e1f6 rucio]# rucio whoami
    status     : ACTIVE
    account    : root
    account_type : SERVICE
    created_at : 2018-02-08T15:37:26
    suspended_at : None
    updated_at : 2018-02-08T15:37:26
    deleted_at : None
    email      : None

To list the RSEs:

    $ rucio list-rses
    SITE1_DISK
    SITE2_DISK

We can add some RSE attributes:

    $ rucio-admin rse set-attribute  --rse SITE1_DISK --key zone --value eu-west-3
    Added new RSE attribute for SITE1_DISK: zone-eu-west-3

    $ rucio-admin rse set-attribute  --rse SITE2_DISK --key zone --value us-west-1
    Added new RSE attribute for SITE2_DISK:  rucio list-rses --expression 'zone=us-west-1'

    $ rucio list-rses --expression 'zone=eu-west-3'
       SITE1_DISK

To list the accounts:

    $ # rucio-admin account list
    jdoe
    root

## Testing dataset upload, creation of dataset and rules

There are no datasets created yet. To generate datasets and copy them to one of the RSEs, you can use a daemon called automatix:

    [root@3a6d4527e1f6 rucio]# /usr/bin/rucio-automatix --run-once --input-file /opt/rucio/etc/automatix.json
    ...
    2018-02-19 13:47:07,532 277     DEBUG   https://localhost:443 "POST /dids/tests/test.24659.automatix_stream.recon.AOD.917/dids HTTP/1.1" 201 None
    2018-02-19 13:47:07,533 277     INFO    Thread [1/1] : Upload operation for tests:test.24659.automatix_stream.recon.AOD.917 done
    2018-02-19 13:47:07,534 277     INFO    Thread [1/1] : Run with once mode. Exiting
    2018-02-19 13:47:07,541 277     INFO    Thread [1/1] : Graceful stop requested
    2018-02-19 13:47:07,541 277     INFO    Thread [1/1] : Graceful stop done

The daemon has created and uploaded a new dataset in the tests scope. One can list all the DIDs in this scope:

    [root@3a6d4527e1f6 rucio]# rucio list-dids tests:*
    +-------------------------------------------------+--------------+
    | SCOPE:NAME                                      | [DID TYPE]   |
    |-------------------------------------------------+--------------|
    | tests:AOD.a9753781316c4b2f8bd88c60e9dd3570      | FILE         |
    | tests:AOD.fc50eb5e2b1949919880f8218bf62108      | FILE         |
    | tests:test.24659.automatix_stream.recon.AOD.917 | DATASET      |
    +-------------------------------------------------+--------------+

And one can list the content of the dataset:

    [root@3a6d4527e1f6 rucio]# rucio list-files tests:test.24659.automatix_stream.recon.AOD.917
    +--------------------------------------------+--------------------------------------+-------------+------------+----------+
    | SCOPE:NAME                                 | GUID                                 | ADLER32     | FILESIZE   | EVENTS   |
    |--------------------------------------------+--------------------------------------+-------------+------------+----------|
    | tests:AOD.a9753781316c4b2f8bd88c60e9dd3570 | 32C89A5A-F0BD-43F6-A958-099C46954C7F | ad:480900d5 | 1.000 MB   |          |
    | tests:AOD.fc50eb5e2b1949919880f8218bf62108 | 1937B5B8-BFE3-4AE0-B5CE-28AFE964F5F8 | ad:32bf834e | 1.000 MB   |          |
    +--------------------------------------------+--------------------------------------+-------------+------------+----------+
    Total files : 2
    Total size : 2.000 MB

Now if you list the rule for this dataset, you will see that there are no files listed on SITE1_DISK:

    [root@3a6d4527e1f6 rucio]# rucio list-rules tests:test.24659.automatix_stream.recon.AOD.917
    ID                                ACCOUNT    SCOPE:NAME                                       STATE[OK/REPL/STUCK]    RSE_EXPRESSION      COPIES  EXPIRES (UTC)    CREATED (UTC)
    --------------------------------  ---------  -----------------------------------------------  ----------------------  ----------------  --------  ---------------  -------------------
    7744c0e0dcce4243b906a2afbc8bc87f  root       tests:test.24659.automatix_stream.recon.AOD.917  OK[0/0/0]               SITE1_DISK               1                   2018-02-19 13:47:06

The information needs to be updated by another daemon called the judge. To run it once:

    [root@3a6d4527e1f6 rucio]# /usr/bin/rucio-judge-evaluator --run-once
    2018-02-19 13:47:37,943 328     DEBUG   re_evaluator[0/0] index query time 0.003242 fetch size is 1
    2018-02-19 13:47:37,951 328     INFO    Re-Evaluating did tests:test.24659.automatix_stream.recon.AOD.917 for ATTACH
    2018-02-19 13:47:38,010 328     DEBUG   Creating locks and replicas for rule 7744c0e0dcce4243b906a2afbc8bc87f [0/0/0]
    2018-02-19 13:47:38,011 328     DEBUG   Creating OK Lock tests:AOD.a9753781316c4b2f8bd88c60e9dd3570 for rule 7744c0e0dcce4243b906a2afbc8bc87f
    2018-02-19 13:47:38,011 328     DEBUG   Creating OK Lock tests:AOD.fc50eb5e2b1949919880f8218bf62108 for rule 7744c0e0dcce4243b906a2afbc8bc87f
    2018-02-19 13:47:38,030 328     DEBUG   Rule 7744c0e0dcce4243b906a2afbc8bc87f  [2/0/0] queued 0 transfers
    2018-02-19 13:47:38,031 328     DEBUG   queue requests
    2018-02-19 13:47:38,031 328     DEBUG   Finished creating locks and replicas for rule 7744c0e0dcce4243b906a2afbc8bc87f [2/0/0]
    2018-02-19 13:47:38,045 328     DEBUG   re_evaluator[0/0]: evaluation of tests:test.24659.automatix_stream.recon.AOD.917 took 0.101811

After this one can see that the 2 files from the dataset are located at SITE1_DISK:

    [root@3a6d4527e1f6 rucio]# rucio list-rules tests:test.24659.automatix_stream.recon.AOD.917
    ID                                ACCOUNT    SCOPE:NAME                                       STATE[OK/REPL/STUCK]    RSE_EXPRESSION      COPIES  EXPIRES (UTC)    CREATED (UTC)
    --------------------------------  ---------  -----------------------------------------------  ----------------------  ----------------  --------  ---------------  -------------------
    7744c0e0dcce4243b906a2afbc8bc87f  root       tests:test.24659.automatix_stream.recon.AOD.917  OK[2/0/0]               SITE1_DISK               1                   2018-02-19 13:47:06

One can then create another rule:

    [root@3a6d4527e1f6 rucio]# rucio add-rule tests:test.24659.automatix_stream.recon.AOD.917 1 SITE2_DISK
    [root@3a6d4527e1f6 rucio]# rucio list-rules tests:test.24659.automatix_stream.recon.AOD.917
    ID                                ACCOUNT    SCOPE:NAME                                       STATE[OK/REPL/STUCK]    RSE_EXPRESSION      COPIES  EXPIRES (UTC)    CREATED (UTC)
    --------------------------------  ---------  -----------------------------------------------  ----------------------  ----------------  --------  ---------------  -------------------
    7744c0e0dcce4243b906a2afbc8bc87f  root       tests:test.24659.automatix_stream.recon.AOD.917  OK[2/0/0]               SITE1_DISK               1                   2018-02-19 13:47:06
    f528e0681ebd404c90d534b7f7a254be  root       tests:test.24659.automatix_stream.recon.AOD.917  REPLICATING[0/2/0]      SITE2_DISK               1                   2018-02-19 13:51:42

Then you can download with:

    [root@3a6d4527e1f6 rucio]# rucio download tests:test.24659.automatix_stream.recon.AOD.917
    2018-02-19 19:32:22,868 INFO    Thread 1/2 : Starting the download of tests:AOD.a9753781316c4b2f8bd88c60e9dd3570
    2018-02-19 19:32:22,869 INFO    Thread 2/2 : Starting the download of tests:AOD.fc50eb5e2b1949919880f8218bf62108
    2018-02-19 19:32:22,922 INFO    Thread 1/2 : File tests:AOD.a9753781316c4b2f8bd88c60e9dd3570 trying from SITE1_DISK
    2018-02-19 19:32:22,922 INFO    Thread 2/2 : File tests:AOD.fc50eb5e2b1949919880f8218bf62108 trying from SITE1_DISK
    ...
    2018-02-19 19:32:23,410 INFO    Thread 1/2 : File tests:AOD.a9753781316c4b2f8bd88c60e9dd3570 successfully downloaded. 1.000 MB in 0.11 seconds = 9.09 MBps
    2018-02-19 19:32:23,410 1032    INFO    Thread 1/2 : File tests:AOD.a9753781316c4b2f8bd88c60e9dd3570 successfully downloaded. 1.000 MB in 0.11 seconds = 9.09 MBps
    ----------------------------------
    Download summary
    ----------------------------------------
    DID tests:test.24659.automatix_stream.recon.AOD.917
    Total files :                                 2
    Downloaded files :                            2
    Files already found locally :                 0
    Files that cannot be downloaded :             0

To delete the rule:

    [root@3a6d4527e1f6 rucio]# rucio update-rule --lifetime -7200 f528e0681ebd404c90d534b7f7a254be
    Updated Rule

    [root@3a6d4527e1f6 rucio]# /usr/bin/rucio-judge-cleaner --run-once
    2018-02-19 19:59:16,258 1388    DEBUG   rule_cleaner[0/0] index query time 0.008735 fetch size is 1
    2018-02-19 19:59:16,258 1388    INFO    rule_cleaner[0/0]: Deleting rule f528e0681ebd404c90d534b7f7a254be with expression SITE2_DISK
    2018-02-19 19:59:16,273 1388    DEBUG   Deleting lock tests:AOD.a9753781316c4b2f8bd88c60e9dd3570 for rule f528e0681ebd404c90d534b7f7a254be
    2018-02-19 19:59:16,281 1388    DEBUG   Deleting lock tests:AOD.fc50eb5e2b1949919880f8218bf62108 for rule f528e0681ebd404c90d534b7f7a254be
    2018-02-19 19:59:16,359 1388    DEBUG   rule_cleaner[0/0]: deletion of f528e0681ebd404c90d534b7f7a254be took 0.100267

    [root@3a6d4527e1f6 rucio]# rucio list-rules tests:test.24659.automatix_stream.recon.AOD.917
    ID                                ACCOUNT    SCOPE:NAME                                       STATE[OK/REPL/STUCK]    RSE_EXPRESSION      COPIES  EXPIRES (UTC)    CREATED (UTC)
    --------------------------------  ---------  -----------------------------------------------  ----------------------  ----------------  --------  ---------------  -------------------
    7744c0e0dcce4243b906a2afbc8bc87f  root       tests:test.24659.automatix_stream.recon.AOD.917  OK[2/0/0]               SITE1_DISK               1                   2018-02-19 13:47:06