SCINGESTOR(1)

NAME
scingestor - SciCat Dataset Ingestor Documentation

SCICAT_DATASET_INGESTOR

  Description

BeamtimeWatcher service SciCat Dataset ingestor.

  Synopsis

scicat_dataset_ingestor [-h] [-c CONFIG] [-r RUNTIME] [-l LOG]

Options:

-h, --help
    show this help message and exit

-c CONFIG, --configuration CONFIG
    configuration file name

-r RUNTIME, --runtime RUNTIME
    stop program after runtime in seconds

-l LOG, --log LOG
    logging level, i.e. debug, info, warning, error, critical

  Example

scicat_dataset_ingestor -c ~/.scingestor.yaml
scicat_dataset_ingestor -c ~/.scingestor.yaml -l debug

SCICAT_DATASET_INGEST

  Description

Re-ingestion script for SciCat RawDatasets.

  Synopsis

scicat_dataset_ingest [-h] [-c CONFIG] [-r RUNTIME] [-l LOG]

Options:

-h, --help
    show this help message and exit

-c CONFIG, --configuration CONFIG
    configuration file name

-l LOG, --log LOG
    logging level, i.e. debug, info, warning, error, critical

  Example

scicat_dataset_ingest -c ~/.scingestor.yaml
scicat_dataset_ingest -c ~/.scingestor.yaml -l debug

SCINGESTOR PACKAGE

  Submodules

  scingestor.beamtimeWatcher module

class scingestor.beamtimeWatcher.BeamtimeWatcher(options)
    Bases: object

    Beamtime Watcher

find_bt_files(path, prefix, postfix)
    find beamtime files with given prefix and postfix in the given path

Parameters

•   path (str) – beamtime directory

•   prefix (str) – file name prefix

•   postfix (str) – file name postfix

Returns
    list of found files

Return type
    list <str>

start()
    start beamtime watcher

stop()
    stop beamtime watcher

scingestor.beamtimeWatcher.main()
    the main program function

  scingestor.configuration module

scingestor.configuration.load_config(configfile)
    load config file

Parameters
    configfile (str) – configuration file name

  scingestor.datasetIngest module

class scingestor.datasetIngest.DatasetIngest(options)
    Bases: object

    Dataset Ingest command

find_bt_files(path, prefix, postfix)
    find beamtime files with given prefix and postfix in the given path

Parameters

•   path (str) – beamtime directory

•   prefix (str) – file name prefix

•   postfix (str) – file name postfix

Returns
    list of found files

Return type
    list <str>

ingest_scandir(path, meta, bpath)
    constructor

Parameters

•   path (str) – scan dir path

•   meta (dict <str,`any`>) – beamtime configuration

•   bpath (str) – beamtime file

start()
    start ingestion

scingestor.datasetIngest.main()
    the main program function

  scingestor.datasetIngestor module

class scingestor.datasetIngestor.DatasetIngestor(configuration, path, dsfile, idsfile, meta, beamtimefile, delay=5)
    Bases: object

    Dataset Ingestor

check_list(reingest=False)
    update waiting and ingested datasets

clear_tmpfile()
    clear waitings datasets

clear_waiting_datasets()
    clear waitings datasets

get_token()
    provides ingestor token

Returns
    ingestor token

Return type
    str

ingest(scan, token)
    ingest scan

Parameters

•   scan (str) – scan name

•   token (str) – access token

ingested_datasets()
    provides ingested datasets

Returns
    ingested datasets list

Return type
    list <str>

reingest(scan, token)
    ingest scan

Parameters

•   scan (str) – scan name

•   token (str) – access token

update_from_tmpfile()
    clear waitings datasets

waiting_datasets()
    provides waitings datasets

Returns
    waitings datasets list

Return type
    list <str>

  scingestor.datasetWatcher module

class scingestor.datasetWatcher.DatasetWatcher(configuration, path, dsfile, idsfile, meta, beamtimefile, delay=5)
    Bases: threading.Thread

    Dataset Watcher

run()
    scandir watcher thread

stop()
    stop the watcher

  scingestor.logger module

scingestor.logger.get_logger()

scingestor.logger.init_logger(name='scingestor.logger', level='debug')
    init logger

  scingestor.safeINotifier module

class scingestor.safeINotifier.EventData(name, masks)
    Bases: object

    event data

class scingestor.safeINotifier.SafeINotifier(*args, **kwargs)
    Bases: threading.Thread

    singleton wrapper for inotifyx

add_watch(path, masks)
    add watch to notifier

Parameters

•   path (str) – watch path

•   mask (int) – watch mask

Returns
    queue providing events and its id

Return type
    [threading.queue, int]

init()
    constructor

rm_watch(qid)
    add watch to notifier

Parameters
    qid (int) – queue id

run()
    scandir watcher thread

stop()
    stop the watcher

  scingestor.scanDirWatcher module

class scingestor.scanDirWatcher.ScanDirWatcher(configuration, path, meta, bpath, delay=5)
    Bases: threading.Thread

    ScanDir Watcher

run()
    scandir watcher thread

stop()
    stop the watcher

  Module contents

•   genindex

•   modindex

•   search

AUTHOR
Jan Kotanski

COPYRIGHT
2022, DESY, Jan Kotanski