SCINGESTOR(1)

NAME
scingestor - SciCat Dataset Ingestor Documentation

The scingestor python package provides a support for scripts which ingest Datasets and OrigDatablocks into the SciCat metadata server.

SCICAT_DATASET_INGESTOR

SciCat Dataset ingestor server ingests scan metadata just after a scan is finished. It can be executed by

scicat_dataset_ingestor -c ~/.scingestor.yaml

  Configuration variables

The configuration written in YAML can contain the following variables

•   scicat_url (str) , default: "http://localhost:3000/api/v3"

•   ingestor_credential_file (str) , default: None

•   beamtime_dirs (list<str>) , default: []

•   beamtime_base_dir (str) , default: ""

•   ingestor_var_dir (str) , default: ""

•   ingestor_username (str) , default: "ingestor"

•   dataset_pid_prefix (str) , default: ""

•   dataset_update_strategy ("no", "patch", "create", "mixed") , default: "patch"

•   relative_path_in_datablock (bool) , default: False

•   scandir_blacklist (list<str>) , default: ["/gpfs/current/scratch_bl", "/gpfs/current/processed", "/gpfs/current/shared"]

•   beamtimeid_blacklist_file (str) , default: None

•   beamtime_type_blacklist (list<str>) , default: ["P"]

•   chmod_json_files (str) , default: None

•   max_scandir_depth (int), default: -1

•   oned_in_metadata (bool) , default: False

•   max_oned_size (int) , default: None

•   scan_metadata_postfix (str) , default: ".scan.json"

•   datablock_metadata_postfix (str) , default: ".origdatablock.json"

•   attachment_metadata_postfix (str) , default: ".attachment.json"

•   metadata_in_var_dir (bool) , default: False

•   use_corepath_as_scandir (bool) , default: False

•   beamtime_filename_postfix (str) , default: "beamtime-metadata-"

•   beamtime_filename_prefix (str) , default: ".json"

•   datasets_filename_pattern (str) , default: "scicat-datasets-{beamtimeid}.lst"

•   ingested_datasets_filename_pattern (str) , default: "scicat-ingested-datasets-{beamtimeid}.lst"

•   file_dataset_metadata_generator (str) , default: "nxsfileinfo metadata -k4 -o {metapath}/{scanname}{scanpostfix} -b {beamtimefile} -p {beamtimeid}/{scanname} -w {ownergroup} -c {accessgroups} {scanpath}/{scanname}.{ext}"

•   dataset_metadata_generator (str) , default: "nxsfileinfo metadata -k4 -o {metapath}/{scanname}{scanpostfix} -c {accessgroups} -w {ownergroup} -b {beamtimefile} -p {beamtimeid}/{scanname}"

•   datablock_metadata_generator (str) , default: "nxsfileinfo origdatablock -s *.pyc,*{datablockpostfix},*{scanpostfix},*~ -p {doiprefix}/{beamtimeid}/{scanname} -w {ownergroup} -c {accessgroups} -o {metapath}/{scanname}{datablockpostfix} "

•   datablock_metadata_stream_generator (str) , default: "nxsfileinfo origdatablock -s *.pyc,*{datablockpostfix},*{scanpostfix},*~ -w {ownergroup} -c {accessgroups} -p {doiprefix}/{beamtimeid}/{scanname} "

•   datablock_metadata_generator_scanpath_postfix (str) , default: " {scanpath}/{scanname} "

•   attachment_metadata_generator (str) , default: "nxsfileinfo attachment -w {ownergroup} -c {accessgroups} -o {metapath}/{scanname}{attachmentpostfix} "

•   plot_file_extension_list (list<str>) , default: ["png", "nxs", "h5", "ndf", "nx", "fio"]

•   master_file_extension_list (list<str>) , default: ["nxs", "h5", "ndf", "nx", "fio"]

•   chmod_generator_switch (str) , default: " -x {chmod} "

•   relative_path_generator_switch (str) , default: " -r {relpath} "

•   oned_dataset_generator_switch (str) , default: " --oned "

•   max_oned_dataset_generator_switch (str) , default: " --max-oned-size {maxonedsize} "

•   override_attachment_signals_generator_switch (bool) , default: " --override "

•   hidden_attributes_generator_switch (str) , default: " -n {hiddenattributes} "

•   hidden_attributes (str) , default: "nexdatas_source,nexdatas_strategy,units"

•   attachment_signals_generator_switch (str) , default: " -s {signals} "

•   attachment_axes_generator_switch (str) , default: " -e {axes} "

•   attachment_frame_generator_switch (str) , default: " -m {frame} "

•   attachment_signal_names (str) , default: ""

•   attachment_axes_names (str) , default: ""

•   attachment_image_frame_number (str) , default: ""

•   ingest_dataset_attachment (bool) , default: False

•   override_attachment_signals (bool) , default: False

•   log_generator_commands (bool) , default: False

•   add_empty_units_generator_switch (str) , default: " --add-empty-units "

•   add_empty_units (bool) , default: True

•   metadata_copy_map_file (str) , default: None

•   metadata_copy_map_file_generator_switch (str) , default: " --copy-map-file {copymapfile} "

•   inotify_timeout (float) , default: 0.1

•   get_event_timeout (float) , default: 0.01

•   ingestion_delay_time (float) , default: 5.0

•   max_request_tries_number (int) , default: 100

•   recheck_dataset_list_interval (int) , default: 1000

•   recheck_beamtime_file_interval (int) , default: 1000

•   request_headers (dict<str,str>) , default: {"Content-Type": "application/json", "Accept": "application/json"}

•   scicat_datasets_path (str) , default: "Datasets"

•   scicat_proposals_path (str) , default: "Proposals"

•   scicat_datablocks_path (str), default: "OrigDatablocks"

•   scicat_users_login_path (str), default: "Users/login"

•   scicat_attachments_path (str), default: "Datasets/{pid}/Attachments"

•   owner_access_groups_from_proposal (bool), default: False

•   metadata_keywords_without_checks (list<str>), default: ["techniques", "classification", "createdBy", "updatedBy", "datasetlifecycle", "numberOfFiles", "size", "createdAt", "updatedAt", "history", "creationTime", "version", "scientificMetadata", "endTime"]

    e.g.

    beamtime_dirs:
      - "{homepath}/gpfs/current"
      - "{homepath}/gpfs/commissioning"
    scicat_url: http://localhost:3000/api/v3
    ingestor_credential_file: "{homepath}/gpfs/pwd"

  Pattern keywords for configuration variables

The datasets_filename_pattern, ingested_datasets_filename_pattern and ingestor_var_dir can contain the {beamtimeid} and {hostname} keywords, e.g. "scicat-ingested-datasets-{beamtimeid}.lst" or "scicat-ingested-datasets-{hostname}-{beamtimeid}.lst" which is instantiated during the ingestor execution.

Similarly, file_dataset_metadata_generator, dataset_metadata_generator, datablock_metadata_generator, datablock_metadata_stream_generator, datablock_metadata_generator_scanpath_postfix, attachment_metadata_generator, chmod_generator_switch, relative_path_generator_switch can contain the following keywords: {beamtimeid} , {scanname}, {chmod}, {scanpath}, {metapath}, {relpath}, {beamtimeid}, {beamline}, {doiprefix}, {beamtimefile}, {scanpostfix}, {datablockpostfix}, {ownergroup}, {accessgroups}, {hostname}, {hiddenattributes}, {ext}

SCICAT_DATASET_INGEST

Re-ingestion script for SciCat Datasets and OrigDatablocks is usually launched at the end of the beamtime.

scicat_dataset_ingest -c ~/.scingestor.yaml

Its configuration written YAML like for scicat_dataset_ingestor

SCICAT_INGEST

General ingestion script for SciCat Models could be used for manual scicat model ingestion, e.g. Sample, Instrument or DerivedDataset.

scicat_ingest  -m Samples  -c ~/.scingestor.yaml  ./metadata.json

Its configuration written YAML like for scicat_dataset_ingestor

  Required packages

•   python3 >= 3.7

•   nxstools >= 3.38.0

•   inotifyx (python3 version)

•   requests

•   setuptools

•   pyyaml

•   pytest (to run tests)

•   sphinx (to build the documentation)

  Install from sources

The code from https://github.com/jkotan/scingestor can be built with

python3 setup.py install

To build the documentation use

python3 setup.py build_sphinx

The resulting documentation can be found below build/sphinx/html in the root directory of the source distribution.

Finally, the package can be tested using

python3 -m pytest test

  Install in conda or pip environment

The code can be installed in your conda environment by

conda create -n myenv python=3.9
conda activate myenv
pip install inotifyx-py3
pip install scingestor

or in your pip environment by

python3 -m venv myvenv
. myvenv/bin/activate
pip install inotifyx-py3
pip install scingestor

  Debian and Ubuntu packages

Debian bookworm, bullseye, buster or Ubuntu lunar , jammy, focal packages can be found in the HDRI repository.

To install the debian packages, add the PGP repository key

sudo su
curl -s http://repos.pni-hdri.de/debian_repo.pub.gpg  | gpg --no-default-keyring --keyring gnupg-ring:/etc/apt/trusted.gpg.d/debian-hdri-repo.gpg --import
chmod 644 /etc/apt/trusted.gpg.d/debian-hdri-repo.gpg

and then download the corresponding source list, e.g. for bookworm

cd /etc/apt/sources.list.d
wget http://repos.pni-hdri.de/bookworm-pni-hdri.list

or jammy

cd /etc/apt/sources.list.d
wget http://repos.pni-hdri.de/jammy-pni-hdri.list

respectively.

Finally,

apt-get update
apt-get install python3-scingestor

SCICAT_DATASET_INGESTOR

  Description

BeamtimeWatcher service SciCat Dataset ingestor.

  Synopsis

scicat_dataset_ingestor [-h] [-c CONFIG] [-r RUNTIME] [-l LOG] [-f LOGFILE] [-t]

Options:

-h, --help
    show this help message and exit

-c CONFIG, --configuration CONFIG
    configuration file name

-r RUNTIME, --runtime RUNTIME
    stop program after runtime in seconds

-l LOG, --log LOG
    logging level, i.e. debug, info, warning, error, critical

-f LOGFILE, --log-file LOGFILE
    log file name

-t, --timestamps
    timestamps in logs

  Example

scicat_dataset_ingestor -c ~/.scingestor.yaml
scicat_dataset_ingestor -c ~/.scingestor.yaml -l debug

SCICAT_DATASET_INGEST

  Description

Re-ingestion script for SciCat Datasets.

  Synopsis

scicat_dataset_ingest [-h] [-c CONFIG] [-r RUNTIME] [-l LOG] [-f LOGFILE] [-t]

Options:

-h, --help
    show this help message and exit

-c CONFIG, --configuration CONFIG
    configuration file name

-l LOG, --log LOG
    logging level, i.e. debug, info, warning, error, critical

-f LOGFILE, --log-file LOGFILE
    log file name

-t, --timestamps
    timestamps in logs

  Example

scicat_dataset_ingest -c ~/.scingestor.yaml
scicat_dataset_ingest -c ~/.scingestor.yaml -l debug

SCICAT_INGEST

  Description

Re-ingestion script for SciCat Datasets.

  Synopsis

scicat_ingest [-h] [-c CONFIG] [-r RUNTIME] [-l LOG] [-f LOGFILE] [-t] [-p TOKENFILE]   metadata_json_file [metadata_json_file ...]

Arguments:
    metadata_json_file metadata json file(s)

Options:

-h, --help
    show this help message and exit

-c CONFIG, --configuration CONFIG
    configuration file name

-l LOG, --log LOG
    logging level, i.e. debug, info, warning, error, critical

-f LOGFILE, --log-file LOGFILE
    log file name

-t, --timestamps
    timestamps in logs

-p TOKENFILE, --token-file TOKENFILE
    file with a user token

  Example

scicat_ingest -m Samples -c ~/.scingestor.yaml ./metadata.json
scicat_ingest -m Attachments -c ~/.scingestor.yaml -p ~/.mytoken.cfg ./metadata.json

SCINGESTOR PACKAGE

  Submodules

  scingestor.beamtimeWatcher module

class scingestor.beamtimeWatcher.BeamtimeWatcher(options)
    Bases: object

    Beamtime Watcher

    constructor

Parameters
    options (argparse.Namespace) -- parser options

running
    (bool) running loop flag

start()
    start beamtime watcher

stop()
    stop beamtime watcher

scingestor.beamtimeWatcher.main(interrupt=0)
    the main program function

Parameters
    interrupt (int) -- test interrupt flag: 1:keyboard, 2:signal

  scingestor.configuration module

scingestor.configuration.load_config(configfile)
    load config file

Parameters
    configfile (str) -- configuration file name

  scingestor.datasetIngest module

class scingestor.datasetIngest.DatasetIngest(options)
    Bases: object

    Dataset Ingest command

    constructor

Parameters
    options (argparse.Namespace) -- parser options

start()
    start ingestion

scingestor.datasetIngest.main()
    the main program function

  scingestor.datasetIngestor module

class scingestor.datasetIngestor.DatasetIngestor(configuration, path, dsfile, idsfile, meta, beamtimefile)
    Bases: object

    Dataset Ingestor

    constructor

Parameters

•   configuration (dict <str, any>) -- dictionary with the ingestor configuration

•   path (str) -- scan dir path

•   dsfile (str) -- file with a dataset list

•   dsfile -- file with a ingester dataset list

•   meta (dict <str, any>) -- beamtime configuration

•   beamtimefile (str) -- beamtime filename

•   pidprefix (str) -- pidprefix

•   ingestorcred (str) -- ingestor credential

•   scicat_url (str) -- scicat_url

append_proposal_groups()
    appends owner and access groups to beamtime

Parameters

•   meta (dict <str, any>) -- beamtime configuration

•   path (str) -- base file path

Returns
    updated beamtime configuration

Return type
    dict <str, any>

check_list(reingest=False)
    update waiting and ingested datasets

clear_tmpfile()
    clear waitings datasets

clear_waiting_datasets()
    clear waitings datasets

get_token()
    provides ingestor token

Returns
    ingestor token

Return type
    str

ingest(scan, token)
    ingest scan

Parameters

•   scan (str) -- scan name

•   token (str) -- access token

ingested_datasets()
    provides ingested datasets

Returns
    ingested datasets list

Return type
    list <str>

reingest(scan, token)
    re-ingest scan

Parameters

•   scan (str) -- scan name

•   token (str) -- access token

update_from_tmpfile()
    clear waitings datasets

waiting_datasets()
    provides waitings datasets

Returns
    waitings datasets list

Return type
    list <str>

class scingestor.datasetIngestor.UpdateStrategy(value)
    Bases: enum.Enum

    Update strategy

CREATE = 2
    (scingestor.datasetIngestor.UpdateStrategy) recreate datasets

MIXED = 3
    (scingestor.datasetIngestor.UpdateStrategy) patch datasets only if scientificMetadata changed otherwise recreate datasets

NO = 0
    (scingestor.datasetIngestor.UpdateStrategy) leave datasets unchanged

PATCH = 1
    (scingestor.datasetIngestor.UpdateStrategy) patch datasets

  scingestor.datasetWatcher module

class scingestor.datasetWatcher.DatasetWatcher(configuration, path, dsfile, idsfile, meta, beamtimefile)
    Bases: threading.Thread

    Dataset Watcher

    constructor

Parameters

•   configuration (dict <str, any>) -- dictionary with the ingestor configuration

•   path (str) -- scan dir path

•   dsfile (str) -- file with a dataset list

•   dsfile -- file with a ingester dataset list

•   meta (dict <str, any>) -- beamtime configuration

•   beamtimefile (str) -- beamtime filename

run()
    scandir watcher thread

running
    (bool) running loop flag

stop()
    stop the watcher

  scingestor.logger module

class scingestor.logger.AccSecFormatter(fmt=None, datefmt=None, style='%', validate=True, *, defaults=None)
    Bases: logging.Formatter

    micro-second formatter

    Initialize the formatter with specified format strings.

    Initialize the formatter either with the specified format string, or a default as described above. Allow for specialized date formatting with the optional datefmt argument. If datefmt is omitted, you get an ISO8601-like (or RFC 3339-like) format.

    Use a style parameter of '%', '{' or '$' to specify that you want to use one of %-formatting, str.format() ({}) formatting or string.Template formatting in your format string.

    Changed in version 3.2: Added the style parameter.

converter()
    timestamp[, tz] -> tz's local time from POSIX timestamp.

formatTime(record, datefmt=None)
    Return the creation time of the specified LogRecord as formatted text.

    This method should be called from format() by a formatter which wants to make use of a formatted time. This method can be overridden in formatters to provide for any specific requirement, but the basic behaviour is as follows: if datefmt (a string) is specified, it is used with time.strftime() to format the creation time of the record. Otherwise, an ISO8601-like (or RFC 3339-like) format is used. The resulting string is returned. This function uses a user-configurable function to convert the creation time to a tuple. By default, time.localtime() is used; to change this for a particular formatter instance, set the 'converter' attribute to a function with the same signature as time.localtime() or time.gmtime(). To change it for all formatters, for example if you want all logging times to be shown in GMT, set the 'converter' attribute in the Formatter class.

scingestor.logger.get_logger()
    provides logger object

Return type
    logging.logger

Returns
    logger object

scingestor.logger.init_logger(name='scingestor.logger', level='debug', timestamps=False, logfile=None)
    init logger

Parameters

•   name (str) -- logger name

•   level (str) -- logging level

•   timestamps (bool) -- timestamps flag

•   logfile (str) -- logger file name

  scingestor.modelIngest module

class scingestor.modelIngest.ModelIngest(options)
    Bases: object

    Dataset Ingest command

    constructor

Parameters
    options (argparse.Namespace) -- parser options

get_token()
    provides ingestor token

Returns
    ingestor token

Return type
    str

start()
    start ingestion

scingestor.modelIngest.main()
    the main program function

  scingestor.pathConverter module

class scingestor.pathConverter.PathConverter(corepath, blpath, usecorepath=False)
    Bases: object

    Path Converter

    constructor

Parameters

•   corepath (str) -- core path

•   blpath (str) -- beamline path

•   usecorepath (bool) -- enabled flag

from_core(path)
    converts core path to notify path

Parameters
    path (str) -- core path

Returns
    notify path

Return type
    str

to_core(path)
    converts notify path to core path

Parameters
    path (str) -- notify path

Returns
    core path

Return type
    str

  scingestor.safeINotifier module

class scingestor.safeINotifier.EventData(name, masks)
    Bases: object

    event data

    constructor

Parameters

•   name (str) -- name

•   masks -- mask description

masks
    (str) mask

name
    (str) name

class scingestor.safeINotifier.SafeINotifier(*args, **kwargs)
    Bases: threading.Thread

    singleton wrapper for inotifyx

    This constructor should always be called with keyword arguments. Arguments are:

    group should be None; reserved for future extension when a ThreadGroup class is implemented.

    target is the callable object to be invoked by the run() method. Defaults to None, meaning nothing is called.

    name is the thread name. By default, a unique name is constructed of the form "Thread-N" where N is a small decimal number.

    args is the argument tuple for the target invocation. Defaults to ().

    kwargs is a dictionary of keyword arguments for the target invocation. Defaults to {}.

    If a subclass overrides the constructor, it must make sure to invoke the base class constructor (Thread.__init__()) before doing anything else to the thread.

add_watch(path, masks)
    add watch to notifier

Parameters

•   path (str) -- watch path

•   mask (int) -- watch mask

Returns
    queue providing events and its id

Return type
    [queue.Queue, int]

daemon = True
    (bool) make notifier to be a daemon

init()
    constructor

rm_watch(qid)
    remove watch from notifier

Parameters
    qid (int) -- queue id

run()
    scandir watcher thread

stop()
    stop the watcher

  scingestor.scanDirWatcher module

class scingestor.scanDirWatcher.ScanDirWatcher(configuration, path, meta, beamtimefile, depth)
    Bases: threading.Thread

    ScanDir Watcher

    constructor

Parameters

•   configuration (dict <str, any>) -- dictionary with the ingestor configuration

•   path (str) -- scan dir path

•   meta (dict <str, any>) -- beamtime configuration

•   beamtimefile (str) -- beamtime file

•   depth (int) -- scandir depth level

run()
    scandir watcher thread

running
    (bool) running loop flag

stop()
    stop the watcher

  Module contents

•   genindex

•   modindex

•   search

AUTHOR
Jan Kotanski

COPYRIGHT
2022, DESY, Jan Kotanski