= Documentation for Researchers
:!toc:
:icons: font
:stem: latexmath
:last-update-label!:
:favicon: ddl_favicon_black.svg
:showtitle!:


This part of the documentation is addressed at the _users_ of the Data Donation Module (i.e., the researchers) and will explain:

- How to set up a data donation project.
- How to monitor an ongoing data donation project.
- How to download and work with the collected data.

It is assumed that a server with DDM set up on it is running.
Readers interested in setting up a server to run DDM for institutional use or their own research should turn to the xref:for_admins.adoc[Documentation for Administrators].
For a general introduction to the module, see xref:index.adoc[this page].


== Setting Up a Data Donation Project

=== Creating a Project

To set up a new data donation project, got to the `/projects/` page and click on "+ Create New Project".
On the project creation page, you will be presented with the following options to configure your project:

Project Name:: Name of the project. Visible to participants in the browser's title bar or a page's tab.

URL Identifier:: Identifier that is included in the URL through which participants can access the project (e.g, https://root.url/project-slug).

Contact Information:: Contact information of the researcher responsible for the project.
Is linked in the footer of the donation interface and can be viewed by data donors at any stage of the data donation process.

Data Protection Statement:: Data protection statement that describes how the data is processed.
Is linked in the footer of the donation interface and can be viewed by data donors at any stage of the data donation process.

Super Secret:: When creating a "super secret" project, you will have to provide
a project password, which will be used to encrypt all collected data donations and survey responses
(for more information on how collected data is encrypted, see xref:topics/encryption.adoc[here]).
This password will not be saved by the application, and the data collected for
this project can only be encrypted by entering the password that was used as a super secret
when the project was created.
Please note that there is no way to recover or reset this password if it is lost.
Furthermore, also note that making a project "super secret" will limit the
functionality of the module: it won’t be possible to create followup questions based on the data donation
(i.e., data points from the data donation cannot be referenced in a question).


=== The Project Hub

Once a project is created, you will be redirected to the "Project Hub".
From here, you can access all settings and information that are relevant for your data donation project.
It consists of four main areas:

1. Project Details

2. Project Configuration

3. Data Center

4. Danger Zone



=== Editing Project Details

==== Base Settings

Project Name:: Name of the project. Visible to participants in the browser's title bar or a page's tab.

External Project Slug:: Identifier that is used to expose the project to participants (e.g, https://root.url/project-slug ).

==== Public Project Information

Contact Information:: Contact information of the researcher responsible for the project.
Is linked in the footer of the donation interface and can be viewed by data donors at any stage of the data donation process.

Data Protection Statement:: Data protection statement that describes how the data is processed.
Is linked in the footer of the donation interface and can be viewed by data donors at any stage of the data donation process.


==== URL Parameter Extraction

Optionally, information can be extracted from parameters passed with the URL when a project is accessed by a participant.
This can be configured using the following settings:

URL parameter extraction enabled:: Enable or disable whether URL parameters should be extracted when participants access the project's briefing page.

Expected URL parameter:: Provide a string containing the parameters that should be extracted.
Separate multiple parameters with a semikolon (e.g., "parameter_A;parameter_B"). The extracted values will
be saved for each participant and is included in the data export. If a parameter is not present, it will be saved
as `None`. Undefined parameters passed in the URL will be ignored.

[TIP]
====
If participants are asked to first fill out a questionnaire created in an external
survey software such as e.g. Unipark or Qualtrics, passing URL parameters can
be used to link data donations with the survey data by passing a participant
ID in the URL parameters.
====


==== Redirect Configuration

Optionally, participants can be redirected to another website from the debriefing page.
This can be configured using the following settings:

Redirect enabled:: Enable or disable the redirection of your participants when they have completed your project.
If enabled, a redirect button will be displayed on the data donation end page that redirects to the URL defined in the `Redirect target` setting.

Redirect target:: URL to which participants will redirected (only required if `redirect` is enabled).

[TIP]
====
The redirect URL can include variables that are populated with the respective values. Currently, the following variables are supported:
An extracted URL parameter (see above), the participant id (as ``ddm_participant_id``), or the project id (as ``ddm_project_id``).
Use the following syntax to include variables in the redirect target: e.g. ``https://redirect.url?param={{URLParameter}}&participant={{ddm_participant_id}}&project={{ddm_project_id}}``
====


==== Project Appearance

The following settings are available to customize the appearance of your data donation project:

Header Image Left/Header Image Right:: Upload an image that will be displayed in the header of your project (e.g., an institution or project logo).


=== Editing Project Configuration

The structure of the Project Configuration follows the steps of the prototypical
data donation process. It consists of the following sections:

1. Welcome Page: Define what is displayed to participants when they enter your project.
2. Data Donation: Define the expected data donations, extraction rules, and donation instructions.
3. Questionnaire: Define questions that will be shown to participants after they have donated their data.
4. End Page: Define what is displayed when participants reach the end of the data donation.

==== Briefing

Briefing Text:: Text displayed to participants on the briefing page.

Briefing Consent Mandatory:: If briefing consent is enabled, participants will
have to explicitly indicate their consent at the bottom of the briefing page before
they can continue. If a participant does indicate that they to do not consent,
they will be redirected to the debriefing page.

Briefing consent label yes/Briefing consent label no:: The labels displayed to
participants to indicate consent ("briefing consent label yes") or reject
consent ("briefing consent label no").


==== Data Donation

The data donation is organized in `File Uploaders` and `Donation Blueprints`.

A "File Uploader" corresponds to the file that is expected to be uploaded.
This file can either be a single file (e.g., a JSON file) or a ZIP container.

For each File Uploader, a set of `Instructions` for participants can be defined
that describe how they can access and upload the requested file.

Each uploader has one or multiple associated `Donation Blueprints` (although
if a File Uploader expects a single file, only one Donation Blueprint can
be associated with it).
A Donation Blueprint defines how the data contained in a single file
(e.g., the uploaded file in the case of a single file upload, or a file contained
in the ZIP container in the case of a ZIP upload) is extracted.

The data donation step can incorporate multiple File Uploaders.

===== Configure File Uploader

Name:: Name of the File Uploader. Will be publicly visible to participants in the header of the file uplaoder.

Upload Type:: Either "single file" or "zip file".

Index:: The position of the file uploader on the data donation page.
Only relevant if multiple file uploaders are displayed – file uploaders with a
lower index will be displayed closer to the top of the page.

Associated Donation Blueprints:: The donation blueprints that apply to the expected file(s) collected with the file uploader.


===== Configure Instructions

Donation Instructions consist of one or multiple instruction pages.
Instruction pages are displayed as a slide show at the top of file uploader.
For each instruction page, the following can be configured:

Text:: The instruction text displayed to the participants.
By default, researchers can also upload and include images or gifs to guide
participants through the data donation process in this field (video upload is not supported -
but videos hosted externally can still be embedded).  //TODO: Check if video can be embedded without problems.

Index:: The position of the page in the slideshow.

NOTE:: If no instructions are defined for a File Uploader, the instruction section will
be hidden in the participation view.

===== Configure Donation Blueprint

Name:: Name of the expected data donations. Will be publicly visible to participants.
Therefore, it is important to define a meaningful name.

Description:: Description of what the blueprint will extract. If defined, the
description will be visible for participants in the data donation step.

Expected File Format:: The file format of the expected data donation. Currently, only JSON and CSV is implemented.

Associated File Uploader:: The `File Uploader` for which the blueprint will be applied.

Regex path:: Here, the path where the file is expected to be located within a ZIP file is defined.
Only necessary, if the Donation Blueprint is part of a Blueprint Container.

WARNING: If a regex expression matches two files, DDM extracts the first one that matches the expression.
Afterwards, it does not look any further, even if the matched file does not match the expected fields.
Therefore, be careful to choose regex expressions that will only match the expected file.

[TIP]
====
*Examples for regex paths to match files*

[%header,cols=2*]
|===
|Regex
|Description

|`^MyActivities.json`
|Matches a file named `MyActivities.json` that is located at the root of the ZIP file.

|`^SpecificFolder/MyActivities\.json`
|Matches a file named `MyActivities.json` that is located in a folder named `SpecificFolder` in the root of the ZIP file.

|`.*MyActivities\.json`
|Matches file if the filename ends with `MyActivities.json` that can be located anywhere in the ZIP file. Warning: This also matches e.g. `BogusMyAcitivties.json`.

|`(\^MyActivities\.json\|^MeineAktivitäten\.json\|^MieAttivita\.json)`
|Matches a file that is located at the root of the ZIP file and either named `MyActivities.json`, `MeineAktivitäten.json`, or `MieAttivita.json`.
Can be helpful to match the same file in different languages.
|===

You can find about more about regex https://developer.mozilla.org/en-US/docs/Web/JavaScript/Guide/Regular_Expressions[here].
On this website, you will https://developer.mozilla.org/en-US/docs/Web/JavaScript/Guide/Regular_Expressions#tools[also find some Tools]
that can help you test regex patterns.
====

Expected fields:: The fields that must be contained in the donated file. If a file does not contain
one or more of the fields defined here, it will not be accepted as a donation.
Put the field names in double quotes (") and separate them with commas ("Field A", "Field B").


// TODO: CSV specific settings


// TODO: JSON specific settings


===== Extraction Rules

The base assumption for the extraction of the data contained in an uploaded file is
that you do not want any data. This means that when you configure your extraction rules,
you first have to add a rule for each field that you want to keep in your data.

Execution Order:: The order in which the extraction rules are applied to a file.

Name:: The name of an extraction rule. For internal organisation only.

Field:: The field to which the rule will be applied.

Extraction Operator:: Defines the main logic of the extraction step. If empty, this indicates
that you want to keep the field in the donated data. For all non-RegEx operations,
if an operations evaluates to True, the row will be deleted from the donated data
(further explanations on the separate rules will follow).

Comparison Value:: The value against which the data contained in the indicated field will be compared according to the selected comparison logic.

Replacement Value:: Only required for operation "Replace match (regex)". The value
that will be used as a replacement if the RegEx pattern matches.


==== Questionnaire

Researchers can optionally define a questionnaire consisting of one or more pages, each consisting of one or more questions.
The questions will be displayed after the data donation.

Currently, the following question types are implemented:

* Single Choice Question
* Multi Choice Question
* Matrix Question
* Semantic Differential
* Open Question
* Transition Block (plain text, without any response options for the participant)

Depending on the question types, the following attributes can be configured:

Name:: Question name - only used for internal organisation.

Blueprint (optional):: If associated to a blueprint, the data extracted by
this blueprint for a given participant is available to be included in the question text
(see [LOCATION] for more information on how to include data in question texts). // TODO: Add link.
If the associated blueprint did not extract any data, the question will not be displayed.
If a question should always be displayed, select the option "General Question" here.

Page:: Number of the page on which the question should be displayed.

Text:: The question text that is displayed to participants.  // TODO: Extend description to explain inclusion of donated data.

Required:: If a question is marked as required, the application will show a
hint to the participant if they missed to answer this question.
This hint will only be shown once. This means that if a participant chooses to
ignore the hint and clicks on 'continue', they are able to skip a required question.

Randomize items:: Enable or disable randomization of items.

Question Items:: The items that will be displayed in the answer part of the question.

Scale Configuration:: Configure how the scale displayed to participants to rate items will look like
(only for Matrix Question and Semantic Differential).


[TIP]
====
*How to Include Donated Data in a Question*

It is possible to include information contained in the donated data or related to the participant in the question text.
Questions that should integrate this behaviour must be linked to a donation blueprint
The donated data related to the associated blueprint will then be available as a
context variable an can be referenced in the question body as follows:
`Display donated data: {{ data }}. Display participant data: {{ participant }}.`

For this, DDM utilizes the `Django template engine <https://docs.djangoproject.com/en/3.2/topics/templates/>`_.
The donated data will be available as a template variable "data" in the question text definition.
This variable can be combined flexibly with Django's `built-in template tags and filters <https://docs.djangoproject.com/en/3.2/ref/templates/builtins/>`_.

// TODO: Include Example.

// TODO: Include a list of accessible template features.
====


==== Debriefing

Debriefing text:: Text displayed to participants on the briefing page.


=== Data Center

In the Data Center, you can find options to `A` access the collected data,
`B` access the project logs, and `C` find some general field statistics about the progress of your project.

==== Data Download: Accessing Collected Data Donations

When downloading your data, you will receive a .json file with the following structure:

[source,json]
----
{
  'project': {
      'pk': 1,
      'name': 'project name',
      'date_created': '2022-12-19T08:49:18.363880+01:00'
  },
  'donations': {
    'blueprint name 1': [
      {'time_submitted': '2022-12-19T08:49:18.363880+01:00',
       'consent': '_',  # TODO: List options
       'status': '_',  # TODO: List options
       'data': [{'extracted_field_1':  'value', 'extracted_field_2':  'value2', ...}],
       'project': 1,
       'participant': 1
      },  # ...
    ],
    'blueprint name 2': [ ... ]
  },
  'responses': [{'time_submitted':  '', 'data':  {}, 'project':  1, 'participant':  1}],
  'participants': [
    {'pk':  1, 'project':  1, 'external_id':  '_', 'start_time':  '_', 'end_time':  '_', 'completed':  '_', 'extra_data':  '_'}
  ]
}
----

There are two options to download your data:

1. Internal Download
When you are logged in, click on {download label}. This will gather your data from the database and
you will be able to download the json file in your browser.


2. External Download via API (advanced option)
There is also the possibility to download your data through an API endpoint remotely.
For this, an API token has to be created which will need to be supplied when sending
the request to the API (see below for an example).


[TIP]
----
Through the admin interface, a project can only be accessed by the user who created it.
This means that the internal download is only accessible for this project owner.
The API token, on the other hand, can be used to share data access with colleagues working on the same project.
However, be careful with whom you share this token as it exposes the sensitive data collected from participants.
We recommend to choose a short expiration date for Tokens that you create.
----


// TODO: Add Example with R

// TODO: Add Example with Python

==== Data Structure


==== Project Log: Monitoring an on-going Project

// TODO: Describe Exception Logs.

==== Participation Statistics

// TODO: Describe.



=== Danger Zone

Here, you can find all options that affect the data collected in the course of your project:

==== Reset Project Data

With this option, you can delete all data collected for a given project.

==== Delete Participant

You can delete the data for a given participant by providing their external participation id.
// TODO: Explain this function further.

==== Delete Project

With this option you can delete the current project. This will also delete all associated data.
