# Output Documentation

This document provides an overview of the structure of the output CSV file generated by the network detection algorithm.
The result file is created by the functions `write_nda_test_results_csv()` and `write_nda_application_results_csv()`.
The output file contains the settings, detailed results of the analysis, and metadata. Below is a description of each column in the output file.
In the end, a short description of a possible result validation process is included.

## Application Results

### Structure
The output is organized as one row per analysis. Each row is structured as follows:
- Experimental and configuration parameters: 15 entries (Content of the configuration file minus the last 8 entries, which are not relevant for analysis)
- Result values: 19 entries
- Additional Info and Metadata: 6 entries

### Analysis
The most important columns to make sense of the result are `network_type`, `found_idcs_0` and `found_idcs_1`, since this can tell us which ROI elements are involved and in what kind of interplay they have an effect.

### Columns
<u> Configuration: </u>

- ``run``: Identifies the specific run or iteration of the analysis (hardcoded right now, default: 1)
- ``bin_method``: See configuration guide.
- ``bin_factor``: See configuration guide.
- ``scoring_method``: See configuration guide.
- ``scoring_thr``: See configuration guide.
- ``scoring_interval``: See configuration guide.
- ``acc_thr``: See configuration guide.
- ``corr_thr``: See configuration guide.
- ``note``: See configuration guide.
- ``fn_flag``: See configuration guide.
- ``subject_id``: See configuration guide.
- ``response_specification``: See configuration guide.
- ``efield_specification``: See configuration guide.

<u> Result: </u>

Let h0 and h1 denote the two found (potential) hotspots.
- ``found_idcs_0``: ROI index of h0.
- ``found_idcs_1``: ROI index of h1.
- ``found_scores_0``: Hotspot score of h0. The higher, the better. Meaning and range depends on the scoring method:
  - 'mi': mutual information score, (value > 0)
  - 'clf': number of promising element combinations involving this element. (value < n_elms)
  - 'regress_data': R2-value of the sigmoidal fit. (value in [0,1])
  - 'regression': R2-value of the multivariable Gaussian fit. (value in [0,1])
- ``found_scores_1``: Hotspot score of h1. (analogous to h0)
- ``found_acc_0``: Accuracy of h0. (value in [0,1]) 
Only differs from the hotspot score when using the CLF-method, in which case this denotes the decision tree classifier accuracy. (= how well the h0 e-field describes the TMS effect)
- ``found_acc_1``: Accuracy of h1. (analogous to h1)
- ``found_hotspots_corr``: Correlation of found hotspots. 
(Measured as the Pearson correlation coefficient of their e-fields.) (value in [0,1])
- ``found_hotspots_distance``: Distance between found hotspots. (Geodesic distance in mm)
- ``found_network_type``: Type of network found, network IDs:
    - (1) `NO`: No network ("pseudonetwork").
    - (2) `AND`: Dual node network: Effect if elm0 AND elm1 are stimulated.
    - (3) `1_INH_0`: Dual node network: elm1 inhibits elm0, elm0 has an effect.
    - (4) `SH_0`: Single hotspot: Only elm0 has an effect.
    - (5) `0_INH_1`: Dual node network: elm0 inhibits elm1, elm1 has an effect.
    - (6) `SH_1`: Single hotspot: Only elm1 has an effect.
    - (7) `XOR`: Dual node network: elm0 inhibits elm1, elm1 inhibits elm0. Both have an effect.
    - (8) `OR`: Dual node network: Effect if either elm0 or elm1 is stimulated.
- ``network_type_certainty``: Certainty of the network type identified. (value in [0,1])
Measured as the maximal entry of the network identification vector divided by its second maximal entry. (How close was the call?)
- ``network_type_vector_[0:8]``: Value of the network type vector for each network type. (in the code: `shape_vector`)
The highest value belongs to the identified network.
- ``sample_size``: The size of the sample analyzed. (n_zaps)

<u> Additional: </u>

- ``response_max``: Maximum response observed.
- ``response_mean``: Mean response.
- ``response_dev``: Standard deviation of the response.
- ``runtime_gen``: Runtime for the data generation step.
- ``runtime_scores``: Runtime for the scoring step.
- ``runtime_eval``: Runtime for the evaluation step.


## Testing Results

### Structure
The output is organized as one row per analysis. Each row is structured as follows:
- Experimental and configuration parameters: 20 entries (Content of the configuration file minus the last 5 entries, which are not relevant for analysis)
- Result values: 20 entries
- Additional Info and Metadata: 6 entries

### Analysis

The most important columns to analyze performance are `network_eval` and `found_eval`. When both of them have the value 1 (True), the detection was completely successful. 

### Columns
<u> Configuration: </u>

- ``run``: Identifies the specific run or iteration of the analysis (hardcoded right now, default: 1)
- `hotspot_idcs_0` : See configuration guide.
- `hotspot_idcs_1`	: See configuration guide.
- `sample_size`	: See configuration guide.
- `rn_seed`	: See configuration guide.
- `dist_type`	: See configuration guide.
- `network_type`	: See configuration guide.
- `real_network_type`	: See configuration guide.
- `effect_full`	: See configuration guide.
- `effect_upper_bound`	: See configuration guide.
- `jitter_ratio`	: See configuration guide.
- `jitter_scale`: See configuration guide.
- ``bin_method``: See configuration guide.
- ``bin_factor``: See configuration guide.
- ``scoring_method``: See configuration guide.
- ``scoring_thr``: See configuration guide.
- ``scoring_interval``: See configuration guide.
- ``acc_thr``: See configuration guide.
- ``corr_thr``: See configuration guide.
- ``note``: See configuration guide.

<u> Result: </u>

Let h0 and h1 denote the two found (potential) hotspots.
- ``found_idcs_0``: ROI index of h0.
- ``found_idcs_1``: ROI index of h1.
- ``found_scores_0``: Hotspot score of h0. The higher, the better. Meaning and range depends on the scoring method:
  - 'mi': mutual information score, (value > 0)
  - 'clf': number of promising element combinations involving this element. (value < n_elms)
  - 'regress_data': R2-value of the sigmoidal fit. (value in [0,1])
  - 'regression': R2-value of the multivariable Gaussian fit. (value in [0,1])
- ``found_scores_1``: Hotspot score of h1. (analogous to h0)
- ``found_acc_0``: Accuracy of h0. (value in [0,1]) 
Only differs from the hotspot score when using the CLF-method, in which case this denotes the decision tree classifier accuracy. (= how well the h0 e-field describes the TMS effect)
- ``found_acc_1``: Accuracy of h1. (analogous to h1)
- ``found_hotspots_corr``: Correlation of found hotspots. 
(Measured as the Pearson correlation coefficient of their e-fields.) (value in [0,1])
- ``found_hotspots_distance``: Distance between found hotspots. (Geodesic distance in mm)
- ``found_network_type``: Type of network found, network IDs:
    - (1) `NO`: No network ("pseudonetwork").
    - (2) `AND`: Dual node network: Effect if elm0 AND elm1 are stimulated.
    - (3) `1_INH_0`: Dual node network: elm1 inhibits elm0, elm0 has an effect.
    - (4) `SH_0`: Single hotspot: Only elm0 has an effect.
    - (5) `0_INH_1`: Dual node network: elm0 inhibits elm1, elm1 has an effect.
    - (6) `SH_1`: Single hotspot: Only elm1 has an effect.
    - (7) `XOR`: Dual node network: elm0 inhibits elm1, elm1 inhibits elm0. Both have an effect.
    - (8) `OR`: Dual node network: Effect if either elm0 or elm1 is stimulated.
- ``network_type_certainty``: Certainty of the network type identified. (value in [0,1])
Measured as the maximal entry of the network identification vector divided by its second maximal entry. (How close was the call?)
- ``network_type_vector_[0:8]``: Value (int) of the network type vector for each network type. (in the code: `shape_vector`)
The highest value belongs to the identified network.
- ``identification_evaluation``: (boolean) Whether the network was identified correctly. (`real_network_type` == `found_network_type`)
- ``localization_evaluation``: (boolean) Whether both hotspots were localized within tolerated area. (which is 10mm around the real hotspots)
- `real_hotspot_dist`: Distance between real hotspots. (Geodesic distance in mm)
- `real_hotspot_corr`: Correlation of real hotspots. 
(Measured as the Pearson correlation coefficient of their e-fields.) (value in [0,1])
- `real_hotspot_emax_0`:	The maximal e-field magnitude of real hotspot 0 over all trials. (As a proxy for stimulatability. The lower the e-mag max, the harder a hotspot is to detect.)
- `real_hotspot_emax_1`:	The maximal e-field magnitude of real hotspot 1 over all trials.
- `found_distance_0`: Distance between `found_hotspot_0` and its assigned real hotspot. (Geodesic distance in mm) (Assignment is done such that overall distances are minimized.)
- `found_distance_1`: Distance between `found_hotspot_1` and its assigned real hotspot. (Geodesic distance in mm)
- `num_hotspot-candidates`: The number of elements with a hotspot score > 0. (Only meaningful in `clf`-method - if even.)

<u> Additional: </u>

- ``response_max``: Maximum response observed.
- ``response_mean``: Mean response.
- ``response_dev``: Standard deviation of the response.
- ``runtime_gen``: Runtime for the data generation step.
- ``runtime_scores``: Runtime for the scoring step.
- ``runtime_eval``: Runtime for the evaluation step.


## Validation procedure 

A challenging part of validation is finding a response measure and according ROI with a reasonable assumption about a network effect 
being in place.
Assuming that is the case, validation of NDA results in a second session could be done as follows:

Based on all known e-field data `e_matrix` from the first session and the detection result, an effect map can be created using the function 
`write_effect_map_hdf5()`.
To create that map, for every ROI element of interest (`elm_i`), an expected effect value is estimated: 
- Of all available stimulations/zaps, the one 
inducing the highest e-field value on element `elm_i` is chosen. 
- For that zap, the e-field magnitudes of the hotspot elements are taken into consideration and using the network type and 
its effect function (see `create_artificial_response_data()`), the expected effect for that zap is computed. 
(E.g. Assuming the hotspot elements are stimulated at this certain value, and form the network type OR, how high is the effect on the response expected to be?)
- This effect value is assigned to the ROI element `elm_i`. 

Applying this procedure to each element leads to an initial ROI effect map, on which stimulation points of interest can be chosen (e.g. maxima, mimima,...)
For those specific stimulation configurations, a more precise effect value can be computed, since the initial map was 
just a proxy (because the needed stimulation positions may not have been included in the first session.). 
After measuring the response values for these stimulation points of interest, their values and deviation from the 
baseline response can be compared to the predicted effect values. 
