# Writing a `pyfeeds` test


 * [Writing a test](#writing-a-test)
 * [Multiple tests](#multiple-tests)
 * [External/shared data](#externalshared-data)
 * [Test input parameters](#test-input-parameters)
 * [Writing your test in Python](#writing-your-test-in-python)
 * [Other examples](#other-examples)


---


A `pyfeeds` test is simply an executable file which is called `feedsRun`, and
which is contained within its own sub-directory. The `feedsRun` file should be
an executable script, or binary, which runs a test.  You can write your
`feedsRun` test in any language you like; you just need to make sure that it
[accepts three parameters](#test-input-parameters), and exits with a return
value of zero to indicate that the test passed, or non-zero to indicate that
the test failed.


Instead of writing a single `feedsRun` script, you may also write multiple
`feedsRun` scripts, giving each of them a unique suffix,
e.g. `feedsRun.test1`, `feedsRun.test2`, etc.


If you need data to test your software, you can store it alongside your test,
as long as it is not too large (more than a few MB).  Or, your test may make
use of [shared data resources](#externalshared-data).


We recommend that you either:

 - Include your `pyfeeds` tests alongside the source code for your project, in
   your version control repository of choice
   (e.g. [git](https://git.fmrib.ox.ac.uk),
   [CVS](http://cvs.fmrib.ox.ac.uk/cgi-bin/cvsweb.cgi), etc).
 - Add your tests to the
   [pyfeeds-tests](https://git.fmrib.ox.ac.uk/fsl/pyfeeds-tests) repository.


## Writing a test


A `pyfeeds` test must be contained within its own sub-directory.  This test
directory may contain data, libraries, code, etc, but above all else must
contain an executable file called `feedsRun`, or multiple executables called
(e.g.) `feedsRun.a`, `feedsRun.b`, etc. . As an example, let's consider the
`examples/flirt/` test:


```
$ ls examples/flirt/

examples/flirt/feedsRun
examples/flirt/input.nii.gz
examples/flirt/xform.mat

$
```


This is a simple test which uses `flirt` to apply an affine transformation
(`xform.mat`) to `input.nii.gz`. The `feedsRun` script is as follows:



```bash
#!/bin/bash -e
#
# A `pyfeeds` test which transforms an image
#

outdir=$1

# Apply the transform to the data
flirt -in ./input.nii.gz                                    \
      -ref $FSLDIR/data/standard/MNI152_T1_2mm_brain.nii.gz \
      -applyxfm                                             \
      -init ./xform.mat                                     \
      -out $outdir/data_xformed.nii.gz
```


`pyfeeds` will then compare generated file with a previously stored benchmark
file. If the files match, the test will pass.


## Self-evaluation


`pyfeeds` will evaluate all of the files generated by each test, but a
`feedsRun` script can also evaluate its own output. The directory containing
benchmark data is passed to `feedsRun` scripts as the third command-line
argument.


The following example shows how to modify the above script to include more
than one test within this single script.  This is generally encouraged so that
the testing is more extensive and especially when running the command takes a
significant amount of time:


```bash
#!/bin/bash -e
#
# A FEEDS test which transforms an image, and does its own
# evaluation by comparing the result to a image
# benchmark using two different metrics.

outdir=$1
benchmarkdir=$3

# Apply the transform to the data
flirt -in ./input.nii.gz                                    \
      -ref $FSLDIR/data/standard/MNI152_T1_2mm_brain.nii.gz \
      -applyxfm                                             \
      -init ./xform.mat                                     \
      -out $outdir/data_xformed.nii.gz

benchmark=$benchmarkdir/data_xformed.nii.gz

# Calculate the difference between the
# transformed image and the benchmark image
fslmaths $outdir/data_xformed -sub $benchmark $outdir/diff.nii.gz

# Calculate the mean error (normalised) between
# the flirt output and the benchmark
norm=`fslstats $benchmark -a -m`
error1=`fslstats $outdir/diff -a -m`
error1=`echo $error1 / $norm | bc -l`
error2=`fslstats $outdir/diff -P 99`
error2=`echo $error2 / $norm | bc -l`

thresh1=0.001; thresh2=0.01;

# If normalised error is < thresh, we
# consider that the test is a pass
if [[ `echo "$thresh1 < $error1" | bc` -eq "1" ]]; then
    echo "Test failed: error1 = $error1"
    exit 1
fi
if [[ `echo "$thresh2 < $error2" | bc` -eq "1" ]]; then
    echo "Test failed: error2 = $error2"
    exit 1
fi
exit 0
```


## External/shared data


If your test has low data requirements (up to a few MB), you can simply store
the test data in the test directory. When the test script runs, this data will
be available in the current directory. For example, in our `examples/flirt/`
test above, the `feedsRun` script assumes that its `input.nii.gz`,
`benchmark.nii.gz` and `xform.mat` files are in the same location as the
`feedsRun` script itself.


For self-contained tests such as this (i.e. where all of the required test
data is stored in the test directory), you can completely ignore the data
directory (the second parameter passed to the test script).  However, if your
test requires access to some large, shared data source, you must specify the
files and directories to which your test needs access.  This can be done by
creating a plain text file in the test directory, called `feedsInputs`, which
simply contains a list of the files and directories that your test needs.


The `examples/flirt_shared/` test is a modification of the `examples/flirt/`
test which accesses its data from an external data directory. Its
`feedsInputs` file looks like this:


```
flirt/input.nii.gz
flirt/xform.mat
```

 > The shared data directory will be made available in a known common location
 > on our file system (e.g. `/vols/Data/fsldev/dataSets/`).  In your
 > `feedsInputs` file, you must specify your data paths relative to this
 > location, such as in the example paths above.


And the `feedsRun` script looks something like the following:


```bash
#!/bin/bash -e
#
# A FEEDS test which registers one image to another.

outdir=$1
indir=$2

# Apply the transform to the data
flirt -in $indir/flirt/input.nii.gz                   \
      -ref $FSLDIR/data/standard/MNI152_T1_2mm.nii.gz \
      -applyxfm                                       \
      -init $indir/flirt/xform.mat                    \
      -out $outdir/data_xformed.nii.gz
```


## Test input parameters


A `pyfeeds` test is called with three parameters:


   - *Test output directory*: This is a directory in which all test output
     must be stored. It may be used as a workspace for temporary files, or for
     storing log files or other outputs. Note that this directory will be
     created before the test script is run, so you do not need to create it
     yourself.

   - *Data directory*: This is a directory in which any external data required
     by the test can be found - this is explained in the [External/shared
     data](#externalshared-data) section. This parameter will contain a valid
     value only for tests which have specified a `feedsInputs` file.

   - *Benchmark directory*: This is a directory which contains the
     previously-generated benchmark data for the test, if it exists.


## Writing your test in Python


While `bash` is great for simple tests, there are some tasks which would
unwieldy to write in `bash`. The `flirt_python` example provides a simple
example of a `pyfeeds` test written in Python:


```python
#!/usr/bin/env python
#
# A FEEDS test which uses FLIRT to register one image to another.

import               sys
import               os
import os.path    as op
import subprocess as sp

outDir  = sys.argv[1]
inDir   = sys.argv[2]
fslDir  = os.environ['FSLDIR']

inimg    = op.join(inDir,   'flirt', 'input')
refimg   = op.join(fslDir,  'data', 'standard', 'MNI152_T1_2mm_brain')
outimg   = op.join(outDir,  'input_xformed')
outxform = op.join(outDir,  'xform.mat')

sp.call('flirt -in {} -ref {} -out {} -omat {}'.format(
    inimg, refimg, outimg, outxform).split())
```


## Other examples


All of these scripts can be found in [pyfeeds/examples/](examples/), along
with a few more examples. Shared and benchmark data (for those example scripts
which use it) can be respectively found in
[pyfeeds/exampleInputData](exampleInputData/) and
[pyfeeds/exampleBenchmarkData](exampleBenchmarkData/).


---


### Example 1


This example loads some data from a CSV file, and runs it through an
algorithm. The result is then compared against the previously-generated
benchmark data.


```
$ ls examples/csv_data/

feedsRun
data.csv

$
```


The `feedsRun` script is as follows:


```python
#!/usr/bin/env python
#
# An example FEEDS test which loads some data from
# a CSV file, and uses it to test an algorithm.
#

import sys
import os.path as op
import numpy   as np


# Read in the command line arguments.
outdir = sys.argv[1]

def fancy_algorithm_that_needs_testing(data):
    return data.mean()


# Load the data from the CSV file.
data = np.loadtxt('data.csv', delimiter=',')

# Run the algorithm.
result = fancy_algorithm_that_needs_testing(data)

with open(op.join(outdir, 'result.txt'), 'wt') as f:
    f.write('{:6.2f}\n'.format(result))
```


---


### Example 2


This example test is similar in structure to the previous example, except that
this test loads some image data from a shared data directory. It runs the data
through an algorithm. `pyfeeds` then compares the result to a known benchmark.
The test sub-directory contains the following:


```
$ ls examples/shared_data/

feedsRun
feedsInputs

$
```


The `feedsInputs` file specifies the shared data resources used by the test:


```
exampleData/input.nii.gz
```


And the `feedsRun` script is as follows:


```python
#!/usr/bin/env python
#
# An example FEEDS test which loads some data from
# a shared data directory, and uses it to test an
# algorithm.
#

import sys
import os.path as op
import nibabel as nib


# Read in the command line arguments.
outdir  = sys.argv[1]
indir   = sys.argv[2]


def fancy_algorithm_that_needs_testing(data):
    return (data + 5) * 20


# Load the input data. The exampleData
# directory contains an input image,
# and a reference image which has been
# generated by the algorithm, and which
# we know to be correct. We can use the
# reference image as a benchmark for
# testing.
inimg = nib.load(op.join(indir, 'exampleData', 'input.nii.gz'))

# Get access to the image data
# through the nibabel wrapper.
indata = inimg.get_data()

# Run the algorithm, save its output
processedData = fancy_algorithm_that_needs_testing(indata)

image = nib.Nifti1Image(processedData,
                        inimg.get_affine(),
                        inimg.header)

nib.save(image, op.join(outdir, 'output.nii.gz'))
```
