# How to use `pyfeeds`


This page covers the basics of using `pyfeeds` - for advanced usage, including
controlling what tests get run and how they are evaluated, see the page on
[configuring `pyfeeds`](doc/configuring_pyfeeds.md).


`pyfeeds` expects you to provide it with one or more directories, in which one
or more tests are contained. A test can be contained anywhere in a specified
test directory - `pyfeeds` recursively searches through them for tests.  A `pyfeeds`
test is simply a directory which contains an executable file called `feedsRun`
- see the page on [writing a test](doc/writing_a_test.md) for more details on
`pyfeeds` tests.


> A pyfeeds directory may alternately contain multiple `feedsRun` scripts, each
> with a different suffix (e.g. `feedsRun.test1`, `feedsRun.test2`, etc.


To get help on how to use `pyfeeds`, simply type `pyfeeds --help`.  `pyfeeds`
has a number of commands which allow you to run and manage the tests that it
finds. To get help on one of these commands, type `pyfeeds <command> --help`.


`pyfeeds` provides the following core commands:


 - `list`:   List available tests
 - `run`     Run tests
 - `bundle`  Bundle tests
 - `compare` Compare two directories


---


`list`


The `list`  command lists the tests that `pyfeeds` detects:


```
$ pyfeeds list examples/

Available tests

Test name     | Test directory
------------- | -----------------------------------------------------
call_cmd      | /Projects/pyfeeds/examples/call_cmd
csv_data      | /Projects/pyfeeds/examples/csv_data
feat          | /Projects/pyfeeds/examples/feat
flirt         | /Projects/pyfeeds/examples/flirt
flirt_shared  | /Projects/pyfeeds/examples/flirt_shared
nested/flirt  | /Projects/pyfeeds/examples/nested/flirt
nested/ntest1 | /Projects/pyfeeds/examples/nested/ntest1
shared_data   | /Projects/pyfeeds/examples/shared_data

$
```


---


`run`


The `run` command runs all detected tests. You need to give `pyfeeds` the
following pieces of information:

 - `-i`: The location of shared input data
 - `-b`: The location of benchmark data to compare the test outputs against
 - `-o`: Where to store the test outputs


If you are running tests which do not use any input data, or do not have any
benchmark data, you do not need to provide those options.

```
$ pyfeeds run -i exampleInputData -b exampleBenchmarkData -o ./exampleOutput examples/

2015-11-25 11:23:41,563 INFO - 8 tests to run
2015-11-25 11:23:42,021 INFO - Running test call_cmd ...
2015-11-25 11:23:43,029 INFO - Running test csv_data ...
2015-11-25 11:23:43,113 INFO - Running test feat ...
2015-11-25 11:26:46,354 INFO - Running test flirt ...
2015-11-25 11:26:46,706 INFO - Running test flirt_shared ...
2015-11-25 11:26:47,066 INFO - Running test nested/flirt ...
2015-11-25 11:26:47,071 INFO - Running test nested/ntest1 ...
2015-11-25 11:26:47,077 INFO - Running test shared_data ...
Test                 | Time (seconds) | Result | Data status
-------------------- | -------------- | ------ | -----------
call_cmd             | 0              | Passed | Passed
csv_data             | 0              | Passed | No data
feat                 | 186            | Failed | Passed
flirt                | 0              | Passed | No data
flirt_shared         | 0              | Passed | Passed
nested/flirt         | 0              | Passed | No data
nested/ntest1        | 0              | Passed | No data
shared_data          | 0              | Passed | Passed
2016-04-06 18:33:05,657 INFO - 7 / 8 tests passed in 3.12 minutes
$
```


If you have multiple cores on your test machine, you can run tests in parallel
by providing the `--jobs`/`-j` argument. For example, the following command
will execute up to four tests in parallel:


```
$ pyfeeds run -j 4 -i exampleInputData -b exampleBenchmarkData -o ./exampleOutput examples/
```


In addition to checking the exit status of each test, `pyfeeds` compares all
files that were generated by the test with the files in the benchmark
directory, using various evaluation routines. If any of the files that were
output by the test fail this evaluation, the test will fail.


For each test, `pyfeeds` will create a file called `feedsResults.log` in the
test's output directory, containing the results for every file that was
evaluated. See the page on [configuring `pyfeeds`](doc/configuring_pyfeeds.md)
for details on how to control which evaluation routines are used.


---


`bundle`


The `bundle` command will copy all detected tests, and associated data, to
a new directory. This is useful for extracting a sub-set of tests. Let's say
we want all tests related to FLIRT:


```
$ pyfeeds bundle -i exampleInputData -o exampleBundle examples/flirt/ examples/flirt_shared/ examples/nested/flirt/

2015-11-25 11:28:04,511 INFO - Adding flirt to bundle...
2015-11-25 11:28:04,513 INFO - Adding flirt_shared to bundle...
2015-11-25 11:28:04,516 INFO - Adding nested_flirt to bundle...

$
```

---


`compare`

The `compare` command allows you to simply compare two directories. It is
useful when you have some benchmark data, and some data generated by a test,
and wish to compare them without re-running the test. `pyfeeds` will
recursively compare every file in the two directories, and will output the
results to the specified output file:


```
$ pyfeeds compare test_output benchmark_output -o results.txt

$
```


---


`pyfeeds` has two other commands, `genhash` and `checkhash`.  These commands
are used for test verification, and are explained in [How `pyfeeds`
works](doc/how_pyfeeds_works.md).
