Metadata-Version: 2.1
Name: gawseed-processing
Version: 0.9.1
Summary: A data processing engine for statistically analyzing time series data
Home-page: https://github.com/gawseed/processing
Author: Wes Hardaker and USC/ISI
Author-email: opensource@hardakers.net
License: UNKNOWN
Platform: UNKNOWN
Classifier: Programming Language :: Python :: 3
Classifier: Operating System :: OS Independent
Requires-Python: >=3.0
Description-Content-Type: text/markdown

# Overview

The GAWSEED processing system allows easy statistical analysis of
time-series data via simple YAML configuration files.  This
architecture:

+ Collect information from data sources
+ Transform it to a standardized tab-separated format (called [FSDB])
+ Temporally bin, count and tabulate features within the dataset, producing a triple indexed output
+ Manipulate individual results using aggregation functions
+ Use extensible mathematical functions to combine multi-column datasets into final temporal waveform that are suitable for feature detection
+ Perform simple, distributed edge-detection to identify signals of interest.

# Architecture

![Architecture Image](./images/architecture.png)

## Incoming Data Source and Filter

## Data Formatter

## Feature Counter

The *Feature Counter* component is responsible for converting all
timeseries records into ``time bins'' and simultaneously counting or
quantifying the data.  The output records will consist of named
*index* keywords extracted from the YAML configuration, and a
*key* and *subkey* value that is extracted from the data
itself (if the second subkey column is not needed, an empty string is
typically used).

Example feature counter yaml config:

	binSize: 5
	timeColumn: time
	featureCounter:
	  outputs:
	    http_host_names:
	      function: gawseed.algorithm.generic.re_match_one
	      arguments: ['col(dnsname)', '.*https?://(^[/]+)/.*']
	    http_type:
	      function: gawseed.algorithm.generic.re_match_one
	      arguments: ['col(dnsname)', '(https:|http:)']

## Aggregator

The aggregator component of the GAWSEED architecture is responsible
for performing any calculations that require a complete dataset for a
given combination of the set *<timebin, index, key>*.  Summing
values from distributed operations and counting of unique subkeys are
the most common computations that occur at this step.

An example YAML configuration for this step in the data architecture
is shown in below.  This outputs both the total counts of the things
counted in the featureCounter step(s), and counts the number of unique
*http_host_names* values seen as well.

Example YAML configuration for the aggregator:

	aggregator:
	  aggregators:
	    - function: summer
	    - function: unique
	      arguments: ['http_host_names', 'unique_http_host_names']

## Relationship Analyzer

The Relationship Analyzer is where any heavy final math is performed.
For example, in the example data in this document, we may want to
calculate the ``popularity'' of each discovered *http_host_names*.
The YAML configuration to do this is shown below:

	relationshipAnalysis:
	  outputs:
	    host_name_popularity_fraction:
	      function: fraction_otherindex_keyval
	      arguments:
	        - unique_http_host_names
	        - http_host_names

Note that the Relationship Analyzer also pivots
the table back to a more normal looking table (at least for humans),
as the distributed processing aspects of the calculations are now
over; specifically, the previous index values are converted back into
table columns.

## Edge Detection and Result Analysis

TBD

# Copyright




