Background on the NetShark architecture
=======================================

.. currentmodule:: steelscript.netshark.core

This document is a brief introduction to the capabilities and
organization of the :term:`NetShark` appliance.  No background or
previous experience with NetShark or :term:`Packet Analyzer` (formerly
:term:`Pilot`) is assumed, but you are encouraged to watch the `Packet
Analyzer with NetShark - An Introduction
<http://www.riverbed.com/about/document-repository/?isSearch=true&14871=1079001&14941=0&14951=0#>`_
and/or spend some time using Packet Analyzer to familiarize yourself
with the platform and its capabilities.

Views
-----

All analysis and reporting from a NetShark appliance revolves around the
concept of a :term:`view`.  Views are extremely flexible and can be
used for many different things but at the core, a view consists simply
of a source of packets to analyze and a set of statistics to be
computed on those packets.  The power of the NetShark platform derives
from the number of statistics that can be exported and the number of
ways those statistics can be combined and organized.

An example of a simple view is "Bandwidth over Time".  In this view,
the *length* statistic is summed across all packets for fixed small
time intervals (e.g., 1 second).  By plotting the sums for a series of
consecutive time intervals, we get a graph of how the traffic load on
a specific network link has varied over time.

A more complex example is the Packet Analyzer "Served Web Pages by
Client IP" view.  In Packet Analyzer, this view presents a table with
three columns: ip address, URL, and count.  There is a separate row in
the table for each unique (client ip address, URL) pair, showing the
number of times the given URL has been fetched from the given client
ip address.  This view could additionally be filtered, for example
showing only URLs fetched using the Chrome web browser or showing only
URLs with mime type ``image/jpeg``.  In this case, the statistics
extracted are more complex -- extracting the URL (or browser or
content type) from an HTTP session requires reassembling the
underlying TCP connection from individual packets.  And the
organization of the statistics is a little more complex, the NetShark
internally constructs and updates a table much like the one presented
in the Packet Analyzer user interface.

These examples are just meant to help illustrate what views are at a
high level.  The details about exactly how views are constructed and
accessed are later in this document and elsewhere in the FlyScript
documentation.

Packet Sources
--------------

This section describes the abstractions provided by a NetShark appliance
for organizing and identifying the packets that can be used as the basis
for creating a view.

Interfaces
~~~~~~~~~~

Perhaps the simplest packet source is a raw network interface, also
called a :term:`capture port`.  A capture port on a NetShark appliance is
typically connected to a switch port that mirrors live traffic to the
NetShark for monitoring and analysis (often called a SPAN port).  When a
view is applied to an interface, every packet that arrives on the
interface is available to the view for processing.  A view applied to
an interface is also a *live* view, meaning that as new packets
arrive, they are immediately processed by the view and show up in the
view outputs.

Capture Jobs
~~~~~~~~~~~~

A :term:`capture job` is a long-running task on a NetShark appliance that
captures, stores, and indexes live traffic.  It logically consists of
a physical network interface and an optional filter to limit which
packets should be captured.  The actual packets and associated
indexing information are all stored locally on the NetShark appliance,
using a custom filesystem tailored to the task of capturing traffic at
a high rate and indexing the capture traffic for efficient queries.
The capabilities of capture jobs are accessed from Python code using
:ref:`capture-job-objects`.

There are actually two different logical packets source associated
with a capture job.  The job itself refers to all the packets stored
on disk that have been previously captured as part of the job.  A view
created with a capture job as the packet source is applied to all
packets that are available on disk, but is *not* applied to new
packets as they arrive.  Instead, each capture job has an associated
*virtual device* that is used for ongoing live traffic.  The virtual
device corresponding to a capture job appears like an interface, as
described above.

The virtual device associated with a capture job may appear redundant
with just applying a view with a filter to an interface.
The filters associated with capture jobs are generally more efficient
than view filters, so on an appliance handling a high sustained
rate of packet capture, using a capture job is preferrable.
A full description of the tradeoffs is beyond the scope of this
document, for more details see the NetShark manual.

Trace Clips
~~~~~~~~~~~

The packets captured as part of a capture job are kept in a
first-in-first-out (FIFO) structured store.  For cases when some
packets need to be retained for longer, a :term:`trace clip` is used.
A trace clip identifies some subset of the packets stored as part of a
capture job and optionally locks them, meaning that those packets will
be kept on disk until the trace clip is deleted.  Note that locking
packets on disk as part of a trace clip effectively reduces the size
of the FIFO store available for other ongoing capture jobs.

The exact set of packets that should go into a trace clip is
identified using filters.  Filters are described more fully below.
But trace clips typically include at least a time filter to select
packets from a specific time interval, and may include filters to
select just traffic to/from a specific IP address and/or a specific
TCP or UDP port number.  Trace clips are accessed in SteelScript via the
methods :py:meth:`NetShark.get_clips` and :py:meth:`Job4.add_clip`.

Extractors
----------

Now that we have established how packets are directed into views, we
turn to the topic of how information is extracted from packets.  The
central concept in this section is :term:`extractor fields <extractor
field>`.  An extractor field is an individual piece of information.
Simple extractor fields are computed from a single packet, typical
examples are the packet length in bytes, or the IP source address of
an IP packet.  More complex extractor fields work on a stream of
packets to compute higher level statistics.  Examples of more
compliated extractor fields are the jitter in a VOIP call or the
number of retransmissions on a TCP connection -- these statistics can
only be derived by looking at the relationship between multiple
packets.

Every extractor field has a name of the form ``extractor.name`` used
to uniquely identify the field, as well as a human-readable
description that briefly explains what the field is.  ``Extractor`` is
a high-level category such as ``generic`` for fields that apply to any
type of packet, ``tcp`` for fields that are specific to TCP, ``voip``
for fields specific to voice-over-IP, etc.  The ``name`` is a unique
short descriptive name.  For example, ``generic.bits`` is the length
of a packet in bits, and ``tcp.src_port`` is the source port from the
TCP header of a TCP packet.  In addition, each field has a fixed type
(e.g., string, integer, IP address, etc.) and always yields values of
that type.

See :ref:`extractor-fields` for details on using extractor fields from
SteelScript.

The next two sections explain how extractor fields are used
in packet filters and in views.

Filters
-------

Packet filters were mentioned above as part of creating views and
trace clips.  As the name suggests, a filter is applied to a packet
source to select some subset of the original packets for view
processing or for inclusion in a trace clip.  NetShark implements two
types of filters: native filters and Wireshark filters.  A native
filter is simply a test applied to an :term:`extractor field`.  For
example, ``ip.src="10.1.2.3"`` matches all packets with source IP
address 10.1.2.3, or ``generic.bytes >= 100`` matches all packets at
least 100 bytes long.

NetShark includes an extensive set of built-in extractor fields for
common protocols including IP, TCP, UDP, RTP, HTTP, and others.  Many
powerful filters can be built using these extractors, but the number
of applications that NetShark has built-in extractors for is modest
compared to the `Wireshark <http://www.wireshark.org>`_ network
analyzer.  Wireshark has well over 100,000 extractor fields -- for
cases that are not covered by the shark built-in extractors, a
wireshark filter can be used instead.  A wireshark filter is specified
using the `Wireshark Display Filter syntax
<http://www.wireshark.org/docs/dfref/>`_.  Although wireshark filters
offer great flexibility, they are implemented by starting a separate
wireshark process and sending packets to that process for filtering,
so they are significantly slower than native filters and may not be
appropriate for applying to very large traces or high-rate live
traffic.

See :ref:`netshark-filters` for details on working with filters from SteelScript.

Inside a View
-------------

We now have all the concepts needed to explain how NetShark views are
constructed.  A view consists of two major pieces: which packets
should be analyzed, and what analysis should be done on those packets.
The first piece (which packets should be analyzed) is specified as a
packet source and optionally, some filters.

The actual analysis is organized around extractor fields.  When a view
is created, a list of extractor fields is supplied, each of which is
designated as either a *key* or a *value*.  We begin with value
fields.  Each value field corresponds to a single output for the view,
but of course there can be many packets processed by the view, so the
field has an associated *calculation* that specifies how the
individual samples are aggregated into a single statistic.  Possible
calculations include summing the individual samples, computing the
average, or finding the minimum or maximum value.

To make this concrete, consider a simple example of a view with just
one field: packet size.  If we use the averaging calculation, this
view computes the average size of all packets in the input.  By
choosing a different calculation we could easily compute the minimum
or maximum packet size, or the total number of bytes across all
packets.  This computation is automatically organized into a series of
discrete time intervals, at a frequency specified when the view is
created.  Using our example above, by using the addition operator to
aggregate packet sizes, we actually get a time series with the total
bandwidth used in each 1 second interval -- effectively a view of
bandwidth over time.  Similarly, with any other extractor fields and
aggregation operators, the view output is a time series indicating how
the aggregated value field has changed over time.

Returning to *key* fields, a view is actually organized as a table
with a column for each extractor field.  If there are no key fields,
as in our examples above, the table simply has one default row where
all statistics are collected.  But when key fields are specified, a
tuple is created consisting of the values of all the key columns, and
a new row is added to the table for each unique tuple.  To make this
concrete, one could create a simple view to compute a list of all
TCP/IP connections by specifying four key columns: source IP address,
source TCP port, destination IP address, and destination TCP port.  In
this way, a new row would be created in the view table for each unique
connection 4-tuple, so reading the contents of the view table yields a
list of all connections.

Combining key and value columns, we can merge our examples above by
creating a view with key columns for the source and destination
addresses and ports and a value column for total packet size.  We now
get a table of all connections with the total number of bytes sent on
each connection.  If we sort this table by the total bytes sent, we
have a "top talkers" report!
