
.. _building-simulation-data:

Building simulation data
========================

Simulation data can be put together by :ref:`SNMP walking donor agent <snmprec.py>`,
:ref:`wire-tapping SNMP traffic <pcap2dev.py>` or :ref:`generating from MIBs <mib2dev.py>`.

Regular output from :ref:`Net-SNMP snmpwalk <snmpwalk-dump>` or :ref:`Agent Pro <agent-pro-dump>`
tools can be used as-is. Finally, *.snmprec* files can be created by hand in a text editor or
automatically generated by some in-house script based on whatever data you got.

Walking SNMP agent
------------------

One way to obtain simulation data is to snapshot some existing SNMP agent
by running the *snmprec.py* tool against your prototype SNMP device. This tool
will execute a series of SNMP GETNEXT queries for a specified range of
OIDs over a chosen SNMP protocol version and store response data in
a .snmprec file.

.. _snmprec.py:

The donor SNMP agent recording session would look like this:

.. code-block:: bash

    $ snmprec.py --agent-udpv4-endpoint=192.168.1.1 --start-oid=1.3.6.1.2.1 \
      --stop-oid=1.3.6.1.2.1.5 --output-file=snmpsim/data/recorded/linksys- \
      system.snmprec
    Scanning "/usr/local/share/snmpsim/variation" directory for variation
    modules...  none requested
    SNMP version 2c
    Community name: public
    Querying UDP/IPv4 agent at 192.168.1.1:161
    Sending initial GETNEXT request....
    OIDs dumped: 304, elapsed: 1.94 sec, rate: 157.00 OIDs/sec
    $
    $ ls -l data/recorded/linksys-system.snmprec
    -rw-r--r-- 1 root users 16252 Oct 26 14:49 data/recorded/linksys-system.snmprec
    $
    $ head data/recorded/linksys-system.snmprec
    1.3.6.1.2.1.1.1.0|4|BEFSX41
    1.3.6.1.2.1.1.2.0|6|1.3.6.1.4.1.3955.1.1
    1.3.6.1.2.1.1.3.0|67|638239
    1.3.6.1.2.1.1.4.0|4|Linksys
    1.3.6.1.2.1.1.5.0|4|isp-gw
    1.3.6.1.2.1.1.6.0|4|4, Petersburger strasse, Berlin, Germany
    1.3.6.1.2.1.1.8.0|67|4

The *snmprec.py* tool can run over SNMPv1, v2c and v3. To configure SNMPv3 USM user
authentication and privacy algorithms, :ref:`follow the instructions <auth-algos>`
for *snmpsimd.py*.

No special requirements exist for device file name and location. Just keep in
mind that at simulation time, *snmpsimd.py* treats *.snmprec* file path as SNMPv1/v2c
community name or SNMPv3 context name.

If you don't readily have some SNMP agent to play with, you're welcome to
use our publicly available :ref:`SNMP Simulator instance <snmp-simulation-service>`.

.. code-block:: bash

    $ snmprec.py --agent-udpv4-endpoint=demo.snmplabs.com --community=public
    SNMP version 2c, Community name: public
    Querying UDP/IPv4 agent at 195.218.195.228:161
    Sending initial GETNEXT request....
    1.3.6.1.2.1.1.1.0|4|SunOS zeus.snmplabs.com 4.1.3_U1 1 sun4m
    1.3.6.1.2.1.1.2.0|6|1.3.6.1.4.1.20408
    1.3.6.1.2.1.1.3.0|67|137765775
    1.3.6.1.2.1.1.4.0|4|SNMP Laboratories, info@snmplabs.com
    1.3.6.1.2.1.1.5.0|4|zeus.snmplabs.com
    1.3.6.1.2.1.1.6.0|4|San Francisco, California, United States
    ...
    1.3.6.1.2.1.11.31.0|65|0
    1.3.6.1.2.1.11.32.0|65|0
    OIDs dumped: 86, elapsed: 2.00 sec, rate: 42.00 OIDs/sec

.. note::

    For better performance, consider using *GETBULK* SNMP command by passing *--use-getbulk*
    option to the *snmprec.py* tool.
    Faster recording may deliver more consistent SNMP objects state.

Since *.snmprec* is a plain text file, you can always edit it in your text editor. For mass changes
consider using the :ref:`datafile.py` tool.

.. _mib2dev.py:

MIB-based synthesis
-------------------

The other way to produce simulation data is to run the *mib2dev.py* tool against
virtually any MIB file. With that method you do not have to have a donor
device and the values, that would otherwise be reported by the donor SNMP
agent, will instead be chosen randomly.

Keep in mind that you may run into either of two issues with these randomly
chosen values:

* Some MIB data suggest certain correlation between formally unrelated
  pieces of information. Such relationships may be described informally,
  e.g. in natural language in the Description field. The automated
  values generation procedure has no chance to assure proper correlations,
  in that case the overall snapshot may appear inconsistent.

* Some data types specified in the MIB may impose certain restrictions on
  the type instance values. For example an integer-typed Managed Object
  may be allowed to be either 0 or 12. If a guessed value turns out to be 2,
  it will be incompatible with this type. While it is possible to introspect
  type objects and generate a compliant value, the *mib2dev.py* tool does
  not do that [yet]. A non-compliant value will result an exception on
  MIB node instantiation. In that case the *mib2dev.py* tool will revert
  to an interactive mode and ask you for a compliant value.

* When building snapshots from MIBs you are not simulating the actual values
  the SNMP agent is reporting. With MIB-based simulation you can basically
  simulate the collection of OIDs, not the dependencies between them or
  their interplay.

On the bright side, the *mib2dev.py* tool will respect Managed Object type
(e.g type associated with the OIDs), and produce valid indices for the MIB
tables.

Examples
++++++++

Here we produce simulation data for a portion of OID space of SNMPv2-MIB:

.. code-block:: bash

    $ mib2dev.py --mib-module=SNMPv2-MIB --start-oid=1.3.6.1.2.1.1.1 \
      --stop-oid=1.3.6.1.2.1.1.8
    # MIB module: SNMPv2-MIB
    1.3.6.1.2.1.1.1.0|4|Portez ce vieux
    1.3.6.1.2.1.1.2.0|6|1.3.6.1.3.39.232.14.10.84.109.1
    1.3.6.1.2.1.1.3.0|67|350728093
    1.3.6.1.2.1.1.4.0|4|whisky
    1.3.6.1.2.1.1.5.0|4|
    1.3.6.1.2.1.1.6.0|4|whisky au juge blond
    1.3.6.1.2.1.1.7.0|2|4
    1.3.6.1.2.1.1.8.0|67|3138976393
    # End of SNMPv2-MIB, 8 OID(s) dumped

The *mib2dev.py* tool can also generate values for SNMP conceptual tables.
It's doing that by iterating over table definition in MIB for specified
number of times.

The following command will analyze given MIB and produce two rows for
the *IF-MIB::ifTable* table:

.. code-block:: bash

    $ mib2dev.py --mib-module=IF-MIB --start-oid=1.3.6.1.2.1.2.2 \
      --stop-oid=1.3.6.1.2.1.2.3 --table-size=2
    # MIB module: IF-MIB
    # Starting table IF-MIB::ifTable (1.3.6.1.2.1.2.2)
    # Synthesizing row #1 of table 1.3.6.1.2.1.2.2.1
    # Finished table 1.3.6.1.2.1.2.2.1 (2 rows)
    1.3.6.1.2.1.2.2.1.1.12|2|12
    1.3.6.1.2.1.2.2.1.1.26|2|26
    1.3.6.1.2.1.2.2.1.2.12|4|vieux whisky
    1.3.6.1.2.1.2.2.1.2.26|4|ce vieux whisky au juge
    1.3.6.1.2.1.2.2.1.3.12|2|29
    1.3.6.1.2.1.2.2.1.3.26|2|1
    1.3.6.1.2.1.2.2.1.4.12|2|28
    1.3.6.1.2.1.2.2.1.4.26|2|16
    1.3.6.1.2.1.2.2.1.5.12|66|3029607807
    1.3.6.1.2.1.2.2.1.5.26|66|3150811331
    1.3.6.1.2.1.2.2.1.6.12|4|
    1.3.6.1.2.1.2.2.1.6.26|4|
    1.3.6.1.2.1.2.2.1.7.12|2|1
    1.3.6.1.2.1.2.2.1.7.26|2|1
    1.3.6.1.2.1.2.2.1.8.12|2|6
    1.3.6.1.2.1.2.2.1.8.26|2|5
    1.3.6.1.2.1.2.2.1.9.12|67|2871454194
    1.3.6.1.2.1.2.2.1.9.26|67|496156868
    1.3.6.1.2.1.2.2.1.10.12|65|1488410552
    1.3.6.1.2.1.2.2.1.10.26|65|3473823260
    1.3.6.1.2.1.2.2.1.11.12|65|1727276906
    1.3.6.1.2.1.2.2.1.11.26|65|342963679
    1.3.6.1.2.1.2.2.1.12.12|65|1511248359
    1.3.6.1.2.1.2.2.1.12.26|65|2207653511
    1.3.6.1.2.1.2.2.1.13.12|65|4226165132
    1.3.6.1.2.1.2.2.1.13.26|65|36536957
    1.3.6.1.2.1.2.2.1.14.12|65|130591184
    1.3.6.1.2.1.2.2.1.14.26|65|1852726355
    1.3.6.1.2.1.2.2.1.15.12|65|3301920138
    1.3.6.1.2.1.2.2.1.15.26|65|470729731
    1.3.6.1.2.1.2.2.1.16.12|65|4148984503
    1.3.6.1.2.1.2.2.1.16.26|65|953020685
    1.3.6.1.2.1.2.2.1.17.12|65|1569764479
    1.3.6.1.2.1.2.2.1.17.26|65|2095562772
    1.3.6.1.2.1.2.2.1.18.12|65|238446444
    1.3.6.1.2.1.2.2.1.18.26|65|3268308217
    1.3.6.1.2.1.2.2.1.19.12|65|3230500934
    1.3.6.1.2.1.2.2.1.19.26|65|566234076
    1.3.6.1.2.1.2.2.1.20.12|65|3549197996
    1.3.6.1.2.1.2.2.1.20.26|65|2834484035
    1.3.6.1.2.1.2.2.1.21.12|66|68812076
    1.3.6.1.2.1.2.2.1.21.26|66|1903146216
    1.3.6.1.2.1.2.2.1.22.12|6|1.3.6.1.3
    1.3.6.1.2.1.2.2.1.22.26|6|1.3.6.1.3.231.101.247.88
    # End of IF-MIB, 44 OID(s) dumped

The range of values for automatic and random selection can be controlled
on a per-type basis with the *--counter-range*, *--counter64-range*,
*--gauge-range*, *--timeticks-range*, *--unsigned-range*,
*--integer32-range* options. Words for strings generations can be passed
via *--string-pool* option.

.. code-block:: bash

    $ mib2dev.py --mib-module=UDP-MIB --table-size=1 --counter-range=0,100 \
        --unsigned-range=100,200
    # MIB module: UDP-MIB
    # Starting table UDP-MIB::udpTable (1.3.6.1.2.1.7.5)
    # Finished table 1.3.6.1.2.1.7.5.1 (1 rows)
    # Starting table UDP-MIB::udpEndpointTable (1.3.6.1.2.1.7.7)
    # Finished table 1.3.6.1.2.1.7.7.1 (1 rows)
    1.3.6.1.2.1.7.1.0|65|66
    1.3.6.1.2.1.7.2.0|65|49
    1.3.6.1.2.1.7.3.0|65|91
    1.3.6.1.2.1.7.4.0|65|14
    1.3.6.1.2.1.7.5.1.1.169.148.104.225.14|64x|a99468e1
    1.3.6.1.2.1.7.5.1.2.169.148.104.225.14|2|14
    1.3.6.1.2.1.7.7.1.1.4.0.127.2.0.137.182|2|4
    1.3.6.1.2.1.7.7.1.2.4.0.127.2.0.137.182|4|
    1.3.6.1.2.1.7.7.1.3.4.0.127.2.0.137.182|66|127
    1.3.6.1.2.1.7.7.1.4.4.0.127.2.0.137.182|2|2
    1.3.6.1.2.1.7.7.1.5.4.0.127.2.0.137.182|4|
    1.3.6.1.2.1.7.7.1.6.4.0.127.2.0.137.182|66|137
    1.3.6.1.2.1.7.7.1.7.4.0.127.2.0.137.182|66|182
    1.3.6.1.2.1.7.7.1.8.4.0.127.2.0.137.182|66|185
    1.3.6.1.2.1.7.8.0|70|9808059939656837207
    1.3.6.1.2.1.7.9.0|70|10931009272993024622
    # End of UDP-MIB, 16 OID(s) dumped

If you wish to specify each value rather then rely on automatic random
selection, use *--manual-value* command line switch. If you would rather
have *mib2dev.py* tool to work out all the values by itself, consider
raising the *--automatic-values* max probes value (default is 5000 probes).

.. _pcap2dev.py:

Snooping SNMP traffic
---------------------

SNMP traffic traveling in a network can also be a source of simulation data.
The *pcap2dev.py* tool can snoop live or process captured traffic
finding SNMP Response messages there and using OID-value pairs for building
*.snmprec* files.

Since many SNMP agents can generate traffic over network within the a snooping
sessions, the *pcap2dev.py* tool is designed to classify captured SNMP traffic
on the per-Agent basis and build dedicated data file for each Agent seen on
the network.

The *--output-dir=<directory>* command-line option specifies a directory
where *pcap2dev.py* tool would put generated data files into. Data files paths
are crafted so that Simulator would act closer to the prototype Agents
meaning:

1. Data files for each Agent is put under a separate directory
   resembling Simulator's transport IDs which correspond to
   UDP ports Simulator is listening on.

2. Original SNMPv1/v2c community names are preserved.

Imagine we have two SNMP Agents (192.168.1.1 & 192.168.1.2) sending
responses over a network we are snooping on. Here's a tcpdump report just
to illustrate the idea:

.. code-block:: bash

    # tcpdump -i lo
    listening on lo, link-type EN10MB (Ethernet), capture size 65535 bytes
    20:05:20.799706 IP 192.168.1.9.55803 > 192.168.1.1.snmp:  GetRequest(28) system .sysDescr.0
    20:05:20.800027 IP 192.168.1.1.snmp > 192.168.1.9.55803:  GetResponse(92) system.sysDescr.0="Linux jupiter 2.6.37.6-smp #2 SMP Fri May 17 22:03:50 CDT 2013 i686"
    20:05:21.125421 IP 192.168.1.9.55803 > 192.168.1.2.snmp:  GetRequest(28) system.sysDescr.0
    20:05:21.924022 IP 192.168.1.2.snmp > 192.168.1.9.55803:  GetResponse(92) system.sysDescr.0="Linux saturn 2.6.37.4-smp #2 SMP Fri May 10 21:31:32 CDT 2013 i686"

The *pcap2dev* tool would create two directories with fixed prefix
(1.3.6.1.6.1.1) and increasing suffix parts (0 & 1) to put generated data files
for each Agent there. That is, all data files for Agent 192.168.1.1 would
go under 1.3.6.1.6.1.1.0/ while data files for Agent 192.168.1.2 would end
up in 1.3.6.1.6.1.1.1/.

Snooped SNMP communities also take part in data file path creation -- they
appear as a last component of the path. For example, if Agent 192.168.1.1
used SNMP communities 'wallace' and 'gromit' (on different occasions) and
Agent 192.168.1.2 responded with community 'cheese', generated data files
would look like this:

.. code-block:: bash

    $ tree /tmp/recording
    /tmp/recording
    |--- 1.3.6.1.6.1.1.0
    |    |
    |     ---- gromit.snmprec
    |    |
    |     ---- wallace.snmprec
    |
    |--- 1.3.6.1.6.1.1.1
         |
          ---- cheese.snmprec

To build data files from a network capture file, use *--capture-file=<file>*
command-line option. Capture file format should be either
`pcap or pcap-ng <http://en.wikipedia.org/wiki/Pcap>`_.
Most capturing tools (like `tcpdump <http://www.tcpdump.org>`_)
support these file formats.

You could also use
`tcpdump filter <http://www.cs.ucr.edu/~marios/ethereal-tcpdump.pdf>`_
as a parameter to *--packet-filter=<ruleset>* option to narrow packets selection
criteria. Default packet filter is *udp and src port 161*.

Examples
++++++++

With all that theory in mind, we can now run a live snooping session:

.. code-block:: bash

    # pcap2dev.py --output-dir=/tmp/recording --listen-interface=lo
    Listening on interface lo in non-promiscuous mode
    Applying packet filter "udp and src port 161"
    Listening on interface "lo", kill me when you are done.
    ^C
    Shutting down process...
    Creating simulation context 1.3.6.1.6.1.1.0/gromit
     at /tmp/recording/1.3.6.1.6.1.1.0/gromit.snmprec
    Creating simulation context 1.3.6.1.6.1.1.0/wallace
     at /tmp/recording/1.3.6.1.6.1.1.0/wallace.snmprec
    Creating simulation context 1.3.6.1.6.1.1.1/cheese
     at /tmp/recording/1.3.6.1.6.1.1.1/cheese.snmprec
    PCap statistics:
        packets snooped: 64
        packets dropped: 24
        packets dropped: by interface 0
    SNMP statistics:
        empty packets: 0
        OIDs seen: 19
        UDP packets: 19
        Response PDUs seen: 19
        contexts seen: 3
        SNMP exceptions: 0
        SNMP errors: 0
        snapshots taken: 0
        agents seen: 2
        unknown L2 protocol: 0
        IP packets: 19
        bad packets: 0

Here's one of data files produced:

.. code-block:: bash

    $ cat /tmp/recording/1.3.6.1.6.1.1.0/gromit.snmprec
    1.3.6.1.2.1.1.1.0|4|Linux jupiter 2.6.37.6-smp #2 SMP Fri May 17 22:03:50 CDT 2013 i686
    1.3.6.1.2.1.1.2.0|6|1.3.6.1.4.1.8072.3.2.10
    1.3.6.1.2.1.1.3.0|67|311441639
    1.3.6.1.2.1.1.4.0|4|postmaster@jupiter
    1.3.6.1.2.1.1.5.0|4|jupiter
    1.3.6.1.2.1.1.6.0|4|Jupiter
    1.3.6.1.2.1.1.8.0|67|1

You can now move data files into your Simulator's data directory and
fire up simulation.

The *pcap2dev.py* tool can also invoke
:ref:`variation modules <recording-with-variation-modules>` to feed recorded
data through them.

.. _snmpwalk-dump:

Using snmpwalk reporting
------------------------

In some cases you may not be able to run :ref:`snmprec.py <snmprec.py>`
against a donor device. That can happen, for instance, if you can't setup
*snmprec.py* on a system from where donor device is available or donor device
is gone leaving you with just Net-SNMP's *snmpwalk* dumps someone have collected
for you.

Simulator provides limited support for snmpwalk-generated data files.
Just save *snmpwalk* output into a file with *.snmpwalk* suffix and put
it under the *--data-dir*. Once Simulator finds and indexes
the *.snmpwalk* files, it will report them just as it does for its
native *.snmprec* files.

.. code-block:: bash

    $ snmpwalk -v2c -c public -ObentU localhost 1.3.6 > myagent.snmpwalk

.. note::

    Make sure you get *snmpwalk* producing plain OIDs and values!

By default snmpwalk tries to beautify raw data from Agent with MIB
information. As beautified data may not contain OIDs and numeric values,
it could not be interpreted by the Simulator. Therefore always run
*snmpwalk* with the "-ObentU" options.

The *.snmpwalk* lines that can't be parsed by the Simulator will be skipped
and details reported to stdout for your further consideration. In particular,
current implementation does not cope well with multi-line strings
sometimes produced by the *snmpwalk* tool.

Alternatively, you can convert the *.snmpwalk* files into *.snmprec* ones
by running them through the :ref:`datafile.py <datafile.py>` tool.

.. _agent-pro-dump:

Using Simple Agent Pro samples
------------------------------

Another possible format for taking and storing SNMP snapshots is
SimpleSoft `Simple Agent Pro <http://www.smplsft.com/SimpleAgentPro.html>`_
data files. Although we have neither seen any documentation on its data files
format nor ever owned or used Simple Agent Pro software, a sample data file
`published on the Internet <http://tech.chickenandporn.com/2011/05/26/snmp-ping/>`_
reveals that SimpleAgentPro's file format is very similar to Net-SNMP's
snmpwalk. It essentially looks like *snmpwalk* output with different field
separators.

.. note::

    SNMP Simulator might not support certain features/dialects of
    SimpleAgentPro data files format so your mileage may vary.

In case you store your SNMP snapshots archives in SimpleAgentPro's
data files and wish to use them with this Simulator, just put your
SimpleAgentPro-formatted SNMP snapshot information (excluding comments)
into text files having *.sapwalk* suffix and let Simulator find and index
them. Once completed, Simulator will report access information for them
just as it does for its native *.snmprec* files.

Alternatively, you can convert the *.sapwalk* files into *.snmprec* ones
by running them through the :ref:`datafile.py <datafile.py>` tool.
