# ucevent uncore monitoring

The largest part of modern CPUs is outside the actual cores.
On Intel CPUs this is part is called the "Uncore" and and has last
level caches, PCI-Express, memory controller, QPI, power management
and other functionalities. To understand its performance the uncore
also provides a range of performance monitoring units (PMU) with
performance counters that can count various events.

This can be useful to monitor power management, IO bandwidth, memory
bandwidth, QPI (interconnect) traffic, cache hit rates and other metrics. 
The Linux kernel as part of perf has an uncore driver for Xeon
systems since Linux 3.8.  But since the uncore is complex, and the perf tool's
view of them is quite raw, they have been hard to use directly.

ucevent is a tool that provides a generic uncore event list, standard
equations and an (as friendly as possible) frontend to the uncore
metrics. It runs as a wrapper around perf stat. The output is similar
to turbostat.

In general the uncore can only monitor metrics. It cannot tell exactly
where in a program something occurs (it is not a profiler). It can, however,
report trends over time.

# Supported systems and configurations

ucevent currently only supports Intel Xeon E5 2600 series (codename Sandy
Bridge EP), Intel Xeon E5/E7 v2 series (codename Ivy Bridge EP),
Intel Xeon E5/E7 v3 Series (codename Haswell EP) and various
derivatives.

For the QPI and memory controller metrics the BIOS also needs to
expose the uncore monitoring PCI devices (this is often disabled by default
and, depending on the system, may need a BIOS update to enable).

When the BIOS does not support these devices only a subset (mostly
power management) of the uncore monitoring is available. Enabling
them may also require setting a BIOS option.

You can check in lspci if the monitoring devices are available. It
should show output similar to this:

	# lspci | grep Performance
	ff:13.1 Performance counters: Intel Corporation Sandy Bridge Ring to PCI Express Performance Monitor (rev 07)
	ff:13.4 Performance counters: Intel Corporation Sandy Bridge QuickPath Interconnect Agent Ring Registers (rev 07)
	ff:13.5 Performance counters: Intel Corporation Sandy Bridge Ring to QuickPath Interconnect Link 0 Performance Monitor (rev 07)
	ff:13.6 System peripheral: Intel Corporation Sandy Bridge Ring to QuickPath Interconnect Link 1 Performance Monitor (rev 07)

The uncore driver is active if a number of uncore files show up
in /sys/bus/event_source/devices/

	# ls -d /sys/bus/event_source/devices/uncore_*

The monitoring works best with a 3.10+ kernel with the patches in
patches-3.10 applied. Later kernels include some/all of
them.  The perf tool should be installed in a new enough version.
Without the right patches there will be various problems.

For Haswell EP support a 3.16+ kernel with the patches in
patches-3.16 applied are needed. Future kernels should also
support this by default.

# Tutorial / Examples

The uncore does not profile individual programs, it profiles
everything going on on a socket with cores (each of which may execute
a different program).  Normally the monitoring is done in the
background parallel to some workload.  By default ucevent prints a
sample every second and runs until stopped with Ctrl-C.

It generates the perf command line to collect the data, runs perf
and then does data post processing (computing equations) and finally
outputs the data.

First set up pmu-tools if you haven't already

	% git clone https://github.com/andikleen/pmu-tools
	% cd pmu-tools

The rest of the tutorial needs to run as root, as the uncore 
is a global resource:

	# cd pmu-tools/ucevent
	# export PATH=$PATH:$(pwd)

"sleep XXX" can be used to define the number of seconds to measure:
Measure PCI bandwidth for 10 seconds, print measurement every two seconds:

	# ucevent.py -I 2000  CBO.PCIE_DATA_BYTES sleep 10
	S0-CBO.PCIE_DATA_BYTES
	|      S1-CBO.PCIE_DATA_BYTES
	384.00 256.00 
	0.00   256.00 
	0.00   0.00   
	0.00   0.00   

The data is output separately for each CPU socket.

As you can see this is a fairly idle system. If we generate some PCI-E 
traffic (for example an ping -f -s1400 to a neighboring system), it looks more like

	S0-CBO.PCIE_DATA_BYTES
	|             S1-CBO.PCIE_DATA_BYTES
	15,369,344.00 0.00  
	15,587,904.00 0.00  
	15,577,408.00 0.00  
	14,942,336.00 0.00  

As you can see the ping was on socket 0, while the PCI-E on socket 1
is still idle.  ping -f is a fairly drastic way to generate load: make
sure the owner of the other system and the network concur.  There is
currently no way to limit the data to specific PCI devices.

Alternatively a program can be also run from ucevent and measure while
the program runs.  However again it is important to keep it mind it
will profile the whole sockets specified unlike normal perf stat.

This example shows the number of DIMM page misses in the DDR memory controller.
This can be a fairly important metric, as it affects memory latency significantly.

	# ucevent.py iMC.PCT_REQUESTS_PAGE_MISS my-workload

Only specific sockets can be measured. In this case it can help
to bind a workload to that socket. Note that a socket in ucevent
is not always the same as a node, but it often is.

	# ucevent.py -v --socket 0 iMC.PCT_REQUESTS_PAGE_HIT numactl --cpunodebind=0 workload
	Expression 1 - (PCT_REQUESTS_PAGE_EMPTY + PCT_REQUESTS_PAGE_MISS)
	Events: iMC.PCT_REQUESTS_PAGE_HIT
	perf stat -e '{uncore_imc_0/event=0x1/, ... lots of events ...  }' -I1000 -x, -C0 ...
	iMC0.PCT_REQUESTS_PAGE_HIT
	|      iMC1.PCT_REQUESTS_PAGE_HIT
	|      |      iMC2.PCT_REQUESTS_PAGE_HIT
	|      |      |      iMC3.PCT_REQUESTS_PAGE_HIT
	9.09   49.33  10.71  8.60   

Here we measure the percentage of DIMM page hits.
This also specifies verbose mode, so you can see the generated perf command line
and some more information.

List derived (higher level) available metrics that can be measured
(this is a subset of the full list)

	# ucevent.py
	...
	CBO CACHE Events
	  CBO.LLC_DRD_MISS_PCT           LLC DRD Miss Ratio                      
	  CBO.MEM_WB_BYTES               Memory Writebacks
        ...

List all PCU (Power Management Control Unit) events with description

	# ucevent.py --desc --unsupported --cat PCU

Display the memory read and write bandwidth

	# ucevent.py  --scale GB iMC.MEM_BW_WRITES iMC.MEM_BW_READS

"--scale GB" scales the metric to GB/s. The values are printed every second
(can be changed with -I)

Display the percentage of time the uncore was running higher than
2.0Ghz and higher than 3.0Ghz.  The filter uses a multiplier of
100. Make sure to use the right filter for the right band.  Upto 4
bands are possible. 

	# ucevent.py PCU.PCT_FREQ_BAND0,filter_band0=20 PCU.PCT_FREQ_BAND1,filter_band1=30

The parameter after the comma is a qualifier. --desc will output the
possible qualifiers / filters for an event.  Some qualifier combinations may 
be non-sensical, the documentation will usually list the useful ones.

List the percentage of time the memory controllers spent in
self-refresh mode. Output the data in CSV mode to a file without any
extra output.

	# ucevent.py -o out.csv -x, iMC.PCT_CYCLES_SELF_REFRESH

Plot the data using R

	% R
	...
	> x <- read.csv("out.csv")
	> summary(x)
	> plot(x$iMC.PCT_CYCLES_SELF_REFRESH, type="o")

Plot the data in a web browser using [dygraphs](http://dygraphs.com/)

	% mkdir web
	% cp dygraph-out.html web/index.html
	% cp out.csv web
	% cd web
 	% python -m SimpleHTTPServer
	... point web browser at localhost:8000 ...

Note that dygraphs only works when the page is served through a HTTP
server, not directly through the file system. Also some firewall
settings may prevent serve, in this case copy the web directory to a
HTTP server directory to view it there.

List how much of the time each socket's frequency is thermally limited

	# ucevent.py PCU.PCT_FREQ_THERMAL_LTD

Using ucevent as non root (insecure). Run as root once:

	# sysctl -w kernel.perf_event_paranoid=-1

# Command Line options reference

	usage: ucevent.py [-h] [--cat CAT] [--unsupported] [--desc] [--name-only] [--equation]
	                  [--attr] [--scale {KB,GB,MB}] [--csv CSV] [--interval INTERVAL]
	                  [--socket SOCKET] [--cpu CPU] [--fieldlen FIELDLEN] [--verbose]
	                  [--no-sum] [--output OUTPUT]
	                  [events [events ...]]
	
	Intel Xeon uncore performance counter events frontend for perf. The uncore is the part
	of the CPU that is not core. This tool allows to monitor a variety of metrics in the
	uncore, including memory, QPI, PCI-E bandwidth, cache hit rates, power management
	statistics and various others.
	
	positional arguments:
	  events                List of events to be converted. May have a comma separate list
	                        of name=val qualifiers after comma for each event (see --desc
	                        output for valid qualifiers). Can use { and } (each as own
	                        argument and quoted) to define groups. Valid to use shell-style
	                        wildcards (quoted) to match multiple events. It is also valid
	                        to specify a equation (quoted, containing space). After -- a
	                        perf argument line can be specified (e.g. sleep NUM or a
	                        workload to run). Please note that ucevent measurements are
	                        always global to the specified socket(s) unlike normal perf
	                        stat.
	
	optional arguments:
	  -h, --help            show this help message and exit
	  --cat CAT             Only print events from categories containing this in list
	  --unsupported         Print all events, including unsupported and potentially broken
	                        ones. Use at your own risk.
	  --desc                Print detailed descriptions in list
	  --name-only           Only print event name in list
	  --equation            Print equations for derived events in list
	  --attr                Print attributes
	  --scale {KB,GB,MB}    Scale numbers to unit (GB,MB,KB)
	  --csv CSV, -x CSV     Enable CSV mode and use argument to separate fields
	  --interval INTERVAL, -I INTERVAL
	                        Measurement interval in ms
	  --socket SOCKET, -S SOCKET
	                        Measure only socket (default all)
	  --cpu CPU, -C CPU     Measure socket associated with CPU, or use that CPU for the
	                        (very few) events that use core events
	  --fieldlen FIELDLEN   Set output field length
	  --verbose, -v         More verbose output
	  --no-sum              Don't sum up multiple instances of units
	  --output OUTPUT, -o OUTPUT
	                        Set output file
# Attributes

Individual events have additional attributes. These can be specified
after the event separated with comma.

	# ucevent PCU.PCT_FREQ_BAND0,filter_band0=20

ucevent --desc will list the available qualifiers for each event. Note
that many qualifier combinations are not useful. The description in
"--desc" usually mentions the useful qualifiers.

Qualifiers can be also used with derived events, but there are some
limitations. The qualifier will always apply to all events used in the
derived event, which can result in some non-working combinations.

The qualifier displayed in "Filter" usually correspond to the filters 
discussed in the description.

# Grouping, event scheduling and measurement inaccuracy

The uncore contains different units (Power Control Unit, QPI, Memory
Controller etc.). These are also sometimes called boxes.  Each unit
can have multiple instances (for example each CPU core has its own CBO
last level cache slice). For more details please see the uncore
manual for the CPU in the references.

Each system can have multiple uncores for each
CPU socket. The individual uncore units have their performance
monitoring counters. Typically they have 4 counters, and some
additional filter registers.  There are also some additional restrictions.

ucevent works by programming different events for each of these
counters. When the output requires more than 4 counters (or runs
into the additional restrictions) for a given unit, or some filter register
is oversubscribed, the perf driver code has to multiplex the measurement.

Perf will then periodically switch to a different set of events during the
measurement period.

Many metrics are actually multiple events in a equation (for example a
ratio to the clock ticks get percent, see --equation output). When the
different terms of a equation are not monitored in the same time slice
this can add large errors. This is a common cause for "impossible results",
like percentages being larger than 100% or negative results.

There is the concept of "groups", that is events that always run
together. A group can be manually specified by using '{' and '}' as
arguments to ucevent.

	# ucevent '{' iMC.MEM_BW_WRITES iMC.MEM_BW_WRITES_TOTAL '}'

Some metrics cannot be run in a group, as they need more counters or
filters thnt are available at the same time by the monitoring units.
In this case putting them in a group will cause "#NC" (not counted) or
"#NS" (not supported) errors.

ucevent by default puts derived metrics and user equations into groups
of their own if they fit and there is no user specified
group. It currently cannot create groups automatically for some units
(primarily CBOX) Different events or equations on the command line can still run
at different time, unless explicit groups are specified. This may
cause visible inaccuracies when different columns are compared.

Also due to perf limitations some valid groups don't run. A common
case of this is multiple events from a different instance of the same
box on the same socket. The scheduler does not know that different 
instances are independent.

**In general avoiding multiplexing will yield the best results**

Some useful metrics require multiplexing though. When the automatic
grouping is not accurate enough the user should specific explicit groups.

Multiplexing works best when the workload is very "steady", and does not fluctuate much.

The multiplexing frequency can be configured in sysfs. For example to switch
to 10ms

	echo 10 > /sys/bus/event_source/devices/uncore_name/perf_event_mux_interval_ms

The default is 4ms. Different multiplexing values may affect the results.

(Kernel before 3.11 or 3.10 with the included patches not applied, did not have
this setting, and had other multiplexing problems)

Another side effect of multiplexing is that it causes wakeups of one
core per socket at the multiplexing frequency.  This can disturb
power measurements.  If that is a problem decrease the
multiplexing frequency or avoid multiplexing.

# Unit summing

When there are multiple instances of a unit in a socket ucevent sums up
the individual instances by default (with some exceptions).
This can be disabled with the --no-sum option. This is primarily for
the cache slices (CBO), of which there is one for each core.

# Equations

Equations can be specified on the command line. They should contain a
(quoted) space or () so that the parser can detect them.

The equations are particularly useful to scale BYTES metrics to more easily readable units.

	ucevent.py "CBO.MEM_WB_BYTES / KB"

Available predefined units are GIGA, MEGA, KILO (10^x) and GB, MB, KB (2^(10x))

Some units have multiple instances, which ucevents automatically expands. When referencing
such multi-instance units in a equation, only events in the same unit can be referenced.
Wildcards are also not allowed in equations.

# Post processing data

The easiest way is to enable the CSV output mode (with -x,) 
and redirect the output to a file (-o file) Other separators can be also specified.
The output is normalized and can be directly processed with gnuplot, spreadsheets, R etc.

# Error display

perf may fail to measure a counter (for example if the group is not schedulable)
or ucevent may fail to compute a value. In this case it shows spreadsheet style
'#' errors. When you see a string of such errors they were summed up from multiple
measurements.

Current errors:

- #NS: Not supported. perf was not able to measure the event.
- #NC: Not counted. perf was not able to schedule the event.
- #NA: Not available. Value missing in perf output.
- #EVAL: ucevent failed to evaluate an equation

Division by zero is silently turned into 0, as that is somewhat common.

# Debugging and testing

ucevent has a variety of test suites. The most useful is the CHECK-ALL script that
runs all events for two seconds and SANITY-RUN which does simple sanity checks on
all the high level derived events. These should be run with some background load.
They require [GNU parallel](http://www.gnu.org/software/parallel/) to be installed.

RUN-ALL will run all tests (some errors are expected). 

ucevent has some undocumented options for debugging. If something goes wrong
--debug '\*' will enable a lot of extra output.

# References

[Xeon E5 2600 series uncore guide](http://www.intel.com/content/dam/www/public/us/en/documents/design-guides/xeon-e5-2600-uncore-guide.pdf)

[Xeon v3 series uncore guide](http://www.intel.com/content/www/us/en/processors/xeon/xeon-e5-v3-uncore-performance-monitoring.html)

# Support

Please contact linux-perf-users@vger.kernel.org

# Other Projects providing uncore monitoring

[Intel PCM](http://software.intel.com/en-us/articles/intel-performance-counter-monitor-a-better-way-to-measure-cpu-utilization) Provides high-level metrics. Runs from user space and provides a C++ library.

[likwid](http://code.google.com/p/likwid/) 

[libpfm4](http://perfmon2.sourceforge.net/) has another program to convert (non derived) uncore events to perf form.

[Vtune](https://software.intel.com/en-us/intel-vtune-amplifier-xe) (not open source) provides GUI and command line interfaces (sep) for uncore metrics. It also supports CPUs that are not supported by ucevent, such as Atom and client systems.

# Author

Andi Kleen 
pmu-tools@halobates.de
