---
title: "Linux Performance Analysis with perf"
description: "Comprehensive guide to using the perf profiling tool for in-depth Linux performance analysis"
keywords: ["linux", "performance", "profiling", "perf", "tracing"]
category: "linux"
---

The perf tool is a powerful Linux profiling utility that provides deep insights into system performance by leveraging hardware performance counters. perf stat allows you to gather high-level statistics on CPU utilization, cache misses, branch mispredictions, and other hardware-level metrics. For example, running `perf stat -a -e cycles,instructions,cache-references,cache-misses,branches,branch-misses -- sleep 10` will report key hardware counter values over a 10-second interval.

Going deeper, perf record can sample the program counter at a high rate to capture a statistical profile of where an application is spending its time. This sampling data can then be visualized using flame graphs, which provide an intuitive top-down view of hotspots in the call stack. The `perf record -F 99 -a -- sleep 30` command will capture a 30-second profile, which can be converted to a flame graph using tools like FlameGraph.

To understand performance issues at the instruction level, perf annotate can disassemble the binary and overlay the sampling data, highlighting which individual instructions are consuming the most cycles. This is valuable for optimizing low-level code. The `perf report --stdio --sort=sym,dso,srcline` command will generate an annotated disassembly.

For dynamic tracing of specific kernel or user-space functions, perf probe allows you to instrument the running system and capture detailed performance metrics. For example, `perf probe 'do_sys_open'` will add a probe to the kernel's `do_sys_open` function, and `perf record -e probe:do_sys_open -- ls` will record activity for that function.

Contention for locks can be a major source of performance degradation, and perf lock can help identify these bottlenecks. The `perf lock record -- sleep 30` command will capture lock events, which can then be analyzed using `perf lock report` to see the most heavily contended locks.

Off-CPU analysis is crucial for understanding time spent waiting, such as in I/O operations or kernel work. perf sched record can capture scheduling events, and the resulting data can be visualized using tools like Kernelshark to identify time spent off the CPU.

Finally, perf c2c can provide detailed analysis of cache behavior, including false sharing, remote accesses, and NUMA effects. The `perf c2c record -- sleep 30` command will capture cache-related events, which can then be inspected using `perf c2c report`.Continuing the comprehensive guide on Linux performance analysis with perf, we'll dive deeper into some additional advanced capabilities of this powerful profiling tool.

One particularly valuable feature of perf is its ability to capture hardware-level metrics beyond just CPU utilization. The `perf stat` command can be used to gather a wide range of hardware counter data, including cycle counts, instructions retired, cache references and misses, branch predictions and mispredictions, and much more. These low-level hardware statistics provide deep insight into microarchitectural behavior and can be instrumental in identifying performance bottlenecks. For example, the command `perf stat -e cycles,instructions,cache-references,cache-misses,branches,branch-misses -a -- ./my_program` will report these key metrics for the duration of the `my_program` execution.

Going a step further, perf's ability to capture call stack samples through `perf record` enables the generation of flame graphs - powerful visualization tools that provide an intuitive top-down view of performance hotspots. The `perf record -F 99 -a -- sleep 30` command will capture a 30-second profile, which can then be converted to a flame graph using the FlameGraph toolset. These flame graphs make it easy to quickly identify the most time-consuming functions and code paths, guiding optimization efforts.

Another advanced perf feature is the ability to perform instruction-level analysis using `perf annotate`. This command can disassemble the target binary and overlay the sampling data, highlighting which individual instructions are consuming the most cycles. This level of granularity is invaluable for optimizing low-level code, especially in performance-critical sections. The `perf report --stdio --sort=sym,dso,srcline` command will generate the annotated disassembly.

Perf also provides dynamic tracing capabilities through the `perf probe` subcommand, allowing you to instrument specific kernel or user-space functions and capture detailed performance metrics. For example, the command `perf probe 'do_sys_open'` will add a probe to the kernel's `do_sys_open` function, and `perf record -e probe:do_sys_open -- ls` will record activity for that function. This flexibility enables targeted profiling of specific system events or application behaviors.

Beyond CPU-centric analysis, perf also excels at identifying contention issues, such as heavily contended locks. The `perf lock record -- sleep 30` command can capture lock events, which can then be analyzed using `perf lock report` to pinpoint the most problematic locks causing performance degradation. Understanding lock contention is crucial for building highly concurrent, scalable applications.

Lastly, perf provides the ability to perform off-CPU analysis, which is essential for understanding time spent waiting, such as in I/O operations or kernel work. The `perf sched record` command can capture scheduling events, and the resulting data can be visualized using tools like Kernelshark to identify periods where the target process was not actively running on the CPU.Continuing our deep dive into Linux performance analysis with the perf tool, we'll explore its advanced cache analysis capabilities.

Perf's `c2c` subcommand provides detailed insight into cache behavior, including false sharing, remote accesses, and NUMA effects. The `perf c2c record -- sleep 30` command will capture a wide range of cache-related events over a 30-second interval, which can then be inspected using the `perf c2c report` command. This report will shed light on cache misses, cache line bounces, and other microarchitectural factors that can have a significant impact on performance. Understanding these low-level cache dynamics is crucial for optimizing memory-intensive workloads, as cache behavior can be a major bottleneck.

Another powerful feature of perf is its ability to perform function-level tracing using the `probe` subcommand. This allows you to dynamically instrument specific kernel or user-space functions and capture detailed performance metrics. For example, the command `perf probe 'do_sys_open'` will add a probe to the kernel's `do_sys_open` function, and `perf record -e probe:do_sys_open -- ls` will record activity for that function. This flexibility enables targeted profiling of specific system events or application behaviors, providing valuable insights that may be obscured in broader profiling data.

Perf also excels at identifying contention issues, such as heavily contended locks, which can be a major source of performance degradation. The `perf lock record -- sleep 30` command can capture lock events, which can then be analyzed using `perf lock report` to pinpoint the most problematic locks causing bottlenecks. Understanding lock contention is crucial for building highly concurrent, scalable applications, as unoptimized locking can severely limit performance.

Beyond CPU-centric analysis, perf provides the ability to perform off-CPU analysis, which is essential for understanding time spent waiting, such as in I/O operations or kernel work. The `perf sched record` command can capture scheduling events, and the resulting data can be visualized using tools like Kernelshark to identify periods where the target process was not actively running on the CPU. This off-CPU profiling can reveal important performance insights that may be overlooked by traditional CPU-bound analysis.

Perf's flexibility and depth of features make it an indispensable tool in the Linux performance analyst's arsenal. By leveraging hardware performance counters, sampling-based profiling, dynamic tracing, and advanced analysis capabilities, perf can provide unparalleled visibility into the inner workings of your system, enabling you to identify and resolve even the most elusive performance bottlenecks.Continuing our exploration of Linux performance analysis with the perf tool, we'll delve into its advanced cache analysis features, which can provide critical insights into memory subsystem behavior.

Perf's `c2c` (cache-to-cache) subcommand offers a deep dive into cache-related performance metrics, including false sharing, remote accesses, and NUMA effects. By running the command `perf c2c record -- sleep 30`, you can capture a wealth of cache-related events over a 30-second interval, which can then be analyzed using the `perf c2c report` command. This report will shed light on cache misses, cache line bounces, and other microarchitectural factors that can significantly impact performance, especially for memory-intensive workloads. Understanding these low-level cache dynamics is essential for optimizing applications that rely heavily on efficient memory access patterns.

Another powerful feature of perf is its ability to perform dynamic tracing of specific kernel or user-space functions using the `probe` subcommand. This allows you to instrument target functions and capture detailed performance metrics in a highly targeted manner. For example, the command `perf probe 'do_sys_open'` will add a probe to the kernel's `do_sys_open` function, and `perf record -e probe:do_sys_open -- ls` will record activity for that function. This flexibility enables you to focus your profiling efforts on the specific system events or application behaviors that are of interest, providing valuable insights that may be obscured in broader profiling data.

Perf also excels at identifying contention issues, such as heavily contended locks, which can be a major source of performance degradation. By running the `perf lock record -- sleep 30` command, you can capture lock events, which can then be analyzed using `perf lock report` to pinpoint the most problematic locks causing bottlenecks. Understanding lock contention is crucial for building highly concurrent, scalable applications, as unoptimized locking can severely limit performance and scalability.

Beyond CPU-centric analysis, perf provides the ability to perform off-CPU analysis, which is essential for understanding time spent waiting, such as in I/O operations or kernel work. The `perf sched record` command can capture scheduling events, and the resulting data can be visualized using tools like Kernelshark to identify periods where the target process was not actively running on the CPU. This off-CPU profiling can reveal important performance insights that may be overlooked by traditional CPU-bound analysis, helping you gain a more comprehensive understanding of your system's behavior.

The depth and flexibility of perf's features make it an indispensable tool in the Linux performance analyst's toolkit. By leveraging hardware performance counters, sampling-based profiling, dynamic tracing, and advanced analysis capabilities, perf can provide unparalleled visibility into the inner workings of your system, enabling you to identify and resolve even the most elusive performance bottlenecks.