---
title: "TCP Performance Tuning for High-Throughput Servers"
description: "Optimize TCP performance for high-throughput server applications"
keywords: ["tcp", "performance", "tuning", "high-throughput", "servers"]
category: "networking"
---

TCP window scaling is an important performance optimization that enables TCP connections to utilize more than the traditional 64KB send and receive windows. This is critical for high-throughput servers that need to maximize the amount of data in-flight. The TCP window scaling option is enabled by default in Linux kernels 2.4 and later, and can be configured with the net.ipv4.tcp_window_scaling sysctl. Operators should ensure this option is enabled for their high-throughput server workloads.

The Nagle algorithm is a TCP optimization that attempts to reduce the number of small packets on the network by buffering small amounts of output until a full-sized packet (typically 1460 bytes) can be sent. While this can be beneficial for some applications, it can hurt performance for high-throughput servers that need to minimize latency. The TCP_NODELAY socket option can be used to disable the Nagle algorithm for specific connections, allowing small packets to be sent immediately without buffering.

Congestion control algorithms play a crucial role in TCP performance tuning. The default cubic congestion control algorithm in Linux is well-suited for high-bandwidth, high-latency networks, but the newer BBR (Bottleneck Bandwidth and Round-trip propagation time) algorithm can provide even better performance in some scenarios. Operators can switch the congestion control algorithm used by their server with the net.ipv4.tcp_congestion_control sysctl.

Proper socket buffer sizing is essential for maximizing TCP throughput. The net.core.rmem_max and net.core.wmem_max sysctl parameters control the maximum receive and send buffer sizes, respectively. These should be increased to at least 8MB (8388608 bytes) on high-throughput servers to ensure the TCP stack has enough buffer space to accommodate large amounts of in-flight data.

The TIME_WAIT state in the TCP state machine can cause issues for high-throughput servers with large numbers of short-lived connections. The net.ipv4.tcp_tw_reuse and net.ipv4.tcp_tw_recycle sysctl parameters can be used to reuse or recycle these TIME_WAIT sockets, reducing the load on the system. Operators should carefully evaluate the tradeoffs of these options, as they can potentially cause compatibility issues with some clients.

Keepalive configuration is another important consideration for high-throughput servers. The net.ipv4.tcp_keepalive_time, net.ipv4.tcp_keepalive_intvl, and net.ipv4.tcp_keepalive_probes sysctl parameters control the keepalive behavior, allowing operators to detect and close idle connections more quickly. This can help reduce the number of open sockets on the server.

TCP maximum segment size (MSS) negotiation is critical for optimizing network throughput. The tcp_mss_default kernel parameter controls the default MSS value, which should be set to the largest possible value that does not result in IP fragmentation (typically 1460 bytes for Ethernet). Operators should also consider using the TCP_MAXSEG socket option to set the MSS on a per-connection basis.

SYN backlog tuning is important for servers that expect to handle large numbers of concurrent connection attempts. The net.core.somaxconn sysctl parameter controls the maximum length of the listen queue for new connections. This should be increased to a value higher than the default (128) to accommodate the expected load on the server.

Finally, the sendfile system call can be used to implement zero-copy file transfers, which can significantly improve performance for high-throughput servers serving static content. Operators should ensure their server software is configured to use sendfile wherever possible to avoid the overhead of copying data between kernel and user space.TCP window scaling is an important performance optimization that enables TCP connections to utilize more than the traditional 64KB send and receive windows. This is critical for high-throughput servers that need to maximize the amount of data in-flight. The TCP window scaling option is enabled by default in Linux kernels 2.4 and later, and can be configured with the net.ipv4.tcp_window_scaling sysctl. Operators should ensure this option is enabled for their high-throughput server workloads.

The Nagle algorithm is a TCP optimization that attempts to reduce the number of small packets on the network by buffering small amounts of output until a full-sized packet (typically 1460 bytes) can be sent. While this can be beneficial for some applications, it can hurt performance for high-throughput servers that need to minimize latency. The TCP_NODELAY socket option can be used to disable the Nagle algorithm for specific connections, allowing small packets to be sent immediately without buffering.

Congestion control algorithms play a crucial role in TCP performance tuning. The default cubic congestion control algorithm in Linux is well-suited for high-bandwidth, high-latency networks, but the newer BBR (Bottleneck Bandwidth and Round-trip propagation time) algorithm can provide even better performance in some scenarios. Operators can switch the congestion control algorithm used by their server with the net.ipv4.tcp_congestion_control sysctl.

Proper socket buffer sizing is essential for maximizing TCP throughput. The net.core.rmem_max and net.core.wmem_max sysctl parameters control the maximum receive and send buffer sizes, respectively. These should be increased to at least 8MB (8388608 bytes) on high-throughput servers to ensure the TCP stack has enough buffer space to accommodate large amounts of in-flight data.

The TIME_WAIT state in the TCP state machine can cause issues for high-throughput servers with large numbers of short-lived connections. The net.ipv4.tcp_tw_reuse and net.ipv4.tcp_tw_recycle sysctl parameters can be used to reuse or recycle these TIME_WAIT sockets, reducing the load on the system. Operators should carefully evaluate the tradeoffs of these options, as they can potentially cause compatibility issues with some clients.

Keepalive configuration is another important consideration for high-throughput servers. The net.ipv4.tcp_keepalive_time, net.ipv4.tcp_keepalive_intvl, and net.ipv4.tcp_keepalive_probes sysctl parameters control the keepalive behavior, allowing operators to detect and close idle connections more quickly. This can help reduce the number of open sockets on the server.

TCP maximum segment size (MSS) negotiation is critical for optimizing network throughput. The tcp_mss_default kernel parameter controls the default MSS value, which should be set to the largest possible value that does not result in IP fragmentation (typically 1460 bytes for Ethernet). Operators should also consider using the TCP_MAXSEG socket option to set the MSS on a per-connection basis.

SYN backlog tuning is important for servers that expect to handle large numbers of concurrent connection attempts. The net.core.somaxconn sysctl parameter controls the maximum length of the listen queue for new connections. This should be increased to a value higher than the default (128) to accommodate the expected load on the server.

Finally, the sendfile system call can be used to implement zero-copy file transfers, which can significantly improve performance for high-throughput servers serving static content. Operators should ensure their server software is configured to use sendfile wherever possible to avoid the overhead of copying data between kernel and user space.

Another important consideration for high-throughput servers is the TCP SYN cookies feature, which can help mitigate SYN flood attacks. The net.ipv4.tcp_syncookies sysctl parameter controls this functionality, and should be enabled on servers that are exposed to the public internet and may be vulnerable to such attacks.

Operators should also consider tuning the TCP FIN timeout, which controls how long a connection remains in the FIN-WAIT-2 state before being terminated. The net.ipv4.tcp_fin_timeout sysctl parameter can be adjusted to reduce the time spent in this state, freeing up resources more quickly for new connections. A value of 30 seconds is often recommended for high-throughput servers.

The TCP timestamp option can also be used to improve performance by providing more accurate round-trip time (RTT) measurements, which are used by the congestion control algorithm. The net.ipv4.tcp_timestamps sysctl parameter controls this feature, and it is generally recommended to leave it enabled for high-throughput server workloads.

Finally, operators should consider using TCP segmentation offload (TSO) and generic segmentation offload (GSO) features provided by modern network interface cards (NICs). These features can help reduce CPU utilization by offloading the segmentation of large TCP packets to the NIC hardware, improving overall server performance. The ethtool command can be used to configure these options on a per-interface basis.Another important aspect of TCP performance tuning for high-throughput servers is the TCP selective acknowledgment (SACK) option. SACK allows the receiver to inform the sender about out-of-order segments that have been successfully received, enabling more efficient retransmissions. The net.ipv4.tcp_sack sysctl parameter controls SACK support, and it is generally recommended to keep this option enabled for high-throughput workloads.

Closely related to SACK is the TCP limited transmit feature, which can improve performance in the face of sporadic packet loss. This feature allows the sender to transmit two new segments upon receiving the first few duplicate acknowledgments, helping to keep the connection active and avoiding unnecessary retransmissions. The net.ipv4.tcp_limited_transmit sysctl parameter enables this functionality.

The TCP early retransmit option is another performance-enhancing feature that can be beneficial for high-throughput servers. This feature allows the TCP stack to retransmit lost segments more aggressively, without waiting for the full retransmission timeout to expire. The net.ipv4.tcp_early_retrans sysctl parameter controls this behavior, and it is generally recommended to enable it for server workloads.

In addition to the sysctl-based tuning options, operators can also leverage socket-level options to fine-tune TCP performance on a per-connection basis. For example, the TCP_QUICKACK socket option can be used to enable a more aggressive acknowledgment strategy, reducing the delay between receiving a packet and sending the corresponding ACK. This can be particularly useful for high-throughput servers with many concurrent connections.

Another socket-level option to consider is TCP_CORK, which can be used to delay the transmission of partially filled TCP segments. This can help reduce the number of small packets on the network, improving overall efficiency. However, operators should use caution when enabling TCP_CORK, as it can also introduce additional latency, which may be undesirable for some high-throughput server workloads.

Finally, operators should be aware of the potential impact of network middleboxes, such as firewalls and load balancers, on TCP performance. These devices may interfere with TCP optimizations or introduce additional latency, and should be carefully configured and tested to ensure they do not negatively impact the performance of high-throughput server applications.

Overall, TCP performance tuning for high-throughput servers requires a comprehensive approach, addressing a wide range of kernel-level parameters, socket-level options, and network infrastructure considerations. By carefully optimizing these various aspects, operators can significantly improve the throughput and efficiency of their server applications, ensuring they can handle the demands of modern, high-traffic workloads.Another important consideration for high-throughput servers is the use of TCP Fast Open (TFO), a feature that can help reduce the latency of new TCP connections. TFO allows the client to include data in the initial SYN packet, effectively skipping the traditional three-way handshake. This can be particularly beneficial for servers handling a large number of short-lived connections. The net.ipv4.tcp_fastopen sysctl parameter controls TFO support, and operators should ensure it is enabled on both the client and server for optimal performance.

In addition to the various sysctl and socket-level tuning options, high-throughput server operators should also consider the impact of network interface card (NIC) drivers and firmware on TCP performance. Modern NICs often include advanced features, such as TCP segmentation offload (TSO), generic segmentation offload (GSO), and receive side scaling (RSS), which can significantly improve throughput and reduce CPU utilization. Operators should ensure they are using the latest stable versions of NIC drivers and firmware, and configure these offload features appropriately for their workloads.

Another important aspect of TCP performance tuning is the use of TCP connection pooling, a technique employed by many high-throughput server applications. By maintaining a pool of pre-established TCP connections, servers can avoid the overhead of setting up new connections for each client request, improving overall responsiveness and throughput. Operators should work closely with their application development teams to ensure the optimal configuration and usage of connection pooling mechanisms.

Finally, high-throughput server operators should consider the impact of network topology and infrastructure on TCP performance. Factors such as network latency, bandwidth, and packet loss can all have a significant effect on TCP throughput. Operators should carefully monitor and optimize the network environment, potentially leveraging technologies like RDMA, DPDK, or kernel-bypass networking, to ensure the underlying network can support the demands of their high-throughput server applications.

In summary, TCP performance tuning for high-throughput servers requires a multi-faceted approach, addressing a wide range of kernel-level parameters, socket-level options, network infrastructure considerations, and application-level optimizations. By carefully configuring and optimizing these various aspects, operators can ensure their server applications can effectively handle the demands of modern, high-traffic workloads, delivering the required levels of throughput and responsiveness to their end-users.