/**
\page yarp_cluster Setting up your network for YARP

\tableofcontents

Along with TCP, YARP supports UDP and multicast communication on a
local network. These are unreliable network protocols, but can be
quite efficient. We discuss here steps that are useful for making your
network friendly to UDP and multicast. But if you find yourself
spending a lot of time on this, you may need to reconsider whether you
have made the right choice of protocol.

NOTE: if you are just getting started with YARP, you
don't need to look at any of this yet!  It only matters when you
are working with a cluster of computers using unreliable protocols,
or run into bandwidth problems.  But it is important to remember that
this page is here for when you need it.

\section yarp_cluster_iperf iperf is your friend

If you are doing any non-trivial networking, you need to learn how
to test your network and debug throughput issues.  A key resource
in this battle is the "iperf" program.

- Download from http://software.es.net/iperf/
- Linux: apt-get install iperf (or similar command)
- macOS: brew install iperf (if you use homebrew)

Please make sure you have iperf available on all your machines
before going any further.

\section yarp_cluster_tcp Checking TCP communication

TCP is the simplest protocol to work with, so let's start with it.
Let's check the throughput between two of your machines.
Pick two machines, and choose one (arbitrarily) to be the iperf
server.  Determine its IP address of domain name -- let's say it is
10.0.0.26. Run the following commands:

\verbatim
 [On server, e.g. 10.0.0.26] iperf -s
 [On client, e.g. 10.0.0.27] iperf -c 10.0.0.26
\endverbatim

This will run for a while and then describe your TCP throughput, for example:

\verbatim
 ------------------------------------------------------------
 Client connecting to 10.0.0.26, TCP port 5001
 TCP window size: 16.0 KByte (default)
 ------------------------------------------------------------
 [  3] local 10.0.0.25 port 34160 connected with 10.0.0.26 port 5001
 [  3]  0.0-10.0 sec  1.08 GBytes    924 Mbits/sec
 ...
\endverbatim

\section yarp_cluster_udp Checking UDP communication

UDP is an unreliable protocol, and there are lots of possible pitfalls
in using it.  YARP will try to send messages broken into rather large
UDP packets (around 60K), and your network may or may not actually
pass those packets through.

To test UDP behavior on your network with iperf, add the "-u" flag to
the client and server. To the server, we can add "-i 1" to get reports
at regular intervals. To the client, we can specify target bandwidth,
e.g. "-b 50m" for a target of 50 MBits/sec.

\verbatim
 [On server, 10.0.0.26] iperf -s -u -i 1
 [On client, 10.0.0.27] iperf -c 10.0.0.26 -b 50m
\endverbatim

To test a gigabit network:

\verbatim
 [On server, 10.0.0.26] iperf -s -u -i 1
 [On client, 10.0.0.27] iperf -c 10.0.0.26 -b 1000m
\endverbatim

On a healthy gigabit network you should get a bandwidth of ~950 Mbits/sec.

Example result on the client side:

\verbatim
------------------------------------------------------------
Client connecting to 10.0.0.26, UDP port 5001
Sending 1470 byte datagrams
UDP buffer size:   122 KByte (default)
------------------------------------------------------------
[  3] local 10.0.0.221 port 52990 connected with 10.0.0.222 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0- 6.1 sec    700 MBytes    956 Mbits/sec
[  3] Sent 499273 datagrams
[  3] Server Report:
[  3]  0.0- 6.1 sec    699 MBytes    955 Mbits/sec  0.014 ms  575/499272 (0.12%)
[  3]  0.0- 6.1 sec  1 datagrams received out-of-order
\endverbatim

Result on the server side:
\verbatim
------------------------------------------------------------
Server listening on UDP port 5001
Receiving 1470 byte datagrams
UDP buffer size:   122 KByte (default)
------------------------------------------------------------
[  3] local 10.0.0.26 port 5001 connected with 10.0.0.27 port 52990
[ ID] Interval       Transfer     Bandwidth       Jitter   Lost/Total Datagrams
[  3]  0.0- 1.0 sec    114 MBytes    954 Mbits/sec  0.012 ms   88/81245 (0.11%)
[  3]  1.0- 2.0 sec    114 MBytes    955 Mbits/sec  0.011 ms   92/81263 (0.11%)
[  3]  2.0- 3.0 sec    114 MBytes    955 Mbits/sec  0.014 ms   90/81287 (0.11%)
[  3]  3.0- 4.0 sec    114 MBytes    955 Mbits/sec  0.012 ms   98/81273 (0.12%)
[  3]  4.0- 5.0 sec    114 MBytes    954 Mbits/sec  0.012 ms   99/81263 (0.12%)
[  3]  5.0- 6.0 sec    114 MBytes    955 Mbits/sec  0.012 ms   95/81288 (0.12%)
[  3]  0.0- 6.1 sec    699 MBytes    955 Mbits/sec  0.014 ms  575/499272 (0.12%)
[  3]  0.0- 6.1 sec  1 datagrams received out-of-order
\endverbatim

A key test for YARP is that you can send big datagrams okay. This is
necessary for efficient transmission of large images using UDP or
MCAST. To check 63KB datagrams, do:

\verbatim
 [On server, 10.0.0.26] iperf -s -u -l 63K -w 200K -i 1
 [On client, 10.0.0.27] iperf -c 10.0.0.26 -u -l 63K -w 200K -b 1000m
\endverbatim

Make sure you try this in both directions! For example, we've seen an
asymmetric problems going from a Linux machine to a Windows machine on
a gigabit network with a particular switch.  If you see problems,
reduce the "-l" size until things work just to understand your system
(but if you want to send large images you'll need to tweak your system
until large packets work).

It could also be worth checking several machines since different network cards
can have different performance.

\section  yarp_cluster_mcast Checking multicast communication

For testing multicast, make sure you have the most recent version
of iperf, and try this:

\verbatim
 iperf -s -u -B 224.0.55.55 -i 1 -T 32 -l 60k
 iperf -c 224.0.55.55 -u -T 32 -t 100 -i 1 -l 60k -b 1000m
\endverbatim

Note the IP addresses are multicast-style, they don't correspond to
individual machines.

On the client ("-c") side, you should see something like:

\verbatim
------------------------------------------------------------
Client connecting to 224.0.55.55, UDP port 5001
Sending 61440 byte datagrams
Setting multicast TTL to 32
UDP buffer size:   122 KByte (default)
------------------------------------------------------------
[  3] local 10.0.0.221 port 60992 connected with 224.0.55.55 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0- 1.0 sec    114 MBytes    958 Mbits/sec
[  3]  1.0- 2.0 sec    114 MBytes    956 Mbits/sec
[  3]  2.0- 3.0 sec    114 MBytes    956 Mbits/sec
[  3]  3.0- 4.0 sec    114 MBytes    956 Mbits/sec
\endverbatim

Server side:

\verbatim
------------------------------------------------------------
Server listening on UDP port 5001
Binding to local address 224.0.55.55
Joining multicast group  224.0.55.55
Receiving 61440 byte datagrams
UDP buffer size:   122 KByte (default)
------------------------------------------------------------
[  3] local 224.0.55.55 port 5001 connected with 10.0.0.221 port 60992
[ ID] Interval       Transfer     Bandwidth       Jitter   Lost/Total Datagrams
[  3]  0.0- 1.0 sec    111 MBytes    931 Mbits/sec  0.380 ms   50/ 1945 (2.6%)
[  3]  1.0- 2.0 sec    111 MBytes    934 Mbits/sec  0.442 ms   46/ 1947 (2.4%)
[  3]  2.0- 3.0 sec    112 MBytes    941 Mbits/sec  0.280 ms   31/ 1946 (1.6%)
[  3]  3.0- 4.0 sec    112 MBytes    935 Mbits/sec  0.362 ms   43/ 1946 (2.2%)
[  3]  4.0- 5.0 sec    110 MBytes    920 Mbits/sec  0.459 ms   75/ 1947 (3.9%)
[  3]  5.0- 6.0 sec    111 MBytes    934 Mbits/sec  0.418 ms   45/ 1946 (2.3%)
[  3]  6.0- 7.0 sec    111 MBytes    933 Mbits/sec  0.380 ms   48/ 1947 (2.5%)
[  3]  7.0- 8.0 sec    111 MBytes    933 Mbits/sec  0.365 ms   49/ 1947 (2.5%)
[  3]  8.0- 9.0 sec    112 MBytes    936 Mbits/sec  0.319 ms   41/ 1946 (2.1%)
\endverbatim

\section yarp_cluster_suggest YARP's suggestion

If MCAST/UDP packets drop when using YARP, YARP may suggest this:

\verbatim
 The UDP/MCAST system buffer limit on your system is low.
 You may get packet loss under heavy conditions.
 To change the buffer limit on linux: sysctl -w net.core.rmem_max=8388608
 (Might be something like: sudo /sbin/sysctl -w net.core.rmem_max=8388608)
 To change the limit use: sysctl for Linux/FreeBSD, ndd for Solaris, no for AIX
\endverbatim

This controls the maximum allowed size for a RECEIVER buffer, so that
a number of packets adding up to a large amount of data can be
received and stored while your program is doing something else. YARP
will try to get a large buffer, and your operating system may not
comply unless you do the above. This configuration step is not
necessary on MS-Windows.

There are a lot more suggestions for Linux here:
- http://wwwx.cs.unc.edu/~sparkst/howto/network_tuning.php

\section yarp_cluster_tweak YARP connection problems

If you are confident of your network, but are having trouble with
making a YARP connection of some kind, read about that kind of
connection in \ref carrier_config and see if there's a tweak that
can be made.  There are some parameters you can control for
the \ref carrier_config_udp and \ref carrier_config_mcast.

\section yarp_cluster_code Relevant source code

In case you need to fix something, the relevant source code in YARP is:

- `src/libYARP_os/src/yarp/os/impl/DgramTwoWayStream.h`
- `src/libYARP_os/src/yarp/os/impl/DgramTwoWayStream.cpp`

We appreciate patches to these files. We only have a few networks to test
on, and this topic can be quite network-dependent and subtle.

*/
