Automating network testing with Iperf3
TCP/IP Networking is a complex topic; it gets really tricky when you are trying to nail down issues like performance, or troubleshooting a problem.
In any case it helps to have tools that can help you to confirm your suspicious or better probe than there is no problem at all; One of those is the Open Source iperf3:
iperf is a tool for active measurements of the maximum achievable bandwidth on IP networks. It supports tuning of various parameters related to timing, protocols, and buffers. For each test it reports the measured throughput / bitrate, loss, and other parameters.
Here you will learn how to do the following:
- Inspect bandwidth issues with Iperf3 between 2 endpoints
- Test UDP multicast connectivity (protocols like PTP use it for time synchronization)
- Uncover CRC errors on a network interface (I want to also show what you can do with ethtool and tcpdump to confirm that traffic is indeed dropped by a bad network interface or cable)
- And how to automate to write more complex scripts using Python 3.
- As a bonus will briefly explain CPU affinity and why it may matter to iperf3
You will need to have the following:
- A Linux distribution (all my examples were executed on a Fedora server)
- Ability to run commands as root (Using SUDO for example)
- Basic understanding of networking principles
Installing iperf is a simple as doing this on Fedora:
sudo dnf install -y iperf3
Without further delay let's get started.
Measuring bandwidth and jitter
So let's start by defining/ remembering a few things:
- Throughput: measures how many packets arrive at their destinations successfully.
- Network bandwidth is defined as the maximum transfer throughput capacity of a network.
- Jitter: Is the time delay between when a signal is transmitted and when it is received. Good connections have consistent response time.
- TCP is a reliable protocol, guarantees arrival of packets on the same order they were sent.
- UDP doesn't have a handshake protocol like TCP; It is faster than TCP but if a packet is lost it won't be resent and there is no guarantee on the order packets will arrive.
Iperf3 works by running a client and a server that talk to each other; For our demostration will run as follows:
- Client and server binding to the wired ethernet interface (I will not use the Wireless interfaces as they are more prone to jitter due external noise)
- Test will use defaults (port, TCP connection unless we override with the flag
--udpon the client)
Also, will confirm if the following is true: * The switch in between the 2 machines supports 1000 Mbits/sec connections, and the interfaces are also configured at that capacity. * Full-duplex mode (send and receive data on the card simultaneously. Will confirm with another tool called ethtool, will explain how it works in a bit)
Running the server:
josevnz@raspberrypi:~$ sudo ethtool eth0|rg -e 'Speed|Duplex' Speed: 1000Mb/s Duplex: Full josevnz@raspberrypi:~$ ip --oneline address|rg 192 2: eth0 inet 192.168.1.11/24 brd 192.168.1.255 scope global dynamic eth0\ valid_lft 2090sec preferred_lft 2090sec josevnz@raspberrypi:~$ iperf3 --server --bind 192.168.1.11 -affinity 1 ----------------------------------------------------------- Server listening on 5201 -----------------------------------------------------------
And now the client:
[josevnz@dmaf5 ~]$ sudo ethtool eno1|rg -e 'Speed|Duplex' Speed: 1000Mb/s Duplex: Full [josevnz@dmaf5 ~]$ iperf3 --client raspberrypi --bind 192.168.1.28 --affinity 1 Connecting to host raspberrypi, port 5201 [ 5] local 192.168.1.28 port 47609 connected to 192.168.1.11 port 5201 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 111 MBytes 932 Mbits/sec 0 2.79 MBytes [ 5] 1.00-2.00 sec 110 MBytes 923 Mbits/sec 0 2.98 MBytes ... - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 1021 MBytes 857 Mbits/sec 0 sender [ 5] 0.00-9.95 sec 1020 MBytes 860 Mbits/sec receiver iperf Done.
Let's digest the results: * Zero retries (Retr column). That is good and expected * Bitrate is around 860 Mbits/sec. The link speed is close to the theoretical bandwidth; Switches have a limit of how much traffic the backplane can handle. * TCP guarantees losses packet transmission so Jitter is not reported here.
If you reverse the test (client is now server, server is now client) you should see similar results.
Testing UDP bandwidth
To test UDP we do the following on the client only:
[josevnz@dmaf5 ~]$ iperf3 --client raspberrypi --bind 192.168.1.28 --udp --affinity 1 Connecting to host raspberrypi, port 5201 [ 5] local 192.168.1.28 port 47985 connected to 192.168.1.11 port 5201 [ ID] Interval Transfer Bitrate Total Datagrams [ 5] 0.00-1.00 sec 129 KBytes 1.05 Mbits/sec 91 [ 5] 1.00-2.00 sec 127 KBytes 1.04 Mbits/sec 90 [ 5] 2.00-3.00 sec 129 KBytes 1.05 Mbits/sec 91 ... - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Jitter Lost/Total Datagrams [ 5] 0.00-10.00 sec 1.25 MBytes 1.05 Mbits/sec 0.000 ms 0/906 (0%) sender [ 5] 0.00-9.99 sec 1.25 MBytes 1.05 Mbits/sec 0.028 ms 0/906 (0%) receiver
- Bit-rate is much closer to the theoretical bandwidth. Also, no packet loss, which is great!
- UDP doesn't guarantee packet loss, so Lost datagrams and Jitter are reported (and they have good values).
Now, you may be wondering what is that '--affinity' flag? We didn't really need it here to test the bandwidth on this simple example but will give us an excuse to talk about affinity.
Quick detour: CPU Affinity, NUMA, isolcpus
If you were curious and checked the documentation and examples of iperf you probably saw references to CPU or Processor affinity;
So what is it? from the Wikipedia site:
Enables the binding and unbinding of a process or a thread to a central processing unit (CPU) or a range of CPUs, so that the process or thread will execute only on the designated CPU or CPUs rather than any CPU.
Why would I want to 'pin' my process to a specific group of CPU's?
No instance with pinned CPUs can use the CPUs of another pinned instance, thus preventing resource contention between instances. Non-Uniform Memory Access or NUMA allows multiple CPUs to share L1, L2, L3 caches, and main memory.
How does a server with several NUMA nodes look like? You can find out with
lscpu| rg NUMA:
[josevnz@dmaf5 ~]$ lscpu|rg NUMA NUMA node(s): 2 NUMA node0 CPU(s): 0-7 NUMA node1 CPU(s): 8-15
Here you have a 16 CPU server, with 2 NUMA nodes (this is a simplified example, a machine with HyperThreading enabled looks different. Depending on the application you may decide to disable it)
Keep in mind than CPU affinity can be used not just to increase networking performance but also disk.
Coming back to iperf3, you can pin it to a specific cpu using
-A, --affinity, for example CPU 3 (Numbered 0 to n-1):
# Equivalent of running iperf3 with numactl: /bin/numactl --physcpubind=2 iperf3 -c remotehost iperf3 --affinity 2 --client remotehost
Keep in mind that you may also need to tell the operating system to avoid running host processes on this CPU's, so you if you use Grubby you can do this with isolcpus:
# Find the default kernel sudo grubby --default-kernel # Use that information and add isolcpus parameter, then reboot sudo grubby --update-kernel=/boot/vmlinuz-5.14.18-100.fc33.x86_64 --args="isolcpus=2" sudo shutdown -r now 'Updated kernel isolcpus, need to reboot'
Again, this is not needed to troubleshoot a networking issue, but it may come handy if you want to make iperf3 behave like one of your fine-tuned applications.
This is a complex topic, so get a cup of coffee (or two) and get ready to start reading.
Using Iperf3 to detect Dropped packets, CRC errors
A CRC error is caused by a faulty physical device (network card, switch port, cable) or a mismatch on Full/Half duplex configurations between 2 devices; These are sometimes difficult to track on switches with cut-through mode which means it forwards received errors out of all ports.
This is a simplified scenario, where we want to make sure a new network card connection works without CRC errors, rx/tx errors (so the card, cable and switch port are OK).
With that in mind we see that we could do a simple test to ensure our link health is good:
- Capture the status of the CRC, dropped packets errors on the network card we are testing
- Run Iperf on TCP mode for a longer time than usual
Capture again the network card CRC stats.
If the difference is greater than zero then:
Check the full duplex on both the card and switch port (ethtool )
- Replace the cable
- Then reseat or replace the network card
- Then change the port on the switch
But you get the picture; iperf3 will help us to "burn" the link and trigger any unwanted behaviour before we use this interface in production.
Time to see this in action, say we take a first snapshot on our iperf3 server:
josevnz@raspberrypi:~$ sudo ethtool --statistics eth0| rg -i -e 'dropped|error' rx_errors: 0 tx_errors: 0 rx_dropped: 0 tx_dropped: 0 rxq0_errors: 0 rxq0_dropped: 0 rxq1_errors: 0 rxq1_dropped: 0 rxq2_errors: 0 rxq2_dropped: 0 rxq3_errors: 0 rxq3_dropped: 0 rxq16_errors: 0 rxq16_dropped: 0
Then the client
[josevnz@dmaf5 ~]$ sudo ethtool --statistics eno1| rg -i -e 'dropped|errors' tx_errors: 0 rx_errors: 0 align_errors: 0
Run the Iperf3 server:
josevnz@raspberrypi:~$ iperf3 --server --bind 192.168.1.11 ----------------------------------------------------------- Server listening on 5201 -----------------------------------------------------------
Run the client, for 120 seconds:
[josevnz@dmaf5 ~]$ iperf3 --client raspberrypi --bind 192.168.1.28 --time 120 Connecting to host raspberrypi, port 5201 [ 5] local 192.168.1.28 port 41337 connected to 192.168.1.11 port 5201 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 111 MBytes 934 Mbits/sec 0 2.94 MBytes [ 5] 1.00-2.00 sec 111 MBytes 933 Mbits/sec 0 2.95 MBytes [ 5] 2.00-3.00 sec 111 MBytes 933 Mbits/sec 0 2.95 MBytes ... - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-120.00 sec 11.0 GBytes 787 Mbits/sec 0 sender [ 5] 0.00-119.70 sec 11.0 GBytes 789 Mbits/sec receiver # Measure again ... [josevnz@dmaf5 ~]$ sudo ethtool --statistics eno1| rg -i -e 'dropped|errors' tx_errors: 0 rx_errors: 0 align_errors: 0
Good. Now let's talk about another tool we have been using to get the network interface statistics, ethtool.
What is ethtool?
ethtool is the primary means in Linux kernel-based operating systems (primarily Linux and Android) for displaying and modifying the parameters of network interface controllers (NICs) and their associated device driver software from application programs running in userspace.
Little trivia question for you (after you're done checking the man page for ethtool):
- What does the
sudo ethtool -g eno1command does?
- And this one?:
sudo ethtool -s eno1 speed 1000 duplex full autoneg on
This is another tool that you should have in your toolset.
Automating Iperf3 with Python 3
If you notice, Iperf3 has also a library that allows you to integrate this tool with other languages like Python:
[josevnz@dmaf5 ~]$ rpm -qil iperf3|rg libiperf /usr/lib64/libiperf.so.0 /usr/lib64/libiperf.so.0.0.0 /usr/share/man/man3/libiperf.3.gz
There are several bindings for Python out there:
- Iperf3 Python has an API to integrate Iperf3 with Python, using those bindings.
- Ethtool Python3 is available but is marked as deprecated. So for what we need, will use it :-).
I will not cover their API here, but rather point you to the source code of a python script that uses iperf3 and ethtool to detect network errors (as we did manually early on). Below you can see it running.
Please check out the repository and run the script, you will be amazed how easy is to automate a few tasks with Python.
Instead, let's see it in action:
What you can do next?
Learning never stops, so here are a few pointers and observations to keep you going:
- Fasterdata has more examples on how to use iperf with different parameters
- Still curious about how to use CPU affinity? Check the RedHat Systemd documentation, it has very useful examples
- Be aware that isolcpus is considered deprecated and usage of cpuset is recommended; Please refer to the following discussion to see how to play with cpuset
- You know now how you can write your own troubleshooting scripts with the Iperf3 Python API. You should probably write now an iperf3 server that can show the results using to a web browser (Maybe combine it with FastAPI?)