Primer: How XDP and eBPF Speed Network Traffic via the Linux Kernel
Every so often, a new buzzword or acronym comes around that really has weight behind it. Such is the case with XDP (eXpress Data Path). This technology allows developers to attach eBPF programs to a low-level hook, implemented by the network device driver, within the Linux kernel. XDP is similar to the Data Plane Development Kit (dpdk), but will not be detached from the driver.
There have been a number of approaches to gain performance benefits for high-performance packet processing. One such approach is kernel bypass, which hands control of a NIC to user-space programs to greatly reduce the overhead introduced into the kernel by things like context switching, networking layer processing, and interruptions. When you’re working at higher networking speeds (10Gbps or higher), this becomes relevant.
However, there are several disadvantages to the kernel bypass method. First off, user-space programs will need to program their own drivers. That’s extra work for developers. Also, when the kernel-space is replaced by user-space, all the network functionality found in the kernel is skipped. That means user-space programmers will have to add necessary functionalities into their work. Finally, user-space programs (under the kernel bypass method) would function within sandboxes, making it very hard to integrate with other areas of the operating system.
Those limitations are exactly why XDP has become the darling of high-performance networking. With this method, user-space programs will be allowed to directly read and write to network packet data and make decisions on how to handle a packet before it reaches the kernel level. In other words, user-space takes care of some of the overhead, so the bulk of these decisions and actions are placed solely on the shoulders of the kernel.
XDP and eBPF Working Together
XDP isn’t a language. Instead, XDP uses the eBPF programming language to gain access to the lower-level kernel hook. That hook is then implemented by the network device driver within the ingress traffic processing function, before a socket buffer can be allocated for the incoming packet.
Let’s look at how these two work together. This outstanding example comes from Jeremy Erickson, who is a senior R&D developer with Duo (which is now a part of Cisco). We’ll illustrate Erickson’s example by way of Ubuntu 20.04. So in order to follow along, you should have a Ubuntu instance at the ready. What we’re going to do is demonstrate XDP and eBPF working together with the help of C and Python. It’s actually quite an interesting little experiment.
What you’ll need to do is SSH into your Ubuntu instance four times (using four different terminals). Once you’ve successfully logged in, we’re going to create a .py file and a .c file. The Python file will serve as our XDP loader that will load the XDP program (the .c file) into the data path.
Create the Python file with the command:
nano main.py
In that file, paste the following:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 |
#!/usr/bin/env python3 from bcc import BPF import time device = "lo" b = BPF(src_file="filter.c")fn = b.load_func("udpfilter", BPF.XDP)b.attach_xdp(device, fn, 0) try: b.trace_print()except KeyboardInterrupt: pass b.remove_xdp(device, 0) |
As you can see, the above file makes use of BPF for the majority of its functionality.
Save and close the file.
Next, create the actual XDP program with the command
nano filter.c
In that file, paste the following:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 |
#define KBUILD_MODNAME "filter" #include <linux/bpf.h> #include <linux/if_ether.h> #include <linux/ip.h> #include <linux/in.h> #include <linux/udp.h> int udpfilter(struct xdp_md *ctx) { bpf_trace_printk("got a packet\n"); void *data = (void *)(long)ctx->data; void *data_end = (void *)(long)ctx->data_end; struct ethhdr *eth = data; if ((void*)eth + sizeof(*eth) <= data_end) { struct iphdr *ip = data + sizeof(*eth); if ((void*)ip + sizeof(*ip) <= data_end) { if (ip->protocol == IPPROTO_UDP) { struct udphdr *udp = (void*)ip + sizeof(*ip); if ((void*)udp + sizeof(*udp) <= data_end) { if (udp->dest == ntohs(7999)) { bpf_trace_printk("udp port 7999\n"); udp->dest = ntohs(7998); } } } } } return XDP_PASS; } |
Again, we’re using BPF to make XDP happen.
Save and close the file.
On the off-chance your Ubuntu machine doesn’t have Python installed, do so with the command:
sudo apt-get install python3 -y
You might also need to install a few dependencies. This can be done with the following three commands:
sudo apt install clang llvm libelf-dev libpcap-dev gcc-multilib build-essential -y
sudo apt install linux-tools-$(uname -r)
sudo apt install linux-headers-$(uname -r) -y
Now, go to the first terminal window and run the following command:
nc -kul 127.0.0.1 7999
The above command will create an arbitrary TCP connection to the loopback device, listening on port 7999.
On the second terminal issue the command:
nc -kul 127.0.0.1 7998
This time we’re listening on port 7998.
From the third terminal, we’re going to issue the command:
nc -u 127.0.0.1 7999
After you issue the command above, type the following (in terminal 3):
Hello, New Stack!
Hit Enter on your keyboard and you should see “Hello, New Stack!” in the output on the first terminal window (Figure 1).
-
Figure 1: Our message was printed out on the first terminal.
Why did the message print out on the first terminal? Because we sent the message to port 7999.
Okay, using [Ctrl]+, exit out of all three commands.
What we’re going to demonstrate now, using Erickson’s handy loader and program, is how XDP can reroute that message from listening port 7999 to port 7998. Why does this happen? Using BPF in the filter.c program, the packet is rerouted to the 7998 port…all of this done in user-space (and not kernel-space).
Issue the same commands you ran above, in the same terminals. Once you have all the commands listening to their necessary ports, go to the fourth terminal and issue the command:
sudo python main.py
Go back to the third terminal (where you ran the command nc -u 127.0.0.1 7999) and type:
Hello, New Stack!
If you go to the first terminal (port 7999) you won’t see the message. If, however, you look at the second terminal (port 7998) you’ll see the message (Figure 2).
-
Figure 2: The packet was rerouted in user-space to the different port.
You will also see the following printed out repeatedly in terminal 4:
b' Raven.Server-26114 [000] .Ns1 54853.031451: 0: got a packet'
And that’s a very simple example of using eBFP to load an XDP program into the data path to reroute network packets. This is a very powerful technology that could have massive implications on the performance of Linux with high-speed networks and the packets that travel from user- to kernel-space.