FreeBSD contributor Drew Gallatin gives a talk about the FreeBSD Network Stack and optimizing NUMA, and its real world application for Netflix’s servers. This presentation occurred at EuroBSDCon 2019 in Lillehammer, Norway.

I will discuss optimizations to keep network connections and their resources local to NUMA domains. These changes include:

Allocating NUMA local memory to back files sent via sendfile(9).
Allocating NUMA local memory for Kernel TLS crypto buffers.
Directing connections to TCP Pacers and kTLS workers bound to the local domain.
Directing incoming connections to Nginx workers bound to the local domain via modifications to SO_REUSEPORT_LB listen sockets.
I will present data from real Netflix servers showing an improvement of almost 2x on AMD EPYC (85Gbs – 165Gbs), and 1.3x on Intel Xeon (140Gb/s – 180Gbs). I will present data from the Xeon system showing a 50% reduction in cross-domain traffic.

NUMA Optimizations in the FreeBSD Network Stack: https://papers.freebsd.org/2019/eurobsdcon/gallatin-numa_optimizations_in_the_freebsd_network_stack/