At the EuroBSD conference, Netflix has shared the configuration of the systems used for streaming content worldwide. The content provider has been able to push the throughput to 400 Gbit/s using a mix of AMD Epyc 7502p processors (32 cores) paired with 256 GB of DDR4-3200 memory. The configuration features two Mellanox ConnectX-s adapters. These are usually limited to four 100 GbR connections.
This means that the overall system throughput remains under 240 Gbit/s. However, developer Drew Gallatin shared in a lecture that he was able to optimize them to exceed data rates of 400 Gbit/s. That’s a considerable 100Gbit+ more than the usually expected 200-something.
Gallatin managed to do this by having the data transport run directly through the network hardware via a TLS offload. The FreeBSD kernel which was was used as the OS for the systems offers this possibility, therefore freeing up CPU resources. The actual TLS session is established in the userspace, but the keys used then eventually migrate using the kernel to the network hardware, where the encryption takes place. A detour via the processor is no longer necessary.
The team has not yet been able to test TLS offloading with Intel hardware. In addition, the team already has hardware prototypes that could enable network connections up to 800 Gbit/s, but these have not yet been tested either.
Source: FreeBSD (Via: Hardware.nl)
