lswcap
Netflix is going through trial and error to find the ideal combination of hardware and software to improve video streaming quality. Netflix is reporting the results of using EPYC, a CPU for AMD servers.
Starting in 2020, Netflix has delivered video content at a rate of 200 Gbps per server. Netflix’s next goal is to double the speed to 400Gbps, and to achieve this goal, it has adopted the AMD’s 2nd generation server CPU, the EPYC 7502P (EPYC 7502P).
Epic 7502P has 32 cores. The maximum operating clock is 2.5 GHz and the maximum clock is 3.35 GHz. The Netflix server is also equipped with 256GB DDR4-3200 and provides 150Gbps bandwidth with an 8-channel memory interface. In addition, 18 Western Digital SN720 2TB per server are used for storage. Two network cards (Mellanox ConnectX-6 Dx) connected through PCIe 4.0 x16 were also introduced per server. This configuration was expected to provide a speed of 250 Gbps.
However, as a result of conducting a connection test on such a server, the first result that appeared was a number of 240 Gbps. Netflix, who saw that the number was lower than expected due to memory bandwidth limitations, tried various NUMA (Non-Uniform Memory Access) configurations to maximize bandwidth. The NUMA configuration allowed the allocation of memory controllers and devices connected via PCIe to the optimal core, but could be delayed depending on the NUMA inter-node process.
This allows Netflix to increase its NUMA nodes to four and deliver speeds of up to 280 Gbps by concentrating on NUMA nodes that store both storage and network-specific processes and are optimally connected to PCIe devices.
Software optimization was considered as the next step to reach 400 Gbps. To avoid hardware bottlenecks, we implemented TLS in FreeBSD and offloaded TLS encryption to the adapter (Mellanox ConnectX-6 Dx) instead of the Epic 7502P, resulting in a maximum of 380Gbps with two 190Gbps per adapter. Processor utilization was 60% without NUMA and 50% with 4 NUMA.
Netflix also said that the Epic 7502P showed the fastest value even when testing with different configurations such as Xeon Platinum 8352V and Ampere Altera Q80-30. Netflix has already set its next goal of reaching 800 Gbps and is likely to release new information in 2022 as well. Related information can be found here.
Add comment