Fleetrec: Large-scale recommendation inference on hybrid gpu-fpga clusters
Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data …, 2021•dl.acm.org
We present FleetRec, a high-performance and scalable recommendation inference system
within tight latency constraints. FleetRec takes advantage of heterogeneous hardware
including GPUs and the latest FPGAs equipped with high-bandwidth memory. By
disaggregating computation and memory to different types of hardware and bridging their
connections by high-speed network, FleetRec gains the best of both worlds, and can
naturally scale out by adding nodes to the cluster. Experiments on three production models …
within tight latency constraints. FleetRec takes advantage of heterogeneous hardware
including GPUs and the latest FPGAs equipped with high-bandwidth memory. By
disaggregating computation and memory to different types of hardware and bridging their
connections by high-speed network, FleetRec gains the best of both worlds, and can
naturally scale out by adding nodes to the cluster. Experiments on three production models …
We present FleetRec, a high-performance and scalable recommendation inference system within tight latency constraints. FleetRec takes advantage of heterogeneous hardware including GPUs and the latest FPGAs equipped with high-bandwidth memory. By disaggregating computation and memory to different types of hardware and bridging their connections by high-speed network, FleetRec gains the best of both worlds, and can naturally scale out by adding nodes to the cluster. Experiments on three production models up to 114 GB show that FleetRec outperforms optimized CPU baseline by more than one order of magnitude in terms of throughput while achieving significantly lower latency.