Bandwidth Optimized Scalable Designs with Inter-Layer Overlapping for MPI Broadcast

Le résumé fourni par la source

MPI (Massage Passing Interface) has been the dominant programming model for developing large-scale parallel applications. Existing work mainly focuses on the vast parallelism of modern multi-/many-cores architectures to parallelize MPI collectives. However, the abundant bandwidth provided by modern interconnects is either underutilized when processing small messages or overwhelmed by large messages. MPI_Bcast is one of the most widely used collective primitives in MPI, which broadcasts data from one process to all processes in the communication domain. In this paper, we address the issue of load imbalance arising from traditional tree-based designs in order to strike a better balance between bandwidth and latency of MPI Broadcast. By evenly distributing the broadcast load across lower layers, we effectively leverage the available resources at upper-layer nodes that recursively execute broadcast across different layers in a sequential and contention-free manner. This approach improves the scalability and performance of large-scale message broadcasts. Additionally, a generic inter-layer overlapped scheme is proposed to reduce overall broadcast latency by fully overlapping inter-layer and intra-layer data transmission. We further implement an online adaptive scheme for tree degree tuning to achieve optimal design at various message and system sizes. Extensive experiments are conducted to evaluate the performance of this Bandwidth-optimized Inter-layer Overlapping (BIO) design at both the microbenchmark and application levels. BIO-based MPI_Bcast designs demonstrate performance speedups of up to 2.71x compared to state-of-the-art MPI libraries. For application-level evaluation, BIO provides up to 165% acceleration for the initialization of the distributed deep learning model of Horovod with the PyTorch application.

Ce résumé expose les affirmations des auteurs. BNTIC ne l’interprète pas comme une validation indépendante des résultats.