Efficient 8-Bit Quantization of Transformer Neural Machine Language\n Translation Model

Le résumé fourni par la source

In this work, we quantize a trained Transformer machine language translation\nmodel leveraging INT8/VNNI instructions in the latest Intel$^\\circledR$\nXeon$^\\circledR$ Cascade Lake processors to improve inference performance while\nmaintaining less than 0.5$\\%$ drop in accuracy. To the best of our knowledge,\nthis is the first attempt in the industry to quantize the Transformer model.\nThis has high impact as it clearly demonstrates the various complexities of\nquantizing the language translation model. We present novel quantization\ntechniques directly in TensorFlow to opportunistically replace 32-bit floating\npoint (FP32) computations with 8-bit integers (INT8) and transform the FP32\ncomputational graph. We also present a bin-packing parallel batching technique\nto maximize CPU utilization. Overall, our optimizations with INT8/VNNI deliver\n1.5X improvement over the best FP32 performance. Furthermore, it reveals the\nopportunities and challenges to boost performance of quantized deep learning\ninference and establishes best practices to run inference with high efficiency\non Intel CPUs.\n

Ce résumé expose les affirmations des auteurs. BNTIC ne l’interprète pas comme une validation indépendante des résultats.