Efficient LLM Workload Hosting at the Telco Edge

Le résumé fourni par la source

Network operators are gradually evolving “telco cloud” infrastructure to deeper edge locations to service a range of network-oriented workloads. To maximise the return-of-investment of such edge compute, additional workload and application types can be considered to run on such infrastructure. Large Language Model (LLM) inference is one such candidate application, however LLMs are potentially very compute resource intensive, and the way in which they consume resources must be subject to careful attention. Achieving a suitable balance of power and performance for such workloads within what may be a particularly constrained resource envelope at the edge, is a significant challenge. This challenge must also consider actual performance targets of the LLM workload itself: is “good enough” performance acceptable if it reduces costs? This may be achievable in cases where more standard CPU-based computing hardware is used in favour of expensive GPUs. Our experiments reveal that although GPU-based cases perform best, there were some very respectable CPU-based results when CPU pinning to the same processor socket was involved; 18 Tokens Per Second and sub-second Time-to-First Token values were observed. We outline the rationale not just for multi-tenant LLM instances to maximise edge server efficiency, but also the prospect of differentiated LLM performance based on allocated resources.

Ce résumé expose les affirmations des auteurs. BNTIC ne l’interprète pas comme une validation indépendante des résultats.