POU: Outlier Uniformity-Based Layer-Wise Pruning Method for Large Language Models

Le résumé fourni par la source

Large Language Models (LLMs) face deployment challenges on edge devices due to their size and computational cost. Pruning reduces parameters by removing redundancy, preserving inference performance. Some LLM pruning methods focus on designing pruning metrics such as Wanda and SparseGPT. These methods usually apply the same sparsity level to all layers, which limits pruning effectiveness. Existing non-uniform layer-wise pruning methods face challenges in balancing pruned model performance and pruning efficiency. We find that outlier uniformity correlates with pruned LLM performance. To leverage this, we propose POU, a layer-wise pruning method using an entropy-based Layer Importance Score (LIS) to allocate sparsity across layers while preserving performance. POU integrates with most pruning methods and further improves their results. Experiment results show that POU achieves a better pruned model performance and pruning efficiency than existing layerwise pruning methods in most scenarios. For example, on the Llama1-7B model at 70 % sparsity, POU reduces perplexity by$\text{1 2. 7 8} \%$(combined with Wanda) and$\text{5. 8 \%}$(combined with SparseGPT) compared to state-of-the-art method AlphaPruning, with nearly half decrease of the pruning time.

Ce résumé expose les affirmations des auteurs. BNTIC ne l’interprète pas comme une validation indépendante des résultats.