Federated Private Fine-tuning Language Models with Multiple Perturbed Inferences

Résumé fourni par la source

Federated learning has emerged as a promising distributed machine learning paradigm that enables collaborative model training across decentralized data sources while preserving data locality. Recent advances have introduced zeroth-order optimization (ZOO) into federated learning to mitigate the high communication overhead inherent in traditional gradient-based methods. However, privacy leakage risks remain insufficiently addressed under such zeroth-order federated frameworks. To this end, this paper proposes FedDPMZO, a Differentially Private Multi-Perturbation Zeroth-Order federated learning framework. The core of FedDPMZO lies in its multi-perturbation gradient estimation mechanism. During each local iteration, clients sample multiple Gaussian perturbations to construct diverse gradient directions. For each direction, the estimated gradient is individually clipped, averaged across multiple mini-batches, and then perturbed by calibrated Gaussian noise to satisfy differential privacy guarantees. Extensive experiments on several natural language understanding fine-tuning tasks demonstrate that FedDPMZO achieves superior model performance and convergence efficiency compared with existing zeroth-order federated methods, while rigorously adhering to differential privacy constraints. These results highlight FedDPMZO’s practical potential for privacy-sensitive federated large-model training scenarios.

Ce résumé expose les affirmations des auteurs. BNTIC ne l’interprète pas comme une validation indépendante des résultats.