| 引用本文: | 刘建春,梁文艺,徐宏力,马千飘,黄刘生.基于块级多输出和知识自蒸馏的高效联邦学习框架.软件学报,2026,37(3):1357-1373 |
| |
|
| |
|
|
| 本文已被：浏览 776次 下载 2054次 |
码上扫一扫！ |
|
|
| 基于块级多输出和知识自蒸馏的高效联邦学习框架 |
|
刘建春1,2, 梁文艺1,2, 徐宏力1,2, 马千飘3, 黄刘生1,2
|
|
1.中国科学技术大学 计算机科学与技术学院, 安徽 合肥 230026;2.中国科学技术大学苏州高等研究院, 江苏 苏州 215004;3.南京理工大学 计算机科学与工程学院, 江苏 南京 210094
|
|
| 摘要: |
| 联邦学习 (federated learning, FL)是一种分布式模型训练框架, 允许多个客户端在边缘计算(edge computing, EC)环境中协同训练全局模型, 同时保护客户端的本地数据隐私. 然而, 在边缘网络中进行联邦学习训练时, 常常面临资源受限和数据异构(或称非独立同分布数据)的问题, 这会导致模型训练性能显著下降. 为了应对这些挑战, 提出了一种高效的联邦学习框架——FedAlt, 以提升边缘网络中模型训练的性能 (如测试精度)和减少资源开销. FedAlt 在经典联邦学习算法FedAvg 的基础上引入了块级多输出和知识自蒸馏技术, 使客户端在本地训练时能够更有效地吸收模型表征层信息, 从而缓解非独立同分布数据对模型训练的负面影响. 具体而言, 将模型划分为多个连续的模型块, 服务器在每个全局训练轮次开始时仅向客户端发送前部分的全局模型块, 从而减少通信开销. 然后, 客户端将全局模型和本地模型进行组合, 并利用知识自蒸馏技术吸收模型表征层的信息, 以应对数据异构带来的挑战. 此外, 考虑到通信开销随传输的模型块数量增加而增加, 分别在服务器和客户端设计了自适应算法, 即服务器分发模型块算法和客户端块级多输出正则化算法, 根据客户端的数据分布、计算能力和通信能力来动态地调整服务器分发的模型块数量. 大量实验结果表明, 与现有方法相比, FedAlt 在有限的通信带宽条件下, 可以提升约 2.64%的平均测试精度. |
| 关键词: 边缘计算 联邦学习 数据异构 资源有限 |
| DOI：10.13328/j.cnki.jos.007466 |
| 分类号:TP18 |
| 基金项目:国家自然科学基金 (62132019); 江苏省自然科学基金 (BK20230275); 安徽省自然科学基金 (2408085QF185) |
|
| Efficient Federated Learning Framework with Block-wise Multi-output and Knowledge Self-distillation |
|
LIU Jian-Chun1,2, LIANG Wen-Yi1,2, XU Hong-Li1,2, MA Qian-Piao3, HUANG Liu-Sheng1,2
|
|
1.School of Computer Science and Technology, University of Science and Technology of China, Hefei 230026, China;2.Suzhou Institute for Advanced Research, University of Science and Technology of China, Suzhou 215004, China;3.School of Computer Science and Engineering, Nanjing University of Science & Technology, Nanjing 210094, China
|
| Abstract: |
| Federated learning (FL) is a distributed model training framework that allows multiple clients to collaboratively train a global model in an edge computing (EC) environment while preserving the privacy of clients’ local data. However, federated learning in edge networks often faces challenges such as resource constraints and data heterogeneity, also known as non-independent and identically distributed (non-IID) data, which significantly degrade model training performance. To address these challenges, this study proposes an efficient federated learning framework—FedAlt, aiming to enhance model training performance (e.g., test accuracy) in edge networks while reducing resource consumption. FedAlt builds upon the classic federated learning algorithm FedAvg by incorporating block-wise multi-output and self-knowledge distillation techniques. These enhancements enable clients to more effectively absorb information from the model’s representational layers during local training, mitigating the negative impact of non-IID data on model training. Specifically, the model is divided into multiple consecutive blocks, and at the start of each global training round, the server sends only the initial blocks of the global model to the clients, reducing communication overhead. Clients then combine the global model with their local models and use self-knowledge distillation techniques to absorb information from the model’s representational layers, addressing data heterogeneity challenges. Moreover, considering that communication overhead increases with the number of transmitted model blocks, adaptive algorithms are designed for both the server and client sides: the model block distribution algorithm and the block-wise multi-output regularization algorithm. These algorithms dynamically adjust the number of distributed model blocks based on the client’s data distribution, computational capacity, and communication capabilities. Extensive experimental results show that, compared to existing methods, FedAlt improves average test accuracy by approximately 2.64% under limited communication bandwidth conditions. |
| Key words: edge computing federated learning data heterogeneity resource limitation |
|
|
|
|
