联邦学习全局模型和个性化模型的现状与展望

doi:10.11871/jfdc.issn.2096-742X.2024.01.011

数据与计算发展前沿 ›› 2024, Vol. 6 ›› Issue (1): 113-124.

CSTR: 32002.14.jfdc.CN10-1649/TP.2024.01.011

doi: 10.11871/jfdc.issn.2096-742X.2024.01.011

联邦学习全局模型和个性化模型的现状与展望

修涵文^1,²(),李贺^1,²,曹荣强^1,^2,^*(),万萌^1,²,李凯^1,²,王彦棡^1,²

1.中国科学院计算机网络信息中心，北京 100083
2.中国科学院大学，计算机科学与技术学院，北京 100049

收稿日期:2022-11-28 出版日期:2024-02-20 发布日期:2024-02-21
通讯作者: * 曹荣强（E-mail: caorq@cnic.cn）
作者简介:修涵文，中国科学院计算机网络信息中心，硕士研究生，主要研究方向为联邦机器学习。
本文承担主要工作是整理数据和论文撰写。
XIU Hanwen is a master’s student at Computer Network Information Center, Chinese Academy of Sciences. Her main research interest is in the area of federated learning.
In this paper, she is mainly responsible for data sorting and paper writing.
E-mail: hwxiu@cnic.cn|曹荣强，中国科学院计算机网络信息中心，副研究员，主要研究方向为人工智能平台。
在本文中负责整体规划、论文指导。
CAO Rongqiang is an associate researcher at Computer Network Information Center, Chinese Academy of Sciences. His main research direction is artificial intelligence platforms.
In this paper, he is responsible for overall planning and paper guidance.
E-mail: caorq@cnic.cn
基金资助:
国家重点研发计划“人工智能算力算法数据一体化开放服务平台建设”(2020AAA0105202)

Global Model and Personalized Model of Federated Learning:Status and Prospect

XIU Hanwen^1,²(),LI He^1,²,CAO Rongqiang^1,^2,^*(),WAN Meng^1,²,LI Kai^1,²,WANG Yangang^1,²

1. Computer Network Information Center, Chinese Academy of Sciences, Beijing 100083, China
2. School of Computer Science and Technology, Chinese Academy of Sciences, Beijing 100049, China

Received:2022-11-28 Online:2024-02-20 Published:2024-02-21

摘要/Abstract

摘要：

【目的】联邦学习是目前的研究热点，本文从模型架构角度出发综述近年来联邦学习方法的研究和进展。【文献范围】本文采用关键词检索和引文二次检索的方法从国际计算机类期刊、会议中收集论文。【方法】在简单讨论联邦学习的定义、架构以及三种异质性问题的基础上，从模型架构角度出发，将联邦学习算法分为学习全局模型和学习个性化模型两大类，进一步讨论两大类别中的联邦学习方法所用数据集、对异构问题的解决以及方法优缺点。【结果】现有的联邦学习方法，可以学习泛化性能强大的全局模型，也可以学习个性化的局部模型。目前研究人员对数据异构问题的关注多于设备异构问题，在测试时所用数据集通常为常规机器学习数据集。【结论】联邦学习领域发展迅速，但仍存在异构问题研究不足、基准测试不成熟的问题，相信未来会有更多在真实场景中针对联邦异构问题的解决方案。

关键词: 联邦学习, 个性化模型, 全局模型, 异构问题

Abstract:

[Objective] Federated learning is a current research hotspot. In this paper, we review the research and progress of federated learning methods in recent years from the perspective of the model architecture. [Coverage] This paper uses keyword search and citation secondary search to collect papers from international journals and conferences on computer. [Methods] Based on a brief discussion of the definition, architecture, and three heterogeneity problems of federated learning, the federated learning algorithms are divided into two categories, learning global models and learning personalized models. We further discuss the federated learning methods in the two categories including the datasets used, the solution to heterogeneous problems, and the advantages and disadvantages of each method. [Results] Existing federated learning methods can both learn global models of strong generalization performance and personalized local models. Researchers are more concerned with data heterogeneous problems than device heterogeneous problems, and the datasets used in testing are usually conventional machine learning datasets. [Conclusions] The field of federated learning is developing rapidly, but there are still some problems of insufficient research on heterogeneous problems and immature benchmarking. It is believed that there will be more solutions to the problem of federation heterogeneity in real scenarios in the future.

Key words: federated learning, personalized model, global model, heterogeneity problem

修涵文, 李贺, 曹荣强, 万萌, 李凯, 王彦棡. 联邦学习全局模型和个性化模型的现状与展望[J]. 数据与计算发展前沿, 2024, 6(1): 113-124.

XIU Hanwen, LI He, CAO Rongqiang, WAN Meng, LI Kai, WANG Yangang. Global Model and Personalized Model of Federated Learning:Status and Prospect[J]. Frontiers of Data and Computing, 2024, 6(1): 113-124, https://cstr.cn/32002.14.jfdc.CN10-1649/TP.2024.01.011.

图/表 4

图1

表1

图2

表2

参考文献 52

[1]	MEHMOOD A, NATGUNANATHAN I, XIANG Y, et al. Protection of big data privacy[J]. IEEE access, 2016, 4: 1821-1834. doi: 10.1109/ACCESS.2016.2558446
[2]	VOIGT P, VON DEM BUSSCHE A. The eugeneral data protection regulation (gdpr)[J]. A Practical Guide, 1st Ed., Cham: Springer International Publishing, 2017, 10(3152676): 10-5555.
[3]	MCMAHAN B, MOORE E, RAMAGE D, et al. Communication-efficient learning of deep networks from decentralized data[C]// Artificial intelligence and statistics, PMLR, 2017: 1273-1282.
[4]	LI W, MILLETARÌ F, XU D, et al. Privacy-preserving federated brain tumour segmentation[C]// International workshop on machine learning in medical imaging. Springer, 2019: 133-141.
[5]	LLIU Y, HUANG A, LUO Y, et al. Fedvision: An online visual object detection platform powered by federated learning[C]// Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 34, 2020: 13172-13179.
[6]	DIMITRIADIS D, KEN’ICHI KUMATANI, GMYR R, et al. A Federated Approach in Training Acoustic Models[C]// Interspeech, 2020: 981-985.
[7]	LI M, ANDERSEN D G, PARK J W, et al. Scaling distributed machine learning with the parameter server[C]// The 11th USENIX Symposium on Operating Systems Design and Implementation[S.l.:s.n], 2014: 583-598.
[8]	DAI W, KUMAR A, WEI J, et al. High-performance distributed ML at scale through parameter server consistency models[C]// Proceedings of the AAAI Conference on Artificial Intelligence, 2015: 79-87.
[9]	NIU F, RECHT B, RE C, et al. HOGWILD! a lock-free approach to parallelizing stochastic gradient descent[C]// Proceedings of the 24th International Conference on Neural Information Processing Systems, 2011: 693-701.
[10]	HO Q, CIPAR J, CUI H, et al. More effective distributed ml via a stale synchronous parallel parameter server[J]. Advances in neural information processing systems, 2013, 26: 1223-1231.
[11]	WU Q, HE K, CHEN X. Personalized federated learning for intelligent IoT applications: A cloud-edge based framework[J]. IEEE Open Journal of the Computer Society, 2020, 1: 35-44. doi: 10.1109/OJCS
[12]	LI T, SAHU A K, ZAHEER M, et al. Federated optimization in heterogeneous networks[J]. Proceedings of Machine Learning and Systems, 2020, 2: 429-450.
[13]	LI Q, DIAO Y, CHEN Q, et al. Federated learning on non-iid data silos: An experimental study[C]// 2022 IEEE 38th International Conference on Data Engineering (ICDE), IEEE, 2022: 965-978.
[14]	KULKARNI V, KULKARNI M, PANT A. Survey of personalization techniques for federated learning[C]// 2020 Fourth World Conference on Smart Trends in Systems, Security and Sustainability (WorldS4), IEEE, 2020: 794-797.
[15]	HUANG Y, CHU L, ZHOU Z, et al. Personalized Cross-Silo Federated Learning on Non-IID Data[C]// AAAI, 2021: 7865-7873.
[16]	ZHU Z, HONG J, ZHOU J. Data-free knowledge distillation for heterogeneous federated learning[C]// International Conference on Machine Learning, PMLR, 2021: 12878-12889.
[17]	GO A, BHAYANI R, HUANG L. Twitter sentiment classification using distant supervision[J]. CS224N project report, Stanford, 2009, 1(12): 2009.
[18]	NETZER Y, WANG T, COATES A, et al. Reading digits in natural images with unsupervised feature learning[C]// NIPS Workshop on Deep Learning and Unsupervised Feature Learning, 2011: 1-9.
[19]	CALDAS S, DUDDU S M K, WU P, et al. Leaf: A benchmark for federated settings[J]. arXiv preprint arXiv:1812.01097, 2018.
[20]	KRIZHEVSKY A, SUTSKEVER I, HINTON G E. Imagenet classification with deep convolutional neural networks[J]. Communications of the ACM, 2017, 60(6): 84-90. doi: 10.1145/3065386
[21]	ZHANG X, ZHAO J, LECUN Y. Character-level convolutional networks for text classification[C]// Proceedings of the 28th International Conference on Neural Information Processing Systems-Volume 1, 2015: 649-657.
[22]	SOCHER R, PERELYGIN A, WU J, et al. Recursive deep models for semantic compositionality over a sentiment treebank[C]// Proceedings of the 2013 conference on empirical methods in natural language processing, 2013: 1631-1642.
[23]	LIU Z, LUO P, WANG X, et al. Deep learning face attributes in the wild[C]// Proceedings of the IEEE international conference on computer vision, 2015: 3730-3738.
[24]	LECUN Y, BOTTOU L, BENGIO Y, et al. Gradient-based learning applied to document recognition[J]. Proceedings of the IEEE, 1998, 86(11): 2278-2324. doi: 10.1109/5.726791
[25]	COHEN G, AFSHAR S, TAPSON J, et al. EMNIST: Extending MNIST to handwritten letters[C]// 2017 international joint conference on neural networks (IJCNN), IEEE, 2017: 2921-2926.
[26]	CALDAS S, DUDDU S M K, WU P, et al. Leaf: A benchmark for federated settings[J]. arXiv preprint arXiv: 1812.01097, 2018.
[27]	POURANSARI H, GHILI S. Tiny imagenet visual recognition challenge[Z]. CS231 N course, Stan-ford Univ., Stanford, CA, USA, 2014.
[28]	LI T, SAHU A K, ZAHEER M, et al. Federated optimization in heterogeneous networks[J]. Proceedings of Machine Learning and Systems, 2020, 2: 429-450.
[29]	LI Q, HE B, SONG D. Model-Contrastive Federated Learning[C]// 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), IEEE, 2021: 10708-10717.
[30]	LI T, HU S, BEIRAMI A, et al. Ditto: Fair and robust federated learning through personalization[C]// International Conference on Machine Learning, PMLR, 2021: 6357-6368.
[31]	KARIMIREDDY S P, KALE S, MOHRI M, et al. Scaffold: Stochastic controlled averaging for federated learning[C]// International Conference on Machine Learning, PMLR, 2020: 5132-5143.
[32]	LI D, WANG J. Fedmd: Heterogenous federated learning via model distillation[J]. arXiv preprint arXiv:1910.03581, 2019.
[33]	HAO W, EL-KHAMY M, LEE J, et al. Towards fair federated learning with zero-shot data augmentation[C]// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021: 3310-3319.
[34]	TANG Z, ZHANG Y, SHI S, et al. Virtual Homogeneity Learning: Defending against Data Heterogeneity in Federated Learning[C]// Proceedings of the 39th International Conference on Machine Learning, PMLR, 2022: 21111-21132.
[35]	GONG M, ZHANG K, LIU T, et al. Domain adaptation with conditional transferable components[C]// International conference on machine learning, PMLR, 2016: 2839-2848.
[36]	WU H, WANG P. Fast-convergent federated learning with adaptive weighting[J]. IEEE Transactions on Cognitive Communications and Networking, 2021, 7(4): 1078-1088. doi: 10.1109/TCCN.2021.3084406
[37]	CHEN H Y, CHAO W L. FedBE: Making Bayesian Model Ensemble Applicable to Federated Learning[C]// International Conference on Learning Representations, 2020: 1-21.
[38]	YUROCHKIN M, AGARWAL M, GHOSH S, et al. Bayesian nonparametric federated learning of neural networks[C]// International Conference on Machine Learning, PMLR, 2019: 7252-7261.
[39]	THIBAUX R, JORDAN M I. Hierarchical beta processes and the Indian buffet process[C]// Artificial intelligence and statistics, PMLR, 2007: 564-571.
[40]	FALLAH A, MOKHTARI A, OZDAGLAR A. Personalized federated learning with theoretical guarantees: A model-agnostic meta-learning approach[J]. Advances in Neural Information Processing Systems, 2020, 33: 3557-3568.
[41]	FINN C, ABBEEL P, LEVINE S. Model-agnostic meta-learning for fast adaptation of deep networks[C]// International conference on machine learning, PMLR, 2017: 1126-1135.
[42]	ACAR D A E, ZHAO Y, ZHU R, et al. Debiasing model updates for improving personalized federated training[C]// International Conference on Machine Learning, PMLR, 2021: 21-31.
[43]	LI T, SANJABI M, BEIRAMI A, et al. Fair Resource Allocation in Federated Learning[C]// International Conference on Learning Representations, 2019: 1-27.
[44]	KHODAK M, BALCAN M F, TALWALKAR A. Adaptive gradient-based meta-learning methods[C]// Proceedings of the 33rd International Conference on Neural Information Processing Systems, 2019: 5917-5928.
[45]	SMITH V, CHIANG C K, SANJABI M, et al. Federated multi-task learning[C]// Proceedings of the 31st International Conference on Neural Information Processing Systems, 2017: 4427-4437.
[46]	GHOSH A, HONG J, YIN D, et al. Robust federated learning in a heterogeneous environment[J]. arXiv preprint arXiv:1906.06629, 2019.
[47]	HARTIGAN J, WONG M. A K-Means Clustering Algorithm[J]. Journal of the Royal Statistical Society, Series C (Applied Statistics), 1979, 28(1): 100-108.
[48]	SATTLER F, MÜLLER K R, SAMEK W. Clustered federated learning: Model-agnostic distributed multitask optimization under privacy constraints[J]. IEEE transactions on neural networks and learning systems, 2020, 32(8): 3710-3722. doi: 10.1109/TNNLS.2020.3015958
[49]	GHOSH A, CHUNG J, YIN D, et al. An efficient framework for clustered federated learning[J]. Advances in Neural Information Processing Systems, 2020, 33: 19586-19597.
[50]	JEONG E, OH S, KIM H, et al. Communication-efficient on-device machine learning: Federated distillation and augmentation under non-iid private data[J]. arXiv preprint arXiv: 1811.11479, 2018.
[51]	LIN T, KONG L, STICH S U, et al. Ensemble distillation for robust model fusion in federated learning[J]. Advances in Neural Information Processing Systems, 2020, 33: 2351-2363.
[52]	LAI F, DAI Y, SINGAPURAM S, et al. Fedscale: Benchmarking model and system performance of federated learning at scale[C]// International Conference on Machine Learning, PMLR, 2022: 11814-11827.

	设备异构	数据异构	模型异构
含义	客户端设备在存储、计算和通信能力方面存在差异。	客户端本地数据非独立同分布	客户端根据其应用场景或优化目标所需要的模型不一致。
问题	训练时间不一致，训练时间延长	模型不稳定，更差的模型性能	灵活性和隐私保护受限

类别	方法	算法名称	年份	数据异构	设备异构	数据集						优点	缺点
类别	方法	算法名称	年份	数据异构	设备异构	MN-IST	EM-NIST	FMN-IST	CIFA-R-10	CIFAR-100	其他	优点	缺点
单一全局模型	修改客户端局部目标	FedProx	2020	√	√	√	√	√			Shakespeare^[3]、Sent140^[17]	容易实现，有效地解决本地更新时的客户端偏移	只有全局模型，模型灵活性受限
		Scaffold	2020	√			√
		MOON	2021	√					√	√	Tiny-Imagenet^[18]
	数据扩充	FED-ZDA	2021	√		√		√	√			直接“纠正”数据异质性，容易实现	容易面临隐私泄露的风险，需要有代表性的数据集
	数据扩充	VHL	2022	√				√	√	√	SVHN^[19]	直接“纠正”数据异质性，容易实现	容易面临隐私泄露的风险，需要有代表性的数据集
	优化模型聚合	FEDBE	2020	√	√				√	√	Tiny-Imagenet	容易实现，可以解决联邦学习公平性问题	只有全局模型，模型灵活性受限
	优化模型聚合	PFNM	2019	√		√			√			容易实现，可以解决联邦学习公平性问题	只有全局模型，模型灵活性受限
	联邦元学习	Per-FedAvg	2020	√		√			√			可以快速学习在少样本上的新任务	计算二阶梯度的计算开销大，实现的复杂度较高
	联邦元学习	PFL	2021	√					√	√		可以快速学习在少样本上的新任务	计算二阶梯度的计算开销大，实现的复杂度较高
个性化模型	联邦多任务学习	MOCHA	2017	√	√				√	√	Shakespeare	相似的客户端可以进行更多的协作，共享信息提高学习能力	对客户端的数据质量敏感
	联邦多任务学习	FedAMP	2021	√		√		√	√			相似的客户端可以进行更多的协作，共享信息提高学习能力	对客户端的数据质量敏感
	联邦聚类	CFL	2020	√		√			√			适合客户端分组情况，容易剔除异常客户端	通常需要在联邦学习过程中额外进行迭代聚类，通信和计算开销大
	联邦聚类	IFCA	2020	√		√	√	√	√			适合客户端分组情况，容易剔除异常客户端	通常需要在联邦学习过程中额外进行迭代聚类，通信和计算开销大
	联邦知识蒸馏	FedDF	2020	√					√	√	ImageNet^[20]AG News^[21]、SST2^[22]	模型结构的灵活性高，客户端模型个性化强。	难以决定最佳的架构设计
	联邦知识蒸馏	FEDGEN	2021	√	√	√	√				CELEBA^[23]	模型结构的灵活性高，客户端模型个性化强。	难以决定最佳的架构设计

联邦学习全局模型和个性化模型的现状与展望

Global Model and Personalized Model of Federated Learning:Status and Prospect

RichHTML

PDF

可视化

摘要/Abstract

引用本文

使用本文

图/表 4

参考文献 52

相关文章 2

编辑推荐

Metrics

本文评价

[1]	赵鑫博,代闯闯,陆忠华. 一种改进的BMUF训练框架及联邦学习系统实现[J]. 数据与计算发展前沿, 2022, 4(6): 105-117.
[2]	陈磊,刘文懋. 合规视角下的数据安全技术前沿与应用[J]. 数据与计算发展前沿, 2021, 3(3): 19-31.