An Improved BMUF Training Framework and Implementation of Federated Learning System

doi:10.11871/jfdc.issn.2096-742X.2022.06.010

Abstract

Abstract:

[Objective] In the increasingly severe condition of privacy protection, federated learning is often used to solve problems such as "data islands" and "data privacy". However, the traditional federated learning architecture is limited by its centralized characteristic, which brings additional privacy risks and costs. The blockchain-based decentralized federated learning architecture has received more attention and research efforts due to its obvious advantages in application. [Methods] This paper improves the BMUF training framework to make better results in the scenarios of unbalanced and non-IID data distributions in federated learning. The differential privacy mechanism is added to the client's local training to protect local privacy. A byzantine detection and robust aggregation algorithm based on the global model update is proposed, which enables the aggregator to detect the byzantine clients in the system and complete the robust aggregation. [Results] Multiple experiments are conducted on the above three points. The experimental results show that the improved BMUF training framework has a better aggregation effect than the FedAvg algorithm in unbalanced and non-IID distribution scenarios. When the differential privacy mechanism is added to the client's local training, the model can still converge and obtain a higher accuracy rate. In the byzantine attack environment, the aggregator can effectively eliminate the byzantine clients and complete the robust aggregation. [Conclusions] This paper improves the BMUF training framework and implements a blockchain-based federated learning system that can effectively protect the local privacy of the client, resist byzantine attacks, and achieve efficient training of the model for different data distribution scenarios under a decentralized architecture.

Key words: blockchain, federated learning, BMUF framework, differential privacy, byzantine attack

ZHAO Xinbo,DAI Chuangchuang,LU Zhonghua. An Improved BMUF Training Framework and Implementation of Federated Learning System[J]. Frontiers of Data and Computing, 2022, 4(6): 105-117, https://cstr.cn/32002.14.jfdc.CN10-1649/TP.2022.06.010.

Figures/Tables 13

Fig.1

Fig.2

Fig.3

Fig.4

Fig.5

Table 1

Fig.6

Fig.7

Fig.8

Table 2

Fig.9

References 27

[1]	Roh Y, Heo G, Whang S E. A survey on data collection for machine learning: a big data-ai integration perspe-ctive[J]. IEEE Transactions on Knowledge and Data Engineering, 2019, 33(4): 1328-1347. doi: 10.1109/TKDE.2019.2946162
[2]	Krizhevsky A, Sutskever I, Hinton G E. Imagenet clas-sification with deep convolutional neural networks[J]. Advances in neural information processing systems, 2012, 25(2): 1097-1105.
[3]	McMahan H B, Moore E, Ramage D, et al. Federated learning of deep networks using model averaging[J]. arXiv preprint arXiv:1602.05629, 2016.
[4]	Ramanan P, Nakayama K. Baffle: Blockchain based agg-regator free federated learning[C]// 2020 IEEE Internati-onal Conference on Blockchain (Block-chain), IEEE, 2020: 72-81.
[5]	Kim H, Park J, Bennis M, et al. Blockchained on-device federated learning[J]. IEEE Communications Letters, 2019, 24(6): 1279-1283. doi: 10.1109/LCOMM.2019.2921755
[6]	McMahan B, Moore E, Ramage D, et al. Communication-efficient learning of deep networks from decentralized data[C]// Artificial intelligence and statistics, PMLR, 2017: 1273-1282.
[7]	Li Y, Chen C, Liu N, et al. A blockchain-based dece-ntralized federated learning framework with committee consensus[J]. IEEE Network, 2020, 35(1): 234-241.
[8]	Awan S, Li F, Luo B, et al. Poster: A reliable and acc-ountable privacy-preserving federated learning frame-work using the blockchain[C]// Proceedings of the 2019 ACM SIGSAC Conference on Computer and Communi-cations Security, 2019: 2561-2563.
[9]	Lugan S, Desbordes P, Brion E, et al. Secure architectures implementing trusted coalitions for blockchained distri-buted learning (TCLearn)[J]. IEEE Access, 2019, 7: 181789-181799. doi: 10.1109/ACCESS.2019.2959220
[10]	Zhu X, Li H, Yu Y. Blockchain-Based privacy preserving deep learning[C]// International Conference on Infor-mation Security and Cryptology, Springer, Cham, 2018: 370-383.
[11]	Chen X, Ji J, Luo C, et al. When machine learning meets blockchain: A decentralized, privacy-preserving and secure design[C]// 2018 IEEE international conference on big data (big data), IEEE, 2018: 1178-1187.
[12]	Lu Y, Huang X, Dai Y, et al. Blockchain and federated learning for privacy-preserved data sharing in industrial IoT[J]. IEEE Transactions on Industrial Informatics, 2019, 16(6): 4177-4186. doi: 10.1109/TII.2019.2942190
[13]	Liu Y, Peng J, Kang J, et al. A secure federated learning framework for 5G networks[J]. IEEE Wireless Comm-unications, 2020, 27(4): 24-31.
[14]	Chen L, Charles Z, Papailiopoulos D. Draco: Robust distributed training via redundant gradients[J]. arXiv preprint arXiv:1803.09877, 2018.
[15]	Guerraoui R, Rouault S. The hidden vulnerability of dis-tributed learning in byzantium[C]// International Confer-ence on Machine Learning, PMLR, 2018: 3521-3530.
[16]	Blanchard P, El Mhamdi E M, Guerraoui R, et al. Ma-chine learning with adversaries: Byzantine tolerant grad-ient descent[J]. Advances in Neural Information Proces-sing Systems, 2017, 30.
[17]	Muñoz-González L, Co K T, Lupu E C. Byzantine-robust federated machine learning through adaptive model averaging[J]. arXiv preprint arXi-v:1909.05125, 2019.
[18]	Yousefpour A, Shilov I, Sablayrolles A, et al. Opacus: User-friendly differential privacy library in PyTorch[J]. arXiv preprint arXiv:2109.12298, 2021.
[19]	Nakamoto S. Bitcoin: A peer-to-peer electronic cash system[J]. Decentralized Business Review, 2008: 21260.
[20]	工信部. 中国区块链技术和应用发展白皮书[R/OL]. [2016-10-18]. http://www.199it.com/archives/526865.html.
[21]	陈凯. 深度学习模型的高效训练算法研究[D]. 中国科学技术大学, 2016.
[22]	Chen K, Huo Q. Scalable training of deep learning mac-hines by incremental block training with intrablock parallel optimization and blockwise model-update filt-ering[C]// 2016 ieee international conference on acoustics, speech and signal processing (icassp), IEEE, 2016: 5880-5884.
[23]	Dwork C. Differential privacy: A survey of results[C]// International conference on theory and applications of models of computation. Springer, Berlin, Heidelberg, 2008: 1-19.
[24]	Dwork C, McSherry F, Nissim K, et al. Calibrating noise to sensitivity in private data analysis[C]// Theory of cryp-tography conference. Springer, Berlin, Heidelberg, 2006: 265-284.
[25]	McSherry F, Talwar K. Mechanism design via diffe-rential privacy[C]// 48th Annual IEEE Symposium on Foundations of Computer Science (FOCS'07), IEEE, 2007: 94-103.
[26]	Dwork C, Kenthapadi K, McSherry F, et al. Our data, ourselves: Privacy via distributed noise generation[C]// Annual international conference on the theory and applic-ations of cryptographic techniques, Springer, Berlin, Heidelberg, 2006: 486-503.
[27]	Lecun Y, Cortes C. The MNIST database of handwritten digits[J/OL]. 2010. http://yann.lecun.com/exdb/mnist/.

Layer(type)	Output Shape	Param
Conv2d-1	[-1,32,24,24]	832
MaxPool2d-2	[-1,32,12,12]	0
Conv2d-3	[-1,64,8,8]	51264
MaxPool2d-4	[-1,64,4,4]	0
Dropout-5	[-1,64,4,4]	0
Linear-6	[-1,512]	524800
Linear-7	[-1,256]	131328
Linear-8	[-1,10]	2570

参与方数量	隐私预算	Test Accuracy
10	ε=1	0.9225
	ε=2	0.9278
	ε=∞	0.9603
15	ε=1	0.9174
	ε=2	0.9235
	ε=∞	0.9578
20	ε=1	0.9141
	ε=2	0.9204
	ε=∞	0.9566