PaddlePaddle: An Open-Source Deep Learning Platform from Industrial Practice

doi:10.11871/jfdc.issn.2096.742X.2019.01.011

Abstract

Abstract:

[Objective] Deep learning is widely recognized as core technology driving the breakthroughs in artificial intelligence. Deep learning frameworks can be considered as the operating system in the era of artificial intelligence. PaddlePaddle, as the only fully-functioning open-source deep learning platform in China, is introduced comprehensively. [Methods] In this paper, a brief history of the deep learning frameworks is introduced, followed by an overview of PaddlePaddle, which is comprised of the core framework, toolkits and service platforms. After that, we elaborate on the core technologies of PaddlePaddle, including the front-end programming language, the modeling paradigm etc. Finally, the main innovations in PaddlePaddle are summarized. [Results] PaddlePaddle has been intensively tested in Baidu production for years, with unique features in supporting distributed training with ultra-large data and fast inference on server, mobile as well as edges. [Conclusions] The main innovations, research and development trends are discussed systematically.

Key words: PaddlePaddle, artificial intelligence, deep learning, deep learning framework

Yanjun Ma,Dianhai Yu,Tian Wu,Haifeng Wang. PaddlePaddle: An Open-Source Deep Learning Platform from Industrial Practice[J]. Frontiers of Data and Computing, 2019, 1(1): 105-115.

Figures/Tables 5

Fig. 1

Fig. 2

Fig. 3

Fig. 4

Table 1

References 22

[1]	Bergstra J, Breuleux O, Bastien F, et al. Theano: a CPU and GPU math expression compiler [C]. Proceedings of the Python for scientific computing conference (SciPy). 2010,4(3).
[2]	Jia Y, Shelhamer E, Donahue J, et al. Caffe: Convolutional architecture for fast feature embedding [C]. Proceedings of the 22nd ACM international conference on Multimedia. ACM, 2014: 675-678.
[3]	Abadi M, Barham P, Chen J, et al. Tensorflow: A system for large-scale machine learning [C]. 12th {USENIX} Symposium on Operating Systems Design and Implementation({OSDI} 16). 2016: 265-283.
[4]	Paszke A, Gross S, Chintala S , et al. Pytorch: Tensors and dynamic neural networks in python with strong gpu acceleration[J]. PyTorch: Tensors and dynamic neural networks in Python with strong GPU acceleration, 2017,6.
[5]	飞桨:源于产业实践的开源深度学习平台[EB/OL]. https://www.paddlepaddle.org.cn/.
[6]	Yu Sun, Shuohuan Wang, Yukun Li, Shikun Feng, Xuyi Chen, Han Zhang, Xin Tian, Danxiang Zhu, Hao Tian, Hua Wu . Ernie: Enhanced representation through knowledge integration. arXiv:1904.09223, 2019.
[7]	Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova . Bert: Pre-training of deep bidirectional transformers for language understanding . arXiv preprint arXiv:1810.04805, 2018.
[8]	Mnih V, Kavukcuoglu K, Silver D , et al. Human-level control through deep reinforcement learning[J]. Nature, 2015,518(7540):529.
[9]	Lillicrap T P, Hunt J J, Pritzel A , et al. Continuous control with deep reinforcement learning[J]. arXiv preprint arXiv:1509.02971, 2015.
[10]	Schulman J, Wolski F, Dhariwal P , et al. Proximal policy optimization algorithms[J]. arXiv preprint arXiv:1707.06347, 2017.
[11]	Espeholt L, Soyer H, Munos R , et al. Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures[J]. arXiv preprint arXiv:1802.01561, 2018.
[12]	Babaeizadeh M, Frosio I, Tyree S , et al. GA3C: GPU-based A3C for deep reinforcement learning[J]. CoRR abs/1611.06256, 2016.
[13]	OpenAI Baselines: ACKTR & A2C[EB/OL] https://openai.com/blog/baselines-acktr-a2c/.
[14]	XBYAK - X86, X64 JIT ASSEMBLER[EB/OL] http://herumi.in.coocan.jp/soft/xbyak_e.html.
[15]	OpenBLAS: An optimized BLAS library[EB/OL] https://www.openblas.net/.
[16]	Intel(R) Math Kernel Librray (Intel(R) MKL) [EB/OL] https://software.intel.com/en-us/mkl.
[17]	Intel(R) Math Kernel Library for Deep Neural Networks (Intel(R) MKL-DNN) [EB/OL] https://github.com/intel/mkl-dnn.
[18]	nGraph Compiler stack (Beta) [EB/OL] https://github.com/NervanaSystems/ngraph.
[19]	NVIDIA cuBLAS, Dense Library Algebra on GPUs[EB/OL] https://developer.nvidia.com/cublas.
[20]	NVIDIA CUDA(R) Deep Neural Network library (cuDNN)[ EB/OL] https://developer.nvidia.com/cudnn.
[21]	Yu Sun, Shuohuan Wang, Yukun Li, Shikun Feng, Hao Tian, Hua Wu, Haifeng Wang . ERNIE 2.0: A Continual Pre-training Framework for Language Understanding. arXiv preprint arXiv:1907.12412v1, 2019.
[22]	Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, Quoc V . Le. XLNet: Generalized Autoregressive Pretraining for Language Understanding. arXiv preprint arXiv:1906.08237, 2019.

领域	模型名称	对标开源框架	Paddle Fluid 1.5.0	对标开源框架
智能视觉（CV）	SE-ResNeXt50	PyTorch 1.1.0	168.33 images/s	163.13 images/s
	Mask-RCNN	PyTorch 1.1.0	3.81 images/s	3.24 images/s
	YOLOv3	MXNet 1.4.1	29.90 images/s	18.58 images/s
	DeepLab V3+	TensorFlow 1.12.0	13.70 images/s	6.40 images/s
	CycleGAN	TensorFlow 1.12.0	7.51 images/s	6.45 images/s
智能文本处理（NLP）	BERT	TensorFlow 1.12.0	129.41 sentences/s	109.44 sentences/s
智能文本处理（NLP）	Transformer	TensorFlow 1.12.0	19927 tokens/s	19456 tokens/s