Spectrum Resource Allocation of Vehicle Edge Computing Network Based on Proximal Policy Optimization Algorithm

doi:10.11871/jfdc.issn.2096-742X.2022.04.014

Abstract

Abstract:

[Objective] In the edge computing of vehicles, a reasonable allocation of spectrum resources is of great significance to improving the quality of vehicle communication. The scarcity of spectrum resources is a crucial issue that affects the quality of vehicle communication. The high mobility of vehicles and the difficulty of accurately collecting channel state information at the base station are challenging for spectrum resource allocation. [Methods] In view of the above problems, the optimization goal is set to the transmission rate of the vehicle-to-vehicle (V2V) link and the capacity of the vehicle-to-infrastructure (V2I) link. This paper proposed a optimization based on the Proximal Policy Optimization (PPO) reinforcement learning algorithm for multi-agent dynamic allocation of spectrum resources. [Results] Multiple V2V links sharing the spectrum resources occupied by V2I links can alleviate the problem of spectrum scarcity. Thus, this problem is further formulated as a Markov Decision Process, and the state, action, and reward are designed to optimize the spectrum allocation strategy. [Conclusions] The simulation results show that, compared with the baseline algorithm, the optimization scheme based on the PPO algorithm proposed in this paper has better performance in terms of channel transmission rate and vehicle information transmission success rate.

Key words: vehicle edge computing, spectrum allocation, Markov Decision Process, Proximal Policy Optimization

ZHAO Jianan,HU Xiaohui,DU Xinxin. Spectrum Resource Allocation of Vehicle Edge Computing Network Based on Proximal Policy Optimization Algorithm[J]. Frontiers of Data and Computing, 2022, 4(4): 142-155, https://cstr.cn/21.86101.2/jfdc.2096-742X.2022.04.014.

Figures/Tables 13

Fig.1

Fig.2

Fig.3

1初始化Actor网络

π o l d / n e w (θ)

和critic网络

V (θ)

2初始化环境,生成车辆和链接和经验池D

3将Actor网络参数赋予给用来采样环境的参数

θ a o l d ← θ a n e w

;

4for iteration= 1,2,... do

5 更新车辆位置以及大规模阴影衰落

(p l v s d v)

;

6 Reset负载

B v = B

和剩余时间

T v = T

v ∈ V

;

7 for each step t do

8 观察初始车辆环境状态

s t

;

9 将状态

s t

输入到Actor-new网络得到动作

a t

;

10 构造正态分布,在此正态分布中对动作的分布采样;

11 在执行一个操作和下一个网络的输入状态

s t + 1

后获得奖励

r t

;

12 存储

[s t, a t, r t, · · ·]

于经验池D中;

13 将

s t + 1

输入到Actor-new网络中;

14 将最终状态输入Critic网络中,根据(18)计算折扣奖励;

15 if更新步骤等于经验池容量时

16 将所有的状态组合输入到Critic网络中并计算优势函数(20) ;

17 计算损失函数(21),反向传播更新critic网络;

18 将所有的状态组合输入Actor-new和Actor-old网络中;

19 将所有的动作组合输入到Actor网络得到正太分布N1和N2得到动作组合概率Prob1和Prob2;

20 计算损失函数

C_l o s s

(21),反向传播更新critic网络;

21 更新Actor-new网络参数

π n e w (θ)

;

22 End

23 用Actor-new网络

π n e w (θ)

更新

π o l d (θ)

24 End

25 End

Table 1

Fig.4

Fig.5

Fig.6

Fig.7

Fig.8

Fig.9

Fig.10

Fig.11

References 22

[1]	Rasheed A, Chong P H J, Ho I W H, et al. An overview of mobile edge computing: Architecture, technology and direction[J]. KSII Transactions on Internet and Inform-ation Systems (TIIS), 2019, 13(10): 4849-4864.
[2]	Chen C, Ding Y, Xie X, et al. TrajCompressor: An online map-matching-based trajectory compression framework leveraging vehicle heading direction and change[J]. IEEE Transactions on Intelligent Transportation Systems, 2019, 21(5): 2012-2028. doi: 10.1109/TITS.2019.2910591
[3]	Xiao Z, Shen X, Zeng F, et al. Spectrum resource sharing in heterogeneous vehicular networks: A noncooperative game-theoretic approach with correlated equilibrium[J]. IEEE Transactions on Vehicular Technology, 2018, 67 (10): 9449-9458. doi: 10.1109/TVT.2018.2855683
[4]	Luo Q, Li C, Luan T H, et al. Optimal utility of vehicles in LTE-V scenario: An immune clone-based spectrum allocation approach[J]. IEEE Transactions on Intelligent Transportation Systems, 2018, 20(5): 1942-1953. doi: 10.1109/TITS.2018.2850311
[5]	Li R, Jin L. Improved cuckoo algorithm for spectrum allocation in cognitive vehicular network[C]// 2018 5th International Conference on Systems and Informatics (ICSAI), IEEE, 2018: 828-833.
[6]	Li Q, Qi W, Guo L. A Prediction-Based Spectrum Al-location Scheme for Two-Layer Cellular Vehicular Netw-orks[C]// 2020 IEEE/CIC International Conference on Communications in China (ICCC), IEEE, 2020: 858-863.
[7]	Liu J, Wang S, Wang J, et al. A task oriented computation offloading algorithm for intelligent vehicle network with mobile edge computing[J]. IEEE Access, 2019, 7: 180491-180502. doi: 10.1109/ACCESS.2019.2958883
[8]	Zia K, Javed N, Sial M N, et al. A distributed multi-agent RL-based autonomous spectrum allocation scheme in D2D enabled multi-tier HetNets[J]. IEEE Access, 2019, 7: 6733-6745. doi: 10.1109/ACCESS.2018.2890210
[9]	Liang L, Ye H, Li G Y. Spectrum sharing in vehicular networks based on multi-agent reinforcement learning[J]. IEEE Journal on Selected Areas in Communications, 2019, 37(10): 2282-2292. doi: 10.1109/JSAC.2019.2933962
[10]	Yang H, Xie X, Kadoch M. Intelligent resource manag-ement based on reinforcement learning for ultra-reliable and low-latency IoV communication networks[J]. IEEE Transactions on Vehicular Technology, 2019, 68(5): 4157-4169. doi: 10.1109/TVT.2018.2890686
[11]	Prasad R. OFDM for wireless communications systems[M]. Artech House, 2004:83-116
[12]	Li B, He D, Feng Y, et al. Spectrum resource alloca-tion scheme for alarm information delivery in V2V communication[C]// 2018 IEEE 88th Vehicular Techn-ology Conference (VTC-Fall), IEEE, 2018: 1-5.
[13]	Wang Y, He H, Tan X. Truly proximal policy optim-ization[C]// Uncertainty in Artificial Intelligence, PMLR, 2020: 113-122.
[14]	Schulman J, Wolski F, Dhariwal P, et al. Proximal policy optimization algorithms[J]. arXiv preprint arXiv: 1707. 06347, 2017.
[15]	LeCun Y, Bengio Y, Hinton G. Deep learning[J]. Nature, 2015, 521(7553): 436-444. doi: 10.1038/nature14539
[16]	Wang X, Mao S, Gong M X. An overview of 3GPP cell-ular vehicle-to-everything standards[J]. GetMobile: Mobile Computing and Communications, 2017, 21(3): 19-25.
[17]	Chen S, Hu J, Shi Y, et al. Vehicle-to-everything (V2X) services supported by LTE-based systems and 5G[J]. IE-EE Communications Standards Magazine, 2017, 1(2): 70-76.
[18]	R1-165704. WF on SLS evaluation assumptions for eV2X[S]. 3GPP TSG RAN WG 1 Meeting #85, 2016.
[19]	Marsch P, Da Silva I, Bulakci O, et al. 5G radio access network architecture: Design guidelines and key consid-erations[J]. IEEE Communications Magazine, 2016, 54 (11): 24-32.
[20]	3rd Generation Partnership Project. Technical specific-ation group radio access network[S]. Study on LTE-based V2X Services (Release14). 3GPP TR 36.885 V14.0.0, 2016.
[21]	Li S, Hu X, Du Y. Deep Reinforcement Learning and Game Theory for Computation Offloading in Dynamic Edge Computing Markets[J]. IEEE Access, 2021, 9: 121456-121466. doi: 10.1109/ACCESS.2021.3109132
[22]	Li S, Hu X, Du Y. Deep Reinforcement Learning for Computation Offloading and Resource Allocation in Unm-anned-Aerial-Vehicle Assisted Edge Computing[J]. Sen-sors, 2021, 21(19): 6499.

参数名称	参数设置
V2I链接的数量	4
V2I链接的数量	4
载频	2GHZ
带宽	4MHZ
BS天线高度	25m
BS信道增益	8dBi
BS接受噪声指数	5dB
车辆天线高度	1.5m
车辆信道增益	3dBi
车辆接受噪声指数	9dB
车辆速度	36km/h
车辆衰落与移动模型	[20]中A.1.2的城市情况
V2I传输功率	23dBm
V2I传输功率	[23,10,5,-100]dBm
噪声功率	-114dBm
V2I路径损耗	LOS in WINNER+B1
V2V与V2I链路信道模型	[9]中的Table Ⅱ
阴影衰落标准差V2I/V2V	8 dB/4 dB
小规模衰落更新频率	1ms
路损与阴影更新频率	A.1.4 in[20] every 100ms
数据包大小B	[1,1.5,2...] ×1060 bytes
传输数据包时间约束 T	100ms