###
Journal of Software:2018.29(2):267-282

一种解决连续空间问题的真实在线自然梯度AC算法
朱斐,朱海军,刘全,陈冬火,伏玉琛
(苏州大学 计算机科学与技术学院, 江苏 苏州 215006;江苏省计算机信息处理技术重点实验室(苏州大学), 江苏 苏州 215006;符号计算与知识工程教育部重点实验室(吉林大学), 吉林 长春 130012;苏州大学 计算机科学与技术学院, 江苏 苏州 215006;符号计算与知识工程教育部重点实验室(吉林大学), 吉林 长春 130012;苏州大学 计算机科学与技术学院, 江苏 苏州 215006;常熟理工学院 计算机科学与工程学院, 江苏 常熟 215500)
True Online Natural Actor-Critic Algorithm for the Continuous Space Problem
ZHU Fei,ZHU Hai-Jun,LIU Quan,CHEN Dong-Huo,FU Yu-Chen
(School of Computer Science and Technology, Soochow University, Suzhou 215006, China;Provincial Key Laboratory for Computer Information Processing Technilogy(Soochow University), Suzhou 215006, China;Key Laboratory of Symbolic Computation and Knowledge Engineering of Ministry of Education (Jilin University), Changchun 130012, China;School of Computer Science and Technology, Soochow University, Suzhou 215006, China;Key Laboratory of Symbolic Computation and Knowledge Engineering of Ministry of Education(Jilin University), Changchun 130012, China;School of Computer Science and Technology, Soochow University, Suzhou 215006, China;School of Computer Science and Engineering, Changshu Institute of Technology, Changshu 215500, China)
Abstract
Chart / table
Reference
Similar Articles
Article :Browse 1631   Download 1508
Received:November 04, 2016    Revised:December 13, 2016
> 中文摘要: 策略梯度作为一种能够有效解决连续空间决策问题的方法得到了广泛研究,但由于在策略估计过程中存在较大方差,因此,基于策略梯度的方法往往受到样本利用率低、收敛速度慢等限制.针对该问题,在行动者-评论家(actor-critic,简称AC)算法框架下,提出了真实在线增量式自然梯度AC(true online incremental natural actor-critic,简称TOINAC)算法.TOINAC算法采用优于传统梯度的自然梯度,在真实在线时间差分(true online time difference,简称TOTD)算法的基础上,提出了一种新型的前向观点,改进了自然梯度行动者-评论家算法.在评论家部分,利用TOTD算法高效性的特点来估计值函数;在行动者部分,引入一种新的前向观点来估计自然梯度,再利用资格迹将自然梯度估计变为在线估计,提高了自然梯度估计的准确性和算法的效率.将TOINAC算法与核方法以及正态策略分布相结合,解决了连续空间问题.最后,在平衡杆、Mountain Car以及Acrobot等连续问题上进行了仿真实验,验证了算法的有效性.
Abstract:Policy gradient methods have been extensively studied as a solution to the continuous space control problem. However, due to the presence of high variance in the gradient estimation, policy gradient based methods are restricted by low sample data utilization and slow convergence. Aiming at solving this problem, utilizing the framework of actor-critic algorithm, a true online incremental natural actor-critic (TOINAC) algorithm, which takes advantage of the natural gradient that is superior to conventional gradient, and is based on true online time difference (TOTD), is proposed. In the critic part of TOINAC algorithm, the efficiency of TOTD is adopted to estimate the value function, and in the actor part of TOINAC algorithm, a novel forward view is introduced to compute and estimate natural gradient. Then, eligibility traces are utilized to turn natural gradient into online estimation, thereby improving the accuracy of natural gradient and efficiency of the method. The TOINAC algorithm is used to integrate with the kernel method and normal distribution policy to tackle the continuous space problem. The simulation tests on cart pole, Mountain Car and Acrobot, which are classical benchmark tests for continuous space problem, verify the effeteness of the algorithm.
文章编号:     中图分类号:TP301    文献标志码:
基金项目:国家自然科学基金(61303108,61373094,61472262);江苏省高校自然科学研究项目(17KJA520004);符号计算与知识工程教育部重点实验室(吉林大学)资助项目(93K172014K04);苏州市应用基础研究计划工业部分(SYG201422);高校省级重点实验室(苏州大学)项目(KJS1524);中国国家留学基金(201606920013) 国家自然科学基金(61303108,61373094,61472262);江苏省高校自然科学研究项目(17KJA520004);符号计算与知识工程教育部重点实验室(吉林大学)资助项目(93K172014K04);苏州市应用基础研究计划工业部分(SYG201422);高校省级重点实验室(苏州大学)项目(KJS1524);中国国家留学基金(201606920013)
Foundation items:National Natural Science Foundation of China (61303108, 61373094, 61472262); Jiangsu College Natural Science Research Key Program (17KJA520004); Program of the Key Laboratory of Symbolic Computation and Knowledge Engineering of Ministry of Education (Jilin University) (93K172014K04); Suzhou Industrial Application of Basic Research Program (SYG201422); Program of the Provincial Key Laboratory for Computer Information Processing Technology (Soochow University) (KJS1524); China Scholarship Council Project (201606920013)
Reference text:

朱斐,朱海军,刘全,陈冬火,伏玉琛.一种解决连续空间问题的真实在线自然梯度AC算法.软件学报,2018,29(2):267-282

ZHU Fei,ZHU Hai-Jun,LIU Quan,CHEN Dong-Huo,FU Yu-Chen.True Online Natural Actor-Critic Algorithm for the Continuous Space Problem.Journal of Software,2018,29(2):267-282