| 本文已被：浏览 1363次 下载 3287次 |
码上扫一扫！ |
|
|
| 融合引力搜索的双延迟深度确定策略梯度方法 |
|
徐平安1, 刘全1,2,3,4, 郝少璞1, 张立华1
|
|
1.苏州大学 计算机科学与技术学院, 江苏 苏州 215006;2.软件新技术与产业化协同创新中心(南京), 江苏 南京 210093;3.符号计算与知识工程教育部重点实验室 (吉林大学), 吉林 长春 130012;4.江苏省计算机信息处理技术重点实验室 (苏州大学), 江苏 苏州 215006
|
|
| 摘要: |
| 近年来, 深度强化学习在复杂控制任务中取得了令人瞩目的效果, 然而由于超参数的高敏感性和收敛性难以保证等原因, 严重影响了其对现实问题的适用性. 元启发式算法作为一类模拟自然界客观规律的黑盒优化方法, 虽然能够有效避免超参数的敏感性, 但仍存在无法适应待优化参数量规模巨大和样本使用效率低等问题. 针对以上问题, 提出融合引力搜索的双延迟深度确定策略梯度方法(twin delayed deep deterministic policy gradient based on gravitational search algorithm, GSA-TD3). 该方法融合两类算法的优势: 一是凭借梯度优化的方式更新策略, 获得更高的样本效率和更快的学习速度; 二是将基于万有引力定律的种群更新方法引入到策略搜索过程中, 使其具有更强的探索性和更好的稳定性. 将GSA-TD3应用于一系列复杂控制任务中, 实验表明, 与前沿的同类深度强化学习方法相比, GSA-TD3在性能上具有显著的优势. |
| 关键词: 深度强化学习 元启发式算法 引力搜索 确定策略梯度 策略搜索 |
| DOI：10.13328/j.cnki.jos.006740 |
| 分类号:TP18 |
| 基金项目:国家自然科学基金(61772355, 61702055, 61876217, 62176175); 江苏高校优势学科建设工程 |
|
| Twin-delayed-based Deep Deterministic Policy Gradient Method Integrating Gravitational Search |
|
XU Ping-An1, LIU Quan1,2,3,4, HAO Shao-Pu1, ZHANG Li-Hua1
|
|
1.School of Computer Science & Technology, Soochow University, Suzhou 215006, China;2.Collaborative Innovation Center of Novel Software Technology and Industrialization (Nanjing), Nanjing 210093, China;3.Key Laboratory of Symbolic Computation and Knowledge Engineering of Ministry of Education (Jilin University), Changchun 130012, China;4.Jiangsu Provincial Key Laboratory for Computer Information Processing Technology (Soochow University), Suzhou 215006, China
|
| Abstract: |
| In recent years, deep reinforcement learning has achieved impressive results in complex control tasks. However, its applicability to real-world problems has been seriously weakened by the high sensitivity of hyperparameters and the difficulty in guaranteeing convergence. Metaheuristic algorithms, as a class of black-box optimization methods simulating the objective laws of nature, can effectively avoid the sensitivity of hyperparameters. Nevertheless, they are still faced with various problems, such as the inability to adapt to a huge scale of parameters to be optimized and the low efficiency of sample usage. To address the above problems, this study proposes the twin delayed deep deterministic policy gradient based on a gravitational search algorithm (GSA-TD3). The method combines the advantages of the two types of algorithms. Specifically, it updates the policy by gradient optimization for higher sample efficiency and a faster learning speed. Moreover, it applies the population update method based on the law of gravity to the policy search process to make it more exploratory and stable. GSA-TD3 is further applied to a series of complex control tasks, and experiments show that it significantly out performs similar deep reinforcement learning methods at the forefront. |
| Key words: deep reinforcement learning (DRL) meta-heuristic algorithm gravitational search deterministic policy gradient policy search |
