| 本文已被：浏览 1627次 下载 7193次 |
码上扫一扫！ |
|
|
| Storm平台下的线程重分配与数据迁移节能策略 |
|
蒲勇霖1, 于炯1, 鲁亮2, 李梓杨1, 卞琛3, 廖彬4
|
|
1.新疆大学 信息科学与工程学院, 新疆 乌鲁木齐 830046;2.中国民航大学 计算机科学与技术学院, 天津 300300;3.广东金融学院 互联网金融与信息工程学院, 广东 广州 510521;4.新疆财经大学 统计与数据科学学院, 新疆 乌鲁木齐 830012
|
|
| 摘要: |
| 作为流式大数据计算的主要平台之一，Storm在设计过程中由于缺乏节能的考虑，导致其存在高能耗与低效率的问题.传统的节能策略并未考虑Storm的性能约束，可能会对集群的实时性造成影响.针对这一问题，设计了资源约束模型、最优线程重分配模型以及数据迁移模型.进一步提出了Storm平台下的线程重分配与数据迁移节能策略（energy-efficient strategy based on executor reallocation and data migration in Storm，简称ERDM），包括资源约束算法与数据迁移算法.其中，资源约束算法根据集群各工作节点CPU、内存与网络带宽的资源占用率，判断集群是否允许数据的迁移.数据迁移算法根据资源约束模型与最优线程重分配模型，设计了数据迁移的最优化方法.此外，ERDM通过分配线程减少了节点间的通信开销，并根据大数据流式计算的性能与能效评估ERDM.实验结果表明，与现有研究相比，ERDM能够有效降低节点间通信开销与能耗，并提高集群的性能. |
| 关键词: 大数据 流式计算 实时性 资源约束 数据迁移 能耗 |
| DOI：10.13328/j.cnki.jos.006074 |
| 分类号:TP311 |
| 基金项目:国家自然科学基金（61862060，61462079，61562086，61562078）；新疆维吾尔自治区研究生科研创新项目（XJ2019G038）；新疆大学博士生科技创新项目（XJUBSCX-201902） |
|
| Energy-efficient Strategy Based on Executor Reallocation and Data Migration in Storm |
|
PU Yong-Lin1, YU Jiong1, LU Liang2, LI Zi-Yang1, BIAN Chen3, LIAO Bin4
|
|
1.College of Information Science and Engineering, Xinjiang University, Urumqi 830046, China;2.College of Computer Science and Technology, Civil Aviation University of China, Tianjin 300300, China;3.College of Internet Finance and Information Engineering, Guangdong University of Finance, Guangzhou 510521, China;4.College of Statistics and Data Science, Xinjiang University of Finance and Economics, Urumqi 830012, China
|
| Abstract: |
| As one of the most popular platforms in big data stream computing, Storm is suffering from the problem of high energy consumption and low energy efficiency due to the lack of consideration for energy saving strategy in the design process. Without taking the performance constraint of Storm into consideration, the traditional energy-efficient strategies may affect the real-time performance of cluster. Aiming at this issue, models of the resource constraint, the optimal executor reallocation, and the data migration are set up, and the energy-efficient strategy based on executor reallocation and data migration in Storm (ERDM) is further proposed, while ERDM is composed of resource constraint algorithm and data migration algorithm. The resource constraint algorithm estimates whether the cluster is appropriate for data migration according to the utilization of CPU, memory, and network bandwidth in each work node. The data migration algorithm designs optimal method to migrate data according to the resource constraint model and the optimal executor reallocation model. Moreover, the ERDM allocates the executors so as to reduce communication cost between nodes. The ERDM is evaluated by measuring the cluster performance as well as energy consumption efficiency in big data stream computing environment. The experimental results show that the proposed strategy can reduce communication cost and energy consequence efficiently while the cluster performance is improved compared with existing researches. |
| Key words: big data stream computing real-time resource constraint data migration energy consumption |
