| 引用本文: | 朱阅岸,简怀兵,龙永超,李彬,王树,吴喜亮,钟治初,张延松.构建新型高性能与高可用的键值数据库系统.软件学报,2021,32(10):3203-3218 |
| |
|
| 本文已被：浏览 1424次 下载 3841次 |
码上扫一扫！ |
|
|
| 构建新型高性能与高可用的键值数据库系统 |
|
朱阅岸1,2, 简怀兵2, 龙永超2, 李彬2, 王树2, 吴喜亮2, 钟治初1, 张延松3
|
|
1.嘉应学院 计算机学院, 广东 梅州 514015;2.广州欢聚时代信息科技有限公司, 广东 广州 511442;3.中国人民大学 中国调查与数据中心, 北京 100872
|
|
| 摘要: |
| 近年来，写密集型应用程序越来越普遍.如何有效地处理这种工作负载，是数据库系统领域深入研究的方向之一.写操作开销主要由以下两个方面的因素构成：（1）硬件级别，即写操作引起的I/O，目前无法在短时间内消除这种开销；（2）软件开销，即修改内存数据拷贝以及构造日志记录造成的多次写操作.日志即数据（log-as-database，称其为单拷贝系统）的架构能够减少写操作引起的I/O，同时降低软件方面的开销.目前，业界对单拷贝系统展现出浓厚的兴趣.现有的单拷贝系统大部分建立在特殊的基础设施之上，例如infiniband或NVRam（非易失性随机存取存储器），这种基础设施尚未达到广泛可用或者是依托他系统（例如Dynamo）构建，这种方法缺乏灵活性与普适性.在商用机器环境中，自底向上构建了一个称为LogStore的键值数据库系统，采用log-as-database设计理念，以充分利用单拷贝系统的优点，在提升写操作性能的同时，有效缩短主备数据之间的差距.在系统中内嵌复制协议达到高可用性而不是依赖其他系统，使得系统灵活可控.系统新颖的查询执行模型将执行线程与特定分片绑定，结合多版本并发控制技术，以无锁的方式消除读写冲突、写写冲突以及上下文切换开销.用YCSB对系统性能进行了详细的评估，对比主流的键值系统HBase以及单拷贝系统实现LogBase，LogStore在写密集型工作负载上性能要优4倍左右.在崩溃恢复方面，LogStore可在1分钟之内完成TB级别数据规模的恢复，比LogBase要快1个数量级以上. |
| 关键词: 单拷贝系统 复制协议 写优化 恢复 细粒度缓冲区管理 |
| DOI：10.13328/j.cnki.jos.006023 |
| 分类号:TP311 |
| 基金项目:国家自然科学基金（61772533） |
|
| Building New Key-value Store with High Performance and High Availability |
|
ZHU Yue-An1,2, JIAN Huai-Bing2, LONG Yong-Chao2, LI Bin2, WANG Shu2, WU Xi-Liang2, ZHONG Zhi-Chu1, ZHANG Yan-Song3
|
|
1.School of Computer Science, Jiaying University, Meizhou 514051, China;2.Guangzhou YY Inc., Guangzhou 511442, China;3.National Survey Research Center at Renmin University of China, Beijing 100872, China
|
| Abstract: |
| In recent year, the write-heavy applications are more and more prevalent. How to efficiently handle this sort of workload is one of intensive research direction in the field of database system. The overhead caused by write operation is mainly issued by two factors. One is the hardware level, i.e., the IO cost caused by write operation. This cost cannot be removed in short period. The other is dual-copy software architecture, i.e., multiple writes caused by modifying in-memory data copy and formulating log records. The log-as-database architecture (the following refers it as single-copy system) can reduce the IOs and software cost caused by write as well. But existing systems treating log-as-database either are built on top of special infrastructure such as infiniband or NVRam (non-volatile random access memory) which is far from widely available or is constructed with the help of other system such as Dynamo, which is lack of flexibility and generality. This study builds from scratch a single copy system called LogStore oriented for commodity environment, which adopts log-as-database design philosophy to fully utilize its advantages that can boost the write performance and minimize the gap between primary and secondary. Embedding consensus module into system other than dependent on auxiliary systems makes it more flexible and controllable. The novel execution model binding thread to certain partition plus multi-version concurrency control technique eliminates read-write, write-write conflict, and context switch overhead in lock-free style. The YCSB benchmark is used to assess system performance thoroughly. Compared to prevalent key-value store HBase and its single-copy implementation LogBase, the proposed system can achieve about 4x better. In term of crash recovery, LogStore can finish recovery within one minute for TB scale data volume, which is one order of magnitude recovery time less than LogBase. |
| Key words: single-copy system replication protocol write optimized recovery fine-grain buffer management |
