构建新型高性能与高可用的键值数据库系统
作者:
作者单位:

作者简介:

朱阅岸(1983-),男,博士,工程师,主要研究领域为高性能数据库系统,OLTP,新硬件数据库技术.
简怀兵(1980-),男,工程师,主要研究领域为分布式系统.
龙永超(1982-),男,学士,主要研究领域为高性能数据库系统,分布式系统.
李彬(1988-),男,工程师,主要研究领域为分布式系统.
王树(1979-),男,工程师,主要研究领域为分布式系统.
吴喜亮(1988-),男,硕士,主要研究领域为信号处理,计算机.
钟治初(1964-),男,副教授,CCF高级会员,主要研究领域为软件工程,编程语言.
张延松(1973-),男,博士,副教授,主要研究领域为内存数据库,OLAP,新硬件数据库技术,GPU数据库.

通讯作者:

朱阅岸,E-mail:iwillgoon@126.com

中图分类号:

TP311

基金项目:

国家自然科学基金(61772533)


Building New Key-value Store with High Performance and High Availability
Author:
Affiliation:

Fund Project:

National Natural Science Foundation of China (61772533)

  • 摘要
  • |
  • 图/表
  • |
  • 访问统计
  • |
  • 参考文献
  • |
  • 相似文献
  • |
  • 引证文献
  • |
  • 资源附件
  • |
  • 文章评论
    摘要:

    近年来,写密集型应用程序越来越普遍.如何有效地处理这种工作负载,是数据库系统领域深入研究的方向之一.写操作开销主要由以下两个方面的因素构成:(1)硬件级别,即写操作引起的I/O,目前无法在短时间内消除这种开销;(2)软件开销,即修改内存数据拷贝以及构造日志记录造成的多次写操作.日志即数据(log-as-database,称其为单拷贝系统)的架构能够减少写操作引起的I/O,同时降低软件方面的开销.目前,业界对单拷贝系统展现出浓厚的兴趣.现有的单拷贝系统大部分建立在特殊的基础设施之上,例如infiniband或NVRam(非易失性随机存取存储器),这种基础设施尚未达到广泛可用或者是依托他系统(例如Dynamo)构建,这种方法缺乏灵活性与普适性.在商用机器环境中,自底向上构建了一个称为LogStore的键值数据库系统,采用log-as-database设计理念,以充分利用单拷贝系统的优点,在提升写操作性能的同时,有效缩短主备数据之间的差距.在系统中内嵌复制协议达到高可用性而不是依赖其他系统,使得系统灵活可控.系统新颖的查询执行模型将执行线程与特定分片绑定,结合多版本并发控制技术,以无锁的方式消除读写冲突、写写冲突以及上下文切换开销.用YCSB对系统性能进行了详细的评估,对比主流的键值系统HBase以及单拷贝系统实现LogBase,LogStore在写密集型工作负载上性能要优4倍左右.在崩溃恢复方面,LogStore可在1分钟之内完成TB级别数据规模的恢复,比LogBase要快1个数量级以上.

    Abstract:

    In recent year, the write-heavy applications are more and more prevalent. How to efficiently handle this sort of workload is one of intensive research direction in the field of database system. The overhead caused by write operation is mainly issued by two factors. One is the hardware level, i.e., the IO cost caused by write operation. This cost cannot be removed in short period. The other is dual-copy software architecture, i.e., multiple writes caused by modifying in-memory data copy and formulating log records. The log-as-database architecture (the following refers it as single-copy system) can reduce the IOs and software cost caused by write as well. But existing systems treating log-as-database either are built on top of special infrastructure such as infiniband or NVRam (non-volatile random access memory) which is far from widely available or is constructed with the help of other system such as Dynamo, which is lack of flexibility and generality. This study builds from scratch a single copy system called LogStore oriented for commodity environment, which adopts log-as-database design philosophy to fully utilize its advantages that can boost the write performance and minimize the gap between primary and secondary. Embedding consensus module into system other than dependent on auxiliary systems makes it more flexible and controllable. The novel execution model binding thread to certain partition plus multi-version concurrency control technique eliminates read-write, write-write conflict, and context switch overhead in lock-free style. The YCSB benchmark is used to assess system performance thoroughly. Compared to prevalent key-value store HBase and its single-copy implementation LogBase, the proposed system can achieve about 4x better. In term of crash recovery, LogStore can finish recovery within one minute for TB scale data volume, which is one order of magnitude recovery time less than LogBase.

    参考文献
    相似文献
    引证文献
引用本文

朱阅岸,简怀兵,龙永超,李彬,王树,吴喜亮,钟治初,张延松.构建新型高性能与高可用的键值数据库系统.软件学报,2021,32(10):3203-3218

复制
分享
文章指标
  • 点击次数:
  • 下载次数:
  • HTML阅读次数:
  • 引用次数:
历史
  • 收稿日期:2019-08-06
  • 最后修改日期:2019-11-23
  • 录用日期:
  • 在线发布日期: 2021-10-09
  • 出版日期: 2021-10-06
您是第位访问者
版权所有:中国科学院软件研究所 京ICP备05046678号-3
地址:北京市海淀区中关村南四街4号,邮政编码:100190
电话:010-62562563 传真:010-62562533 Email:jos@iscas.ac.cn
技术支持:北京勤云科技发展有限公司

京公网安备 11040202500063号