This Month in TiKV - October 2019

Ana Hobden

November 20, 2019

It’s time for a very non-spooky October edition of This month in TiKV! We have a few treats for you while we’ve been teaching TiKV some new tricks.

News

This month our team made one TiKV minor release (v3.0.5)! This release includes bug fixes and minor, backwards compatible features.

You can review the changelogs here: 3.0.5

Upgrading? This release includes only bugfixes!

Reading materials

PingCAP wrote “INSERT INTO tidb.hackathon_2019 VALUES (“Hack”, “Fun”, “TiDB Ecosystem”)" talking about the recent Hackathon they held in China! Two of the top three winning projects were improving TikV! Great job!

See “Using Raft to mitigate TiDB Cross-DC latency” (CN) and “Unified Thread Pool” (EN) for more information!

Notable PRs

“Batch (raw) get requests to same region across batch commands” by @tabokie reduces pressure on the readpool and CPU.
“Generate flamegraph at runtime” by @YangKeao introduces a new HTTP interface to get flamegraphs from TiKV at runtime.
“Coprocessor: v2 row format for decode” by @niedhui furthers the implementation of the v2 row format.
“txn: protect primary locks of pessimistic transactions from being collapsed” by @sticnarf adds some additional safety to pessimistic locks.
“Improve Coprocessor Table Lookup Performance” by @breeswish introduces a new interface on the MVCC level which can reuse cursors when keys are in ascending order, improving lookup performance.
“util: compare raw and encoded keys without explicit decoding” by @sticnarf allows TiKV to determine if an encoded and raw format refer to the same key without needing to encode or decode.

Notable issues

@siddontang found a performance regression in our upgrade to RocksDB 6.4.
Many Copr and PCP tagged issues have appeared, Copr issues are related to the Coprocessor SIG that has started, and the Performance Challenge Program (PCP) competition being facilitated by PingCAP in China.
@onitake asked about binary releases on Github and we confirmed they’re in progress!
We begun talking with relevant parties about getting an independent security audit.
We were able to discover a few new occasionally failing tests as part of our ongoing efforts to rid TiKV’s bugs.

Current projects

Here’s some of the things our contributors have been working on over the last month:

@winkyao and @dcalvin lead a proposal to create additional community structure and processes to adapt to our growing community and where we hope to go in the future.
New Special Interest Groups (SIG) are forming in November! November will bring a Coprocessor SIG as well as an Engine SIG. Interested in getting involved? Check their public channels (#copr-sig and #engine-sig) on our Community Chat to met the groups and see how you can help.
- More info on SIGs soon! We’re starting small, learning by experience, and figuring out the best way to do things!
@niedhui contributed a milestone to the ongoing work to improve TiKV’s row format.
@brson has been leading the engine trait abstraction task, trying to allow TiKV to support other storage backends in addition to RocksDB.
@hicqu and some of the team have been working on ensuring TiKV is free from memory leaks.
@overvenus has lead more work on our upcoming backups feature to ensure it is reliable.
@zhangjinpeng1987 has been diligently working to reduce transient test failures.
@Connor1996 and @yiwu-arbug have been working to improve our RocksDB plugin, titan.

If any of these projects sound like something you’d like to contribute to, let us know on our chat and we’ll try to help you get involved.

New contributors

We’d like to welcome the following new contributors to TiKV and thank them for their work!

If you’d like to get involved, we’d love to help you get started. You might be interested in tackling one of these issues. If you don’t know how to begin, please leave a comment and somebody will help you out. We’re also very keen for people to contribute documentation, tests, optimizations, benchmarks, refactoring, or other useful things.

This Week in TiDB

For more detailed and comprehensive information about TiDB and TiKV, we have weekly updates. The following cover October: