This Month in TiKV - August 2019

Ana Hobden

September 18, 2019

Happy day! Welcome to the second edition of ‘This Month in TiKV’, covering August 2019. (Yes we know it’s later than the first week of September, sorry!)

The TiKV authors have been busy working on improving stability, fixing bugs, and laying out the foundations of TiKV 4.0. That is a lot to cover, so let’s get started!

News

This month our team made four TiKV minor releases! These minor releases include bug fixes and minor, backwards compatible features.

You can review the changelogs here:

Upgrading? Please take note of these things:

In 2.1.17 and 3.0.3 a new config-check option allows you to verify a configuration is correct before trying to start a node with it.
In 3.0.2 some logs which require no manual intervention (TiKV resolves them automatically) were reduced to INFO level.
In 2.1.16 raw_scan and raw_batch_scan now supports reverse.

Reading materials

Here are some articles our contributors have published over the last month:

@sunxiaoguang published some details of the deployment including TiKV at Zhihu in Lesson Learned from Queries over 1.3 Trillion Rows of Data Within Milliseconds of Response Time at Zhihu.com. Some highlights from the article are:
- 1.3 trillion records, growing ~3 billion per day.
- 12 million queries processed per second at peak.
- Sub-90ms response time.
@aknuds1 published Practical Networked Applications in Rust, Part 2: Networked Key-Value Store where he explores the Rust training course created by @brson and @sticnarf.
@siddontang published Porting TiDB to ARM64 for Greater Flexibility where he explores building and benchmarking TiKV and TiDB on ARM64 architectures.
@ethercflow published a follow up article how TCP Small Queues become a bottleneck for our test deployments on ARM64 architectures.
@c4pt0r published (CN) A Talk about the New Features of TiDB: From Follower Read where he discusses follower read, a new feature we’re planning for TiKV to be leveraged by query layers like TiDB.
@zhangjinpeng1987 published a new (CN) Source Code Reading: Distributed Transactions discussing the transaction model powering TiKV.

Roadmap

We’ve also began nailing down which features we want to include in a future TiKV 4.0 release. Here’s a few of our hopes and dreams:

(This is not an official list and may be subject to change)

Publish our RocksDB wrapper as tirocks on crates.io.
Abstract our engine so that we adapt to other storage technologies and offer more flexibility.
Begin adopting Joint Consensus, first to allow our clusters to more safely replace nodes in running deployments.
Further battle-hardening and improvements to the Titan storage backend.
Follower Snapshot, allowing new follows to catch up to the rest of a cluster without placing load on the current leaders.
Support Quiescent Region, reducing the heartbeats of inactive (‘cold’) regions.
Further coprocessor support and better documentation around coprocessor functionality, making it even easier to make your first contribution to TiKV.
A 1.0.0 release of our Rust client, along with more improvements to our other clients.
Reduce WAL usage to lower write pressure.

Notable PRs

In TiKV 2.1.17 gRPC was upgraded to fix a segfault causing abnormal exits #5441.
In TiKV 2.1.17 and 3.0.3 incorrect timestamps should no longer be reported from a Region #5296.
In TiKV 3.0.3 a possible request drop from ReadIndex when there is no leader was fixed #5316.
TiKV 2.1.16 now returns region errors when closing so the client will retry on other nodes #4820.
In TiKV 3.0.2 we fixed a bug in constraint checking during insertion when pessimistic transactions are enabled #5128.
In TiKV 3.0.2 we fixed the problem that TiKV loses some logs while panicking #5174.
@fullstop000 has a draft of Follower Replication for Raft pingcap/raft#249.
@nrc opened a PR to enable both prost and rust-protobuf to be used in TiKV #5379.
@sticnarf opened a series of PRs to begin enabling transaction support in the Rust client tikv/client-rust#108, tikv/client-rust#97, tikv/client-rust#92.
@MyonKeminta opened #5497 and #5390 to forward larger transaction support.

Notable issues

In TiKV 2.1.15 and 2.1.16, region statistics may not be correct #5306. Fixed in #5415 and released as part of 2.1.17.
In TiKV 3.0 rc2 to 3.0.2, ReadIndex might fail to respond to requests due to a duplicate context #4764. Fixed in #5213 and released as part of 3.0.3.
In TiKV #5291 we discovered a possible panic during region merge. Fixed in #5291 and released in 3.0.3.
In #4560 and #4581 we discovered a possible panic while catching up after a restart. Fixed in #4595 and released in 3.0.2.

Current projects

Here’s some of the things our contributors have been working on over the last month:

@sticnarf and @nrc have been working on transaction support in our Rust Client.
@fullstop000 has been working to implement follower replication in Raft, for future use in TiKV.
@hicquhas been leading the effort on supporting Joint Consensus in TiKV.
Several members of the @pingcap team have been working on further shrinking and optimizing TiKV’s build graph, resulting in faster compile times and less dependencies.
@zhouqiang-cl and @hoverbear have been working on improving our release process and docker containers.
@dcalvin and @hoverbear have been overhauling the TiKV.org documentation to better support our users.
@busyjay and @nrc have been collaborating to land the final PRs to support using Prost library.
@nrc has been working on Prost to help land some optimizations.
@overvenus has been driving forward our new backup and restore process for TiKV.
@MyonKeminta has been leading on refining and hardening our experimental pessimistic transaction support.
@zhouqiang-cl‘s team at @pingcap has been working to improve our CI and automation to help reduce time-to-merge, expand testing, and improve automation.
@yiwu-arbug and [Connor1996] are battle hardening Titan and incorporating feedback from some of our users.
@sunxiaoguang has been battle testing TiKV on massive, China-scale workloads at Zhihu (See his article!)
@busyjay and @zhangjinpeng1987 have been evaluating bloom filters.
@5kbpers, @brson and @aknuds1 have begun work on the engine abstraction project.
@MyonKeminta has been driving a project to facilitate larger transactions.

If any of these projects sound like something you’d like to contribute to, let us know on our chat and we’ll try to help you get involved.

New contributors

We’d like to welcome the following new contributors to TiKV and thank them for their work!

If you’d like to get involved, we’d love to help you get started. You might be interested in tackling one of these issues. If you don’t know how to begin, please leave a comment and somebody will help you out. We’re also very keen for people to contribute documentation, tests, optimizations, benchmarks, refactoring, or other useful things.

This Week in TiDB

For more detailed and comprehensive information about TiDB and TiKV, we have weekly updates. The following cover August,