Why SHAREit Selects TiKV for Data Storage for Its 2.4-Billion-User Business

Linlin Yan

December 22, 2021

Industry: Information Technology Author: Linlin Yan (Database Engineer at SHAREit Group) SHAREit Group is a global internet technology company, dedicated to mobile application research and development, global mobile advertising solutions, and cross-border payment solutions. Our core app, SHAREit, has seen nearly 2.4 billion downloads worldwide. Our business network has reached over 200 countries and regions, especially Southeast Asia, South Asia, the Middle East and Africa. With billions of users, our products generate tremendous amounts of data every second.

NodeJS Client and CNCF LFX Mentorship Experience

Jiaxiao Zhou (Mossaka)

November 24, 2021

The project The goal of the ‘21 summer TiKV LFX mentorship project is to implement a client library for TiKV using in NodeJs. TiKV client has existing clients for these languages: C, Python, Go, Rust, and Java. Among these clients, the clients for Go and Rust are the most stable, actively developed and feature-complete. The clients for C and Python are thin wrappers of the Rust client using Rust’s Foreign Function Interface.

Getting Started with JuiceFS Using TiKV

Herald Yu

September 8, 2021

As a cloud-native distributed storage system, JuiceFS was designed into a plug-in structure at the beginning of its birth to ensure that new technologies can be continuously integrated into the JuiceFS ecosystem. According to your needs, you can flexibly choose two core components, the data storage engine and the metadata engine. The data storage engine is mainly object storage. It supports almost all public and private cloud object storage services, as well as KV storage, WebDAV, and local disks.

Looking Back at the LFX Mentorship Program Spring '21: My Journey to Becoming a TiKV Contributor

Andreas Zimmerer

May 31, 2021

I am a graduate student in the department of computer science at the Technical University of Munich (TUM), Germany. This spring (2021), I had the pleasure to contribute a major part of the new coprocessor framework to TiKV under the LFX Mentorship Program together with my mentors Andy Lok and Alex Chi. LFX Mentorship Program logo Background Before I applied to the LFX Mentorship Program, I had only contributed minor stuff to various open-source projects in my free time, like enhancing the documentation, fixing some small issues, and so on.

TiKV Rust Client - 0.1 release

Nick Cameron

May 14, 2021

We’re pleased to announce the 0.1 release of the TiKV Rust client. TiKV is a distributed key-value store. TiKV is powerful, mature, and widely used as part of TiDB (a ‘NewSQL’ database). It is open source and written in Rust. However, up until now it has been very difficult to use as a standalone store in a Rust program. With the Rust client, that has changed; it is now easy to access reliable, persistent storage from Rust.

How I Added Coprocessor ENUM/SET Support to TiKV in an LFX Mentorship Program


March 30, 2021

In September 2020, I was honored to participate in the TiKV Linux Foundation’s (LFX) Mentorship Program. I worked on the Coprocessor support ENUM/SET project to help enable TiKV Coprocessor to support ENUM and SET calculations. This helped improve TiKV’s calculation performance. In this post, I’ll share with you some background information about my project, how I implemented the project, my lessons learned during community cooperation, and my future plans for this project.

SIG-txn reading group, Nov + Dev 2020

Nick Cameron

November 4, 2020

The TiKV transactions SIG is starting up its reading group. We’ll try to read and discuss a paper each month. If you’re interested in research in distributed transactions, come join in! We’ll start by reading ‘Industrial-Strength OLTP Using Main Memory and Many Cores’, by Avni et al. at Huawei Research Centre. It was published in the proceedings of VLDB 2020 and you can download it from them. The paper describes a new storage engine for GaussDB (an HTAP (OLTP + OLAP) database from Huawei), that includes a transaction system, and a JIT compiler for SQL processing.

This Month in TiKV - September 2020

TiKV Authors

September 30, 2020

TiKV graduated within CNCF in September 2020! This is a piece of exceptionally exciting news for the community because it marks the CNCF’s recognition of TiKV’s maturity. Let’s take a look at what we have achieved in September! News @Siddontang shared TiKV Graduation Journey in CNCF at our monthly community meeting, which wrapped up important moments in TiKV’s graduation journey. The recorded video is available here. To celebrate TiKV’s CNCF graduation, we have made limited edition SWAGs for our contributors and adopters!

Generics and Compile-Time in Rust

Brian Anderson

September 17, 2020

The Rust programming language compiles fast software slowly. In this series we explore Rust’s compile times within the context of TiKV. Rust Compile-time Adventures with TiKV: Episode 2 In the previous post in the series we covered Rust’s early development history, and how it led to a series of decisions that resulted in a high-performance language that compiles slowly. Over the next few we’ll describe in more detail some of the designs in Rust that make compile time slow.

Documentation quest!

Nick Cameron

September 4, 2020

The TiKV Transactions SIG‘s documentation quest begins today! It is our first community activity. There are tons of bite-size tasks and some more involved work, with the goal of making TiKV’s transaction components well-documented, easy to understand, and a go-to resource for learning how to implement distributed transactions. Well-documented code is a joy to work with. Figuring out what undocumented code does is harder and takes longer, and you’re more likely to miss some detail which causes a bug later on.

Celebrating TiKV’s CNCF Graduation

TiKV Authors

September 2, 2020

Today we are excited to announce that TiKV is now a Cloud Native Computing Foundation (CNCF) graduated project! This marks CNCF’s recognition of TiKV, including its growing adoption, governance processes, project maturity, commitment to the community, and sustainability as well as inclusivity. “TiKV was one of our first Rust based projects and is truly a flexible and extensible cloud native key-value store,” said Chris Aniszczyk, CTO/COO of the Cloud Native Computing Foundation.

This Month in TiKV - August 2020

TiKV Authors

August 31, 2020

August ends with the completion of KubeCon + CloudNative Con EU Virtual and our mentorship programs. We can’t wait to see more to come in the future months! Let’s take a look at what we have achieved! News To clear the concern of the community, we transferred Placement Driver (PD) from PingCAP to TiKV side. Following this migration, Scheduling SIG joined the TiKV community at #sig-scheduling on the Slack channel.

Rustconf 2020

Nick Cameron

August 27, 2020

Rustconf 2020 happened last week (August 20th). For the first time, it was an online conference. Ironically that made it more difficult for me to attend because of timezones (and I was on vacation, so even less inclined to wake up at 4am). So I caught up with the talks earlier this week. I’ll write some short summaries of some of the talks I think will be of interest to the TiKV community.

Doubling System Read Throughput with Only 26 Lines of Code

Edward Huang

August 20, 2020

Follower Read is a highlight open-source feature that improves the throughput of the TiKV clients and reduces the load on the Raft leader. To understand how important this feature is, you’ll need a bit of background. TiKV stores data in basic units called Regions. Multiple replicas of a Region form a Raft group. When a read hotspot appears in a Region, the Region leader can become a read bottleneck for the entire system.

Announcing PD Transfer to TiKV Project

TiKV Authors

August 14, 2020

We decide to move the Placement Driver library entirely to TiKV org, happening at 11 AM, UTC+8, August 17, 2020.

My CommunityBridge Mentorship with TiKV Project

Chi Zhang

August 12, 2020

CommunityBridge is a 12-week full-time mentorship program developed by The Linux Foundation. It is designed to help developers with the necessary skills to experiment, learn, and contribute effectively to open source communities. This program is actively used by Cloud Native Computing Foundation as a mentorship platform across the CNCF projects. I was honored and lucky to participate in the Full Chunk-based Computing project proposed by TiKV, a CNCF incubating project. In this post, I would like to share my experience in the CommunityBridge Mentorship Program as a mentee to TiKV project.

This Month in TiKV - July 2020

TiKV Authors

July 30, 2020

The warmest month in the Northern Hemisphere is coming to an end soon. It’s time to wrap up everything we have achieved in July! Let’s get started. News TiKV is up for graduation from CNCF! This is a major milestone we have been waiting for, which consummates the efforts and commitments of the TiKV community since it was born. It would not be possible without you, our contributors and adopters! If you would like to support TiKV in the vote, follow the instructions to join the CNCF TOC group and post your replies here!

Announcing the Transaction SIG

The transaction SIG

July 21, 2020

TiKV has been open source since almost its beginning. Over the past year or so we’ve been trying to build a better, more open community around TiKV. Recently, we’ve been forming special interest groups (SIGs) to better organize and govern our community. Today, we are pleased to announce the official launch of the Transaction SIG. Transactions Transactions have been part of database systems for decades. In distributed databases they are the key mechanism for supporting properties such as isolation and consistency (well known from the ACID property and the CAP theorem), in other words, making it possible to manage a distributed database in a similar way to a non-distributed one.

This Month in TiKV - June 2020

TiKV Authors

July 1, 2020

How time flies! It’s halfway through the year already! We are expecting the next six months of this year to be as fruitful as the first six. Let’s take a look at what we have achieved in June! News TiKV 4.0 has reached general availability! Thanks to all users and contributors who helped with the release! TiKV has been used by more than 1,000 adopters in production scenarios across multiple industries worldwide.

TiKV 4.0 GA Release

The TiKV Authors

June 18, 2020

We are excited to announce the general availability of TiKV 4.0! We’re so glad and proud to have seen almost 1000 adopters use TiKV in production scenarios across multiple industries worldwide. Thanks to all users who have given us feedback, bug reports, and pull requests contributions over the past year. We’ve applied the lessons learned from these deployments to bring new features and better security to support users’ growing needs.

TiKV Performance Tuning with Massive Regions

Bokang Zhang

June 11, 2020

In TiKV clients, data is split into Regions, each storing data for a specific key range. These Regions are distributed among multiple TiKV instances. As data is written into a cluster, millions of or even tens of millions of Regions are created. Too many Regions on a single TiKV instance can bring a heavy burden to the cluster and affect its performance. This article introduces the workflow of Raftstore (a core module of TiKV), explains why a massive amount of Regions affect the performance and offers methods for tuning TiKV performance.

This Month in TiKV - May 2020

TiKV Authors

May 28, 2020

As the summer tiptoed in, we were cheerful about everything it brings to us: sunshine, ice cream, and a busy but fulfilling May! Let’s take a look at what we have accomplished. News The first CNCF ambassador spotlight goes to Queeny Jin, who has been spreading the word about TiKV to English speakers since its inception in 2016. We appreciate Queeny’s contribution and “together we can go further!” The full blog is available here.

TiKV in JD Cloud & AI

Can Cui

May 26, 2020

Industry: Cloud Computing Author: Can Cui (Infrastructure Specialist at JD Cloud & AI) JD Cloud & AI is a smart technology provider under JD Group and is built upon JD Group’s business expertise and technological accumulations in areas such as artificial intelligence, big data, cloud computing and the internet of things. It has established a technology ecosystem that delivers unmatched customer value through comprehensive services spanning from foundational platform building to business consultation and planning, business platform construction and operations and maintenance, and is driven by industry leading products that enable smart and digital enterprises and governments through solutions across a wide variety of scenarios.

Building a Large-scale Distributed Storage System Based on Raft

Edward Huang

May 21, 2020

In recent years, building a large-scale distributed storage system has become a hot topic. Distributed consensus algorithms like Paxos and Raft are the focus of many technical articles. But those articles tend to be introductory, describing the basics of the algorithm and log replication. They seldom cover how to build a large-scale distributed storage system based on the distributed consensus algorithm. Since April 2015, we PingCAP have been building TiKV, a large-scale open-source distributed database based on Raft.

Implement Raft in Rust

Siddon Tang

May 14, 2020

Consensus is one of the most important challenges in designing and building distributed systems–how to make sure multiple nodes (or agents, processes, threads, tasks, participants, etc.) in a group agree on a specific value proposed by at least one of the nodes. As an open-source distributed transactional key-value database, TiKV uses the Raft Consensus Algorithm to ensure data consistency, auto-failover, and fault tolerance. TiKV has thus far been used by almost 1000 adopters in their production environments in a wide range of industries, from e-commerce and food delivery, to fintech, media, gaming, and travel.

This Month in TiKV - April 2020

TiKV Authors

April 30, 2020

April 1st marked TiKV’s 4th birthday. We are grateful for the support and contribution of the community while focusing on making a better TiKV. Let’s take a look at what we have achieved in April! News We’re working on another API - VerKV, a multi-versioned KV without transactions, to help TiKV expand application scenarios. It is an API halfway between RawKV and TxnKV. The work plan proposed by @fredchenbj is available here; join us by picking a task you are interested in!

Quickly Find Rust Program Bottlenecks Online Using a Go Tool

Ke’ao Yang

April 23, 2020

Profiling large Rust applications online is difficult. Current profilers are not up to the job. When we need to analyze a Rust program’s performance, we often think about perf. To use perf, we need to: Install a complete perf program to sample stack traces. Use a set of script tools to process the file obtained by sampling. Visualize the output of the processing result in the previous step.

Tick or Tock? Keeping Time and Order in Distributed Databases

Siddon Tang

April 16, 2020

Preface At re:Invent 2017, Amazon Web Services (AWS) announced Amazon Time Sync Service, a highly accurate and reliable time reference that is natively accessible from Amazon EC2 instances. It is much like the Google TrueTime published in 2012. Why do Google and AWS both want to make efforts to provide global time service? Is there any inspiration for building a distributed database? This topic is important to think about.

This Month in TiKV - March 2020

TiKV Authors

April 2, 2020

In this unprecedented global situation, we hope you can all keep a positive attitude and stay healthy. Let’s take a look at what we have achieved in March! News TiKV has finished a third-party security assessment! This assessment of the TiKV scope, commissioned by CNCF and executed by Cure53 in early 2020, concludes with generally positive results. The full report is available here. The TiKV community held the first-ever community meeting on March 26, 2020.

RocksDB in TiKV

Siddon Tang

March 18, 2020

This is the speech presented by Liu (Siddon) Tang at the RocksDB meetup on August 28, 2017. Some of the content has been adapted to be up to date. Speaker Introduction Why did we choose RocksDB? How are we using RocksDB? TiKV Architecture Region Raft InsertWithHint Prefix Iterator Table Property for Region Split Check Table Property for GC Check Ingest the SST File Others How are we contributing?

This Month in TiKV - February 2020

TiKV Authors

March 12, 2020

As the cold winter has passed, we are welcoming a beautiful spring with blossoms, gentle breeze, and of course our monthly TiKV update, covering February 2020. Let’s get started! Releases This month our team made 2 TiKV releases! You can review the changelogs here: 3.0.10 bug fixes 4.0.0-beta.1 bug fixes compatibility improvements new features including supporting fetching configuration items from the status port via HTTP API in #6480 and optimizing the performance of Chunk Encoder in Coprocessor in #6341 Reading materials For raft-based implementations like TiKV, stale read may happen due to a brain split in the Raft group.

TiKV Security Audit Results

TiKV Authors

March 5, 2020

Today we’re glad to announce that TiKV just finished a third-party security assessment, funded by CNCF/The Linux Foundation. From Feb 10 to March 03, 2020, the team from Cure53 performed comprehensive tests on TiKV in the following aspects: General security posture checks that entail: Language specifics (Rust) External Libraries & Frameworks Configuration Concerns Access Control Logging/Monitoring Unit/Regression and Fuzz-Testing Documentation Security Contact/Fix Handling Bug Tracking & Review Process Manual code auditing & penetration testing This independent security audit was performed on locally installed systems, as well as with a typical production environment provided by the TiKV team.

How TiKV Uses "Lease Read" to Guarantee High Performances, Strong Consistency and Linearizability

Siddon Tang

February 27, 2020

TiKV, an open source distributed transactional key-value store (also a Cloud Native Computing Foundation incubating project), uses the Raft consensus algorithm to ensure strong data consistency and high availability. Raft bears many similarities to other consensus algorithms like Paxos in its ability to ensure fault-tolerance and its performance implementations, but generally easier to understand and implement. While our team has written extensively on how Raft is used in TiKV (some examples: Raft in TiKV, the Design and Implementation of Multi-raft, Raft Optimization), one topic we haven’t discussed is when stale read happens due to a brain split within a Raft group, and the old Raft leader is not aware that a new leader is elected, what should we do?

This Month in TiKV - January 2020

TiKV Authors

February 19, 2020

Welcome to the first monthly wrap-up of 2020. In January, we had regular releases, some great articles, some great PRs and issues, and of course, new members of the community. Let’s take a look! Releases This month our team made 2 TiKV releases: The 3.0.9 release includes the configuration change to speed up the Region scattering, some optimized configuration items, and a fixed issue for Region Merge. The 4.0.0-beta release includes major updates such as quick backup and restoration support, RocksDB version upgrade, and also bugs fixes.

The Rust Compilation Calamity

Brian Anderson

January 31, 2020

The Rust programming language was designed for slow compilation times. I mean, that wasn’t the goal. As is often cautioned in debates among their designers, programming language design is full of tradeoffs. One of those fundamental tradeoffs is run-time performance vs. compile-time performance, and the Rust team nearly always (if not always) chose run-time over compile-time. So Rust compile times are bad. This is kinda infuriating, as almost everything that matters about Rust is pretty damn good.

This Month in TiKV - December 2019

Ana Hobden

January 18, 2020

Happy new year and happy new decade! How did our contributors wrap up the decade? Let’s take a look, and then take a peek at what’s coming this year! Releases This month our team made 4 TiKV releases - 3.0.x and 2.1.x releases are minor releases, and 3.1.0-beta is our upcoming next major release before 4.0. You can review the changelogs here: 3.0.7 3.0.8 2.1.19 3.1.0-beta Upgrading? Here’s things you should know:

This Month in TiKV - November 2019

Ana Hobden

December 11, 2019

As winter descends on the northern hemisphere, we encourage you to keep warm by compiling TiKV! In November we had new minor releases, a new crate release, and some community updates! Let’s take a look! Releases This month our team made 2 minor TiKV releases! You can review the changelogs here: 2.1.18 3.0.6 Upgrading? Things to note: In 3.0.6, #5697 introduced the ability to generate flamegraphs of TiKV at runtime!

How TiKV reads and writes

Tang Liu

December 6, 2019

This article introduces in detail how TiKV handles read and write operations. Together we will explore how TiKV, as a distributed key-value (KV) database, stores the data contained in a write request and how it retrieves the corresponding data with consistency guaranteed. Before you read it Before we begin, we need to introduce some essential concepts of TiKV to help you fully understand the process. If you are already familiar with these, you can skip to the next section.

TiKV celebrates 200 contributors!

Tang Liu

November 22, 2019

Dear TiKV contributors: As you might’ve noticed, the TiKV project has recently hit its 200th contributor in its core repository. This marks a very important milestone in our community growth. As the lead of the TiKV team at PingCAP and the senior Maintainer of the TiKV project, I would like to thank you all for helping us achieve this milestone. TiKV was originally created by PingCAP to complement TiDB in Jan, 2016 and open sourced in April, 2016.

This Month in TiKV - October 2019

Ana Hobden

November 20, 2019

It’s time for a very non-spooky October edition of This month in TiKV! We have a few treats for you while we’ve been teaching TiKV some new tricks. News This month our team made one TiKV minor release (v3.0.5)! This release includes bug fixes and minor, backwards compatible features. You can review the changelogs here: 3.0.5 Upgrading? This release includes only bugfixes! Reading materials PingCAP wrote “INSERT INTO tidb.hackathon_2019 VALUES (“Hack”, “Fun”, “TiDB Ecosystem”)" talking about the recent Hackathon they held in China!

This Month in TiKV - September 2019

Ana Hobden

October 18, 2019

Wonderful day! Welcome to the third edition of ‘This Month in TiKV’, covering September 2019. News This month our team made one TiKV minor release (v3.0.4)! This release includes bug fixes and minor, backwards compatible features. You can review the changelogs here: 3.0.4 Upgrading? Please take note of these things: There is a new batch split region command. There is a new grpc-memory-pool-quota configuration option. Reading materials Here are some articles our contributors have published over the last month:

This Month in TiKV - August 2019

Ana Hobden

September 18, 2019

Happy day! Welcome to the second edition of ‘This Month in TiKV’, covering August 2019. (Yes we know it’s later than the first week of September, sorry!) The TiKV authors have been busy working on improving stability, fixing bugs, and laying out the foundations of TiKV 4.0. That is a lot to cover, so let’s get started! News This month our team made four TiKV minor releases! These minor releases include bug fixes and minor, backwards compatible features.

Migrating the TiKV Rust client from futures 0.1 to 0.3

Nick Cameron

August 7, 2019

I recently migrated a small/medium-sized crate from Futures 0.1 to 0.3. It was fairly easy, but there were some tricky bits and some things that were not well documented, so I think it is worth me writing up my experience. The crate I migrated is the Rust client for the TiKV database. It is about 5500 LoC and uses futures fairly heavily (because it communicates with TiKV using gRPC and the interface to that is async).

This Month in TiKV - July 2019

Nick Cameron

August 7, 2019

Hi! Welcome to the first ever edition of ‘This Month in TiKV’, covering July 2019. As the name suggests, this is a monthly newsletter covering interesting things happening in the world of TiKV, an open-source, distributed key-value store. We’re just getting started with the newsletter, and you should expect it to evolve as we go along. We hope to cover news and events of interest to TiKV contributors and users, significant PRs and issues in the TiKV repo, and generally keep you informed of what is going on.

TiKV 3.0 GA Release

July 17, 2019

Today we’re proud to announce the general availability of TiKV 3.0! Whether spanning hundreds of nodes or storing over a trillion key-value pairs, we’ve seen our users put TiKV to the test in serious, real-world, production scenarios. In 3.0, we’ve applied the lessons learned from these deployments to bring a host of new features that can better support users’ growing demands. Before release, TiKV 3.0 additionally underwent an official Jepsen test with TiDB.

Distributed Systems Training in Rust and Go

Brian Anderson, Senior Database Engineer for TiKV

June 20, 2019

On the TiKV team we love the Rust and Go programming languages. They are the languages in which we write most of our software, with TiKV in Rust, and its sister project, TiDB, in Go. They have empowered us to build these fast and reliable distributed systems from the ground up, and iterate on them quickly and confidently. These languages are the future of systems programming. Creating reliable distributed systems like TiKV demands a lot from contributors — they not only need to be experts in storage and distributed systems, but also to be comfortable expressing that knowledge in these modern languages.

CNCF TOC votes to move TiKV to Incubating Status

The TiKV Team

May 21, 2019

Today, the Cloud Native Computing Foundation’s (CNCF) Technical Oversight Committee (TOC) voted to accept TiKV as an incubation-level hosted project. The full announcement can be found on the CNCF blog, and you can find us listed under the CNCF Incubating Projects. We’d like to thank all of our contributors, users, and advocates over the last several years for helping us reach this very important milestone! Thank you! As a CNCF incubating project we’ll continue to enjoy many of the benefits and services the foundation offers, as well as greater worldwide visibility.

TiKV Community Connections

Ana Hobden, TiKV Ecosystem Lead

April 3, 2019

A few months ago at TiDB DevCon we talked about how we’d be opening new TiKV community forums & a community chat. We’re very happy to announce we’re releasing forum.tikv.org for forums, and tikv-wg slack for chat.

TiKV 2.1: More Raft, Less Hotspots

Siddon Tang, Chief Engineer at PingCAP and TiKV Project Lead

December 4, 2018

The TiKV team is proud to announce today that TiKV 2.1 is ready for General Availability (GA). TiKV is an open-source distributed transactional key-value store built in Rust.

TiKV 2.1 GA Release Notes

November 30, 2018

On November 30, 2018, TiKV 2.1 GA is released. See the following updates in this release. Compared with TiKV 2.0, this release has great improvements in stability, performance, and usability.