How Discord Stores Trillions of Messages in Its Database? (film, 7 minutes)
In this video, ByteByteGo discusses not just a database migration, but a colossal task Discord engineers undertook, moving trillions of messages from one database to another. The author highlights the fascinating scale of what it takes to migrate such data. Discord faced serious performance issues using Cassandra and decided to migrate to ScillaDB, a faster and more reliable alternative. The insights are based on public information, including Bo Ingram's presentation at the ScillaDB Summit 2023. The engineering culture at Discord is key here, showcased by an innovative approach to complex tasks. A significant part of their success involved starting the migration with smaller databases, allowing the team to test processes. They also implemented a new intermediate layer between the API monolith and the database using Rust. The concept of request coalescing drastically reduced server load.
A crucial step was creating the Superdisk, which combined local SSDs with Google Cloud's Persistent Disk. The Discord team focused on low-latency disk reads, which were vital for system performance. The Superdisk used RAID 0 for local SSDs and RAID 1 for Persistent Disk, yielding desired results. Once preparations were finished, the team tackled migrating the largest database - the Cassandra Messages Cluster. This complicated process took just 9 days, which is impressive considering it was done with zero downtime. The result was a much quieter system operating with significantly fewer nodes, down from 177 to 72.
Migrating a production database at such a large scale is no easy feat, but through innovative solutions and well-executed plans, Discord achieved exceptional results. The engineering team is immensely proud of their accomplishment, and the video excellently summarizes the endeavor. This entire operation showcases the importance of proper strategies and risk management in modern systems.
As of the time of writing this article, the video has garnered 182,182 views and 7,214 likes. For anyone interested in engineering and system development, it's worth following such topics. Additionally, the author suggests subscribing to their system design newsletter that covers the latest trends and topics in large-scale systems, trusted by 400,000 readers. Sign up at blog.bybygo.com to stay updated with the latest information!
Toggle timeline summary
-
Introduction to the colossal database migration task faced by Discord engineers.
-
An overview of migrating trillions of messages from one database to another.
-
The speaker shares excitement about discussing the database migration since joining Discord.
-
Details about the migration process from Cassandra to ScillaDB, a more reliable database.
-
Bo Ingram's presentation at ScillaDB Summit 2023 provided insights into the migration.
-
Discord's initial challenges with performance issues using Cassandra due to rapid growth.
-
Introduction of ScillaDB as a Cassandra-compatible database with better performance.
-
Discussion on the improvements ScillaDB offers, such as being garbage collection-free.
-
Creation of an intermediate layer called Data Services to improve database interactions.
-
Concept of Superdisk designed to handle disk latency challenges in migration.
-
Discord's innovative solution to prioritize low-intensity disk reads for improved performance.
-
Successful implementation of Superdisk technology to enhance database operations.
-
Details of the final migration of the Cassandra Messages Cluster in just 9 days.
-
The outcome was a more efficient system with fewer nodes and improved latency.
-
Recognition of the difficulty and success of migrating a production database at scale.
-
Invitation to subscribe to the system design newsletter for ongoing insights.
Transcription
In this video, we're going to discuss not just a database migration, but a colossal task Discord engineers took on, moving trillions of messages from one database to another. If you ever wonder what it takes to migrate data of such an unimaginable scale, you're going to love this. Let's dive right in. I've been eager to talk about this topic ever since I joined Discord. The engineering culture there is dynamic and innovative, and this story is a perfect illustration of that. Though this is all from public information, the insights and takeaways are my own. Discord recently peeled back the curtain on how they migrated trillions of messages from Cassandra, the database that housed your chats and conversations, to ScillaDB, a faster and more reliable alternative. The process was first shared at ScillaDB Summit 2023 by Bo Ingram, followed by a detailed blog post. A big shout out to Bo for presenting this complex process so clearly. So what did Bo share? I will summarize and share my own takeaways. You can watch his video and read his blog post if you want to learn more about it. Let's rewind back a few years. Discord found themselves in a bit of a pickle with their database choice. They were using Cassandra, but as the platform continued to grow, they faced serious performance issues. It took a lot of effort to maintain the main Cassandra cluster that held messages. The latency was unpredictable, and the frequent on-call incidents put huge strain on the team. By 2022, the Cassandra cluster held trillions of messages across 177 nodes. They knew they needed something different, but this message cluster was critical to Discord. If the message cluster is slow, Discord is going to be slow. If the message cluster is down, Discord is going to be down. Their solution? ScillaDB, a Cassandra-compatible database, but with a more powerful C++ base engine under the hood. Instead of jumping in headfirst to solve the riskiest and biggest problem, they started the migration with smaller databases. And this is my first takeaway. At Discord, the engineers move fast when mistakes are reversible. But if the solution is a one-way door, they spend a lot more time and care to get it right. They use these small migrations as an opportunity to test their waters and iron out as many issues as possible before tackling the big beast that held trillions of messages. Let's talk about ScillaDB. It was written in C++ and promised better performance, faster repairs, and most importantly, it was garbage collection-free. For a team that has so many issues with Cassandra's garbage collector, this was a breath of fresh air. The next key step was to create an intermediate layer between the API monolith and the database clusters. This layer is called Data Services. It was written in Rust, which is a safe and highly performance language. And it's a joy to write in Rust. The really cool idea about this layer is something called request coalescing. If multiple users request the same data, the database only needs to be queried once. This drastically reduced the potential for hot petitions. Imagine all those unintended add everyone messages in hugely popular Discord servers. With this layer in between, it is no problem for the database at all. Then they came up with the concept of a Superdisk, the database cluster run on Google Cloud. When faced with the challenges of disk latency, they couldn't rely on the local NVMe SSDs on the virtual machines for the critical data storage due to reliability and durability issues. The alternative was Google Cloud's Persistent Disk. While it was reliable and flexible, it had the disadvantage of higher latency since they were network-attached rather than directly attached. So what did Discord do? They went back to the drawing board and focused on creating a solution tailored to their specific needs. They chose to prioritize low-intensity disk reads over all other disk metrics while maintaining the existing database uptime guarantee. What they came up with was a Superdisk combining the best of local SSDs and Persistent Disk at the software level. Their Superdisk was a two-layer RAID solution. They used RAID 0 to combine multiple local SSDs into one low-latency virtual disk, and then RAID 1 to mirror this RAID 0 away with a Persistent Disk. They then configured the Linux kernel to direct the write to the Persistent Disk, which has a strong durability guarantee, and the read to the local SSDs, which offer low latency. This setup ensured low-latency reads from the local SSDs and write durability from the Persistent Disks. What's so great about this is an excellent example of problem-solving from first principles. Discord was dealing with a problem that had no ready-made solution. Instead of trying to make do with what was available, they redefined the problem based on their specific needs and then built a solution to match. The Superdisk worked. At peak low, the databases no longer started queuing up disk operations, and they saw no change in query latency. Finally, with all this prep work done, it was time to migrate their largest database, the Cassandra Messages Cluster. With trillions of messages and nearly 200 nodes, it was a daunting task. But with a newly written disk migrator in Rust and some clever strategies, they managed to do it in just 9 days. Yes, you heard it right, they migrated trillions of messages with no downtime in less than 2 weeks. And the payoff? A significantly quieter, more efficient system. It went from running 177 Cassandra nodes to just 72 ScalaDB nodes. It drastically improved latencies and the quality of life for the on-call staff. It was no ordinary task, but with clever risk mitigation and innovative solutions, they put it off. So there you have it. I am incredibly proud of what the team was able to pull off. Migrating a production database is no joke. And doing it at this scale, well, it is hard. If you like our videos, you may like our system design newsletter as well. It covers topics and trends in large-scale system design, trusted by 400,000 readers. Subscribe at blog.bybygo.com.