Stripe’s Post

Ever wondered how Stripe maintains 99.999+% availability? Our internal Database as a Service system handles version upgrades, resharding, and more while ensuring consistency without any downtime. Check out our latest blog post to lean!

It's amazing to watch a plan come together! This article details the world class engineering work that my Document Database teams have been working on. Five nines (99.999%) of reliability with zero-downtime data migrations supporting $1T+ in payments volume. Totally EPIC! https://lnkd.in/eKJRhYsY

Stripe recently shared insights on their blog about the impressive uptime of their document database and the innovative data movement platform they’ve developed. Here are the key components: Proxy Management: With multiple shards in play, a proxy is essential to direct queries to the appropriate shard. Why Metadata Server?: The metadata server plays a critical role by mapping data chunks to their respective database shards, enabling the proxy server to identify where the needed data resides. The Migration Platform: Stripe's data migration platform was designed to seamlessly migrate data in the background without any downtime for clients. Here’s how they built the data migration platform: * Chunk Management: Source and target chunks are registered for migration, along with indexing the target shard. * Import and Replication: Data is imported, and replication is initiated between the source and destination shards. * Correctness: The system ensures data integrity and updates routes in the metadata server to redirect queries to the correct destination chunk. You can find the full article - https://lnkd.in/gfUSmskD

Interesting - another complex proxy layer built in house. Reminds me a bit of the recent Figma blog about their Postgres sharding. And I've seen lots of other companies do it too - even with Oracle for data persistence. I have mixed feelings about this. I wonder if simpler solutions would suffice in many cases, like multiple parallel full app stacks each with a single HA database setup, but not trying to support all the flexibility of a "platform" that's managed more like a single large distributed system. Seems like adding a lot of complexity to the code base you own and maintain, mainly for more sophisticated cross-shard queries and rebalancing. Rebalancing processes and cross-shard query capabilities that are more manual might be ok, in exchange for a much simpler stack.

If you had to handle transactions worth $1,000,000,000,000 with up to 3 nines of availability (99.999%),and as little downtime as possible. How would you do it? This is exactly the challenge Stripe faced last year in 2023 and took an impressive approach with their systems. Here’s how they did it: → What were the requirements of the system?  - High availability, durability, and performance standards. - Minimal database functions exposed to avoid suboptimal queries. - Horizontal scalability with sharding and strong security. - First-class support for multitenancy with enforced quotas. → What was the challenge?  - Handling large-scale, real-time data with MongoDB while maintaining 99.999% uptime. - Migrating data across shards without downtime, ensuring consistency and completeness. - Managing complex distributed systems for financial data, including asynchronous replication and data integrity. → What was the solution?  - DocDB: A self-managed DBaaS built on MongoDB Community, customized for low latency and diverse query handling. Data Movement Platform: This platform enabled horizontal scaling and online data migrations, allowing shard splitting and merging without downtime. → How did the system work?  - DocDB Infrastructure: Used MongoDB to store data across thousands of shards. Applications accessed data through custom proxy servers that handled queries and routing. - Database Sharding: Data was split into chunks and stored in different shards. A metadata service helped find the right shard for each query. - Data Movement Platform: This platform managed data migrations across shards without downtime. It handled tasks like registering migrations, importing data, syncing changes, and switching traffic. - Oplog and CDC Systems: Tracked changes and ensured real-time data consistency across all shards. - Fault Tolerance: Automatic failover and data replication ensured data availability and reliability. Key Features:    - Client-transparent migrations with a Coordinator component.   - Optimized bulk data import and async replication systems.   - Traffic switch protocol for rerouting queries during migrations.   - Continuous improvements like shard autoscaling and heat management for optimal performance. – Read the article in detail here, it’s quite interesting: https://lnkd.in/eBddWznb

🚀 Recently, I read an incredible blog post by the engineering team at Stripe that dives deep into how they maintain a stunning 99.999% uptime while processing $1 trillion in payments in 2023! 🚀 Here’s a glimpse into their approach: The blog provides a detailed look at their in-house Database-as-a-Service (DBaaS), known as DocDB, which is built on MongoDB Community. DocDB handles over 5 million queries per second, ensuring seamless operations for Stripe's diverse applications. This system's flexibility and scalability allow it to efficiently manage petabytes of critical financial data spread across thousands of database shards. One of the most impressive aspects of their infrastructure is the Data Movement Platform. This platform enables client-transparent data migrations across thousands of database shards with zero downtime. It ensures continuous availability by merging underutilized shards for improved efficiency and splitting them during traffic surges. This ability to adapt to changing traffic patterns is crucial for maintaining optimal performance. Stripe's infrastructure is designed with scalability and reliability at its core. They use a multi-layered architecture that includes in-house developed database proxy servers. These servers handle query routing, combining results from various shards, and enforcing concerns of reliability and scalability. Advanced replication mechanisms ensure data consistency and availability, even during peak times, meeting the highest performance and reliability standards. The blog also touches on future developments, such as a heat management system to balance data across shards and shard autoscaling to respond to traffic changes dynamically. Read the full blog post for an in-depth look at their journey and technical innovations. 👉 https://lnkd.in/ef7QTuu3 #TechInnovation #EngineeringExcellence #SoftwareEngineering #TechTrends #DatabaseEngineering #SystemsEngineering #DatabaseEngineering #DataManagement #DatabaseDesign #DatabaseScalability #DatabasePerformance #DataMigration

Customers and businesses now expect payments to be processed immediately! 💳 In 2023 Stripe processed $1 trillion in payments—all while maintaining 99.999% uptime with zero-downtime data migrations. Check out this blog post that explores how Stripe’s DocDB, an extension of MongoDB Community, expertly manages enormous amounts of real-time data at scale. ⬇️ https://lnkd.in/gbDsDXYH #database #documents #data

I've a huge fan of Stripe for several years, booth from a customer perspective using it for online shopping, and as a developer, implementing its amazing API on mobile and web applications. Check out this blog post that explores how Stripe’s DocDB, an extension of MongoDB Community, successfully processed $1 trillion in payments—all this while maintaining 99.999% 🤯 uptime with zero-downtime data migrations; yes five nines! (~5m 13s of downtime for the whole year) https://lnkd.in/e2zd3Rjr

Our recent customer success story with YNAB details how they optimized their Postgres database performance and reduced costs with Crunchy Bridge. Read more to learn about why those chose to work with us over other vendors. https://lnkd.in/e4XCNFz4

Customers and businesses now expect payments to be processed immediately! 💳 In 2023 Stripe processed $1 trillion in payments—all while maintaining 99.999% uptime with zero-downtime data migrations. Check out this blog post that explores how Stripe’s DocDB, an extension of MongoDB Community, expertly manages enormous amounts of real-time data at scale. ⬇️ https://lnkd.in/gF2HMFYW