Redpanda Data

🚀 WHAT'S NEW: R1 Engine, AI connectors, Cloud availability, and more 🚀 Ready to level up your #streamingdata? We're thrilled to bring you six exciting announcements, including new features, upcoming betas, and a peek into the vision driving our biggest reveal yet. Here's what's in store: ☀ 𝐑𝐞𝐝𝐩𝐚𝐧𝐝𝐚 𝐎𝐧𝐞 - our upcoming multimodal streaming data engine, "R1", with flexible topics to meet diverse latency, storage, durability, and security requirements. 🔌 𝐑𝐞𝐝𝐩𝐚𝐧𝐝𝐚 𝐂𝐨𝐧𝐧𝐞𝐜𝐭 𝐟𝐨𝐫 𝐂𝐥𝐨𝐮𝐝 - build powerful #data pipelines with over 100 connectors and stream processors now available in Redpanda Cloud. ⚡ 𝐍𝐞𝐰 𝐀𝐈 𝐜𝐨𝐧𝐧𝐞𝐜𝐭𝐨𝐫𝐬 - integrate your #AI pipelines with popular #LLMs and vector databases. Plus, GPU runtime support for high-performing AI workloads. 🔍 𝐀𝐩𝐚𝐜𝐡𝐞 𝐈𝐜𝐞𝐛𝐞𝐫𝐠 𝐓𝐨𝐩𝐢𝐜𝐬 - easily access your streaming data from #Iceberg tables using standard #SQL engines. ☁ 𝐂𝐥𝐨𝐮𝐝 𝐓𝐨𝐩𝐢𝐜𝐬 - mix and match different topics within the same Redpanda cluster to optimize each workload for latency, cost, and performance. 👜 𝐊𝐚𝐟𝐤𝐚 𝐌𝐢𝐠𝐫𝐚𝐭𝐨𝐫 - move your workloads from any #Kafka system to Redpanda in a single command. Read all the announcements here👇 https://lnkd.in/gW9cMuuF

🚤 “You're gonna need a bigger boat.” Handling huge volumes of #realtime data can become troublesome if you're not prepared. This is especially true for industries like #fintech and financial services where you need manage real-time #data efficiently while keeping costs down and latencies low. To help you keep everything afloat, this blog post explains how #SaaS companies can build scalable real-time data systems that grow with demand. Dive right in 🦈 https://lnkd.in/guubeCbF #realtimedata #cloudarchitecture #datainfrastructure

⚙ VIRTUAL WORKSHOP: Building GenAI applications with Redpanda and #AWS Join our action-packed session with AWS Partners where you'll learn all about #GenAI, #serverless architecture, and #streamingdata! With friendly experts as your guides, you'll build an interactive game with AI-driven characters in just a few hours. Mark your calendar for: 🗓 Thursday, October 10th 🕘 8:00am-10:30am EDT | 1:00pm-3:30pm BST Sign up to join👇 https://lnkd.in/ggDZvz-h

The latest episode of Analytics Everywhere is here! Join us for an in-depth conversation with Redpanda Data founder & CEO Alexander Gallego, as we dive into all sorts of topics from the project's origin story, to trends in streaming data, AI, and much more. On Youtube: https://hubs.li/Q02Rz_8w0 On Spotify: https://hubs.li/Q02RzH360 Check out Redpanda at ⁠https://hubs.li/Q02RzJyL0 Try Preset for free, forever, at ⁠https://hubs.li/Q02RzMTd0 Learn more about how Preset can help your business: https://hubs.li/Q02RzWhM0

💻 ENGINEERING CORNER: New Iceberg integration Leaves are changing, lattes are getting spiced, and questionable sweaters are making a comeback 🍂 — but our demos are in style all year round. Watch our latest video from James Cipar on enabling #Iceberg support so you can stream directly from topics in Redpanda for advanced #SQL analysis ✌ https://lnkd.in/gKsyM6Jk

Are you in #fintech or financial services and using #kafka? You may be experiencing the following symptoms: 🔧 Endless #JVM tuning 😖 Overly-complex infrastructure 💰 Snowballing costs ❌ Inability to scale as needed We have something for that. 🏥 Redpanda is a leaner, easier-to-use #streamingdata platform that's fully compatible with Kafka APIs — without the usual Kafka costs or headaches. If you're ready to feel better about your real-time financial #data systems, check out this Redpanda vs. Kafka comparison. (Psst! It also tells you how to boost performance with 6x lower costs 💸) Read the blog here👇 https://lnkd.in/g7mE4Pfr

🎓 MASTERCLASS TOMORROW: Redpanda Connect 🎓 Redpanda is all about making things better and simpler — and Redpanda Connect is a prime example. So if you're a curious #developer looking for a better way to stream #data, join this interactive Masterclass to learn how to: 🔧 Set up pipelines with Redpanda Connect ⚙ Transform data in real time using #Bloblang 🎯 Handle external integrations 🚀 Implement advanced features (like retry queues and buffers) Interested? Mark your calendar for: 🗓 Thursday, October 3 🕘 9 am PDT | 12 pm EDT | 5 pm BST Sign up to join (or get the recording if you can't make it)!👇 https://lnkd.in/gtiuTRN2

🐾 REDPANDA SIGHTING: #MongoDBlocal If you're at MongoDB.local in #London, drop by our booth to chat about all things #streamingdata with our smart and friendly pandas (who couldn't agree on which of our soft shirts they should all wear). Happily enough, they have plenty of shirts to give away so you can enjoy them too. Great conversation and the cutest swag await!👇

A "Customer 360" is an all-in-one profile of a customer that aggregates #data from various touchpoints and interactions across a company's channels and platforms. In #telecom, this unified profile gives agents helpful context to better serve their customers, like recalling previous call history. Pretty handy, right? Good thing we have a tutorial on how to build a customer 360 view in just five steps using Redpanda and #Flink 💪 Follow along👇 https://lnkd.in/gw2s5Bai

Consumer rebalancing is the process where Kafka dynamically adjusts the partition assignments among consumers within a group. This can happen when consumers join, leave, or when partitions are added to the topic. In a broader sense, the following events can trigger a rebalance. ✔️ A new consumer joins the group. ✔️ An existing consumer leaves the group. ✔️The number of partitions in the topic changes. ✔️ Consumer group coordinator failure or session expiration. Consumer rebalancing is an advanced topic with many moving parts. However, let’s try to understand the basics with this example. 1️⃣ Initial Setup: We start with a Kafka topic having multiple partitions, say 5 partitions. Imagine two consumers (C1, C2) assigned to a topic in a consumer group. C1 gets P1, P2, P3, while C2 handles P4, P5. This is our baseline partition assignment. 2️⃣ Triggering a Rebalance: Next, we introduce a new consumer (C3) to the group. As C3 joins, Kafka triggers a rebalance. Partitions get redistributed across C1, C2, and C3—each consumer now gets a new set of partitions to ensure load balancing. During the rebalance, partitions shift smoothly—C1 releases some partitions (like P2 and P3), and C3 picks them up. Kafka dynamically adjusts the assignments for optimal performance. 3️⃣ What Happens When C2 Crashes? In case C2 crashes or disconnects, Kafka automatically triggers another rebalance. C1 and C3 take over C2’s partitions (P4, P5), ensuring no downtime in message consumption. The rebalancing ensures the consumer group remains resilient, distributing the load across the remaining consumers. 💡Partition assignment strategies Partition assignment strategy is a crucial aspect of Kafka consumer rebalancing that determines how partitions are distributed among consumers in a consumer group. It defines the algorithm used to allocate partitions to consumers when a rebalance occurs, ensuring efficient and balanced message consumption across all consumers in the group. 1️⃣ Round-robin assignment strategy - Kafka evenly distributes partitions across all consumers in a consumer group. This means that each consumer is assigned partitions in a sequential, cyclic manner, ensuring that the load is balanced as evenly as possible. For example, if there are 8 partitions and 3 consumers, each consumer will get 2-3 partitions in a rotating fashion, without considering past assignments. 2️⃣ Sticky assignment strategy - introduced in Kafka 2.4+, tries to minimize the number of partition reassignments during a rebalance. It keeps consumers attached to their previously assigned partitions as much as possible, only making necessary changes when new consumers join or partitions are added. This reduces disruption in message processing and provides more stability in high-throughput systems. #kafka #rebalancing #sketchnote