May 2024 Newsletter - Real-Time Fraud Detection at Instacart, ClickHouse Developer Certification, K-Means Clustering
Welcome to the May ClickHouse newsletter, which will round up what’s been happening in real-time data warehouses over the last month.
This month, we have recursive CTEs in the 24.4 release, the launch of ClickHouse developer certification, real-time fraud detection at Instacart, and more!
Inside this issue
Featured community member
This month's featured community member is Daniel Goodman , co-founder and CEO of Tangia, a service that hosts interactive live streams.
Dan has been part of the ClickHouse community for at least 18 months and frequently gives the engineering team feedback on missing features and how existing ones can be improved.
Dan writes a blog about distributed systems, where he’s previously written about topics like range partitioning and building a Fly.io scheduler.
A few weeks ago he wrote a blog post titled Finding Trends With Approximate Embedding Clustering. In the post, he explains the importance of approximation techniques when working with big datasets and walks through how to implement the Dynamic K-Means++ algorithm with ClickHouse.
Upcoming events
Dubai Meetup - May 28th
AWS Summit Dubai - May 29th
v24.5 Community Call - May 30th
San Francisco Meetup - June 4th
AWS Summit Stockholm - June 4th
Tokyo Meetup - June 5th
AWS Summit Madrid - June 5th
ClickHouse Fundamentals - June 26th & 27th
AWS Summit D.C. - June 26th
Amsterdam Meetup - June 27th
Paris Meetup - July 9th
New York Meetup - July 9th
24.4 release
The standout feature in the 24.4 release is recursive CTEs (Common Table Expressions), and we made a London Underground-themed example to show you how they work. This release also sees improvements to JOIN performance and the QUALIFY clause to filter the values of window functions.
Become a ClickHouse Certified Developer
Rich Raposa recently announced the launch of the official ClickHouse Developer Certification Program, the only certification directly from ClickHouse.
This certification program validates developers’ proficiency in using ClickHouse to build robust, high-performance data solutions. This certification will showcase your mastery of ClickHouse and help you distinguish yourself as a trusted database management and analytics expert.
Learn more about certification
Real-time Fraud Detection at Instacart
Nicholas Shieh, Shen Zhu, and Xiaobing Xia have written a blog post that walks us through Yoda, a decision platform service they built at Instacart to detect fraudulent activities and take action quickly. ClickHouse was chosen as the primary real-time datastore for this system because it can both ingest and query large amounts of data in real time. I especially liked the part of the post where they describe how real-time features fed into the service are derived from ClickHouse SQL queries.
K-Means Clustering with ClickHouse
Recently, when helping a user who wanted to compute centroids from vectors held in ClickHouse, we realized that the same solution could be used to implement K-Means clustering. They wanted to do this at scale across potentially billions of data points while ensuring memory could be tightly managed. In this post, we show how to implement K-means clustering using just SQL and show that it can scale to billions of records while running significantly faster than the same code in scikit-learn.
Simplified Kubernetes Logging with Fluentbit and ClickHouse
Logging is one of the hot ClickHouse use cases of the moment, so I was excited to come across this blog post by Muthukumaran. Fluentbit is a lightweight logging and metrics processor and forwarder designed for containerized environments. Muthukumaran walks us through the steps to setup a metrics server to monitor resource utilization in Kubernetes and then shows how to configure Fluentbit to get those metrics into ClickHouse.
The New Building Blocks of Observability
This article focuses on what the author coins the three new elements in the observability period table: OpenTelemetry, eBPF, and ClickHouse. OpenTelemetry has emerged as the de facto standard for telemetry data, eBPF makes it possible to generate traces and metrics with zero instrumentation, and ClickHouse is used to ingest and query all this data. The article also covers a series of Observability startups that are using ClickHouse - Groundcover, SigNoz, and DeepFlow.
Using ClickHouse for Financial Charts
After giving a brief crash course into when (and when not) to use ClickHouse, Adis Nezirović demonstrates how to ingest, query, and visualize financial time-series data. Along the way, he shows how to use the Null table engine to massage data and aggregate states to reduce the amount of data kept around. To conclude, Adis creates a candlestick chart using the Grafana QueryBuilder.
Post of the Month
Our favorite post this month was by ludwig who was impressed by both the speed of ClickHouse queries and the quality of its data compression.
After giving a brief crash course into when (and when not) to use ClickHouse, Adis Nezirović demonstrates how to ingest, query, and visualize financial time-series data. Along the way, he shows how to use the Null table engine to massage data and aggregate states to reduce the amount of data kept around. To conclude, Adis creates a candlestick chart using the Grafana QueryBuilder.