TopicPartition
Podcast Description
A in-depth engineering podcast about Apache Kafka
Podcast Insights
Content Themes
Covers advanced topics related to Apache Kafka with a special emphasis on architectural innovations like KIP-1150, as well as discussions on data replication, batch coordination, and latency requirements. Episodes delve into specific concepts like diskless topics, caching strategies, and the implications of open-source contributions.
A in-depth engineering podcast about Apache Kafka
An interview with Greg Harris and Ivan Yurchenko, part of the authors of the new KIP-1150 that introduces diskless topics to Apache Kafka.
Diskless topics are a special kind of topic that write directly to an object store. This outsources the data replication and durability guarantees to the store (e.g., S3) and avoids the costly inter-zone data transfer fees that clouds charge you for. The result is a 80%+ reduced Kafka bill and a simpler architecture where brokers are stateless (or have less state) – making them significantly easier to maintain operationally.
In this podcast, we dive deep into the technical details and tradeoffs in this complex KIP.
TIMELINE
- 04:00 – Why Diskless?
- 09:17 – Architecture Walkthrough: How Does It Work
- 18:50 – The Batch Coordinator
- 22:00 – The Different Batch Coordinator Implementations
- 28:00 – The Inkless Fork (And Why)
- 39:40 – Topic-Based Batch Coordinator
- 45:40 – Bottlenecks in the Batch Coordinator?
- 47:48 – The Read Path
- 54:40 – Caching
- 01:03:00 – Shared Log Segment Merging (AKA Object Compaction)
- 01:19:06 – Latency
- 01:26:30 – Practical Real-World Latency Requirements
- 01:39:00 – The Power of Open Source
- 01:47:30 – Cost Breakdown
- 01:58:00 – Ops and Engineering Time Savings
- 02:06:30 – Client Bootstrapping, Leaderless Partitions, One Broker Connection per Client
- 02:11:20 – Multi-Tenant Kafka Future
- 02:14:05 – Iceberg, Parquet, and First-Class Schema Support
- 02:23:10 – S3 Express
If you found this episode interesting, please give it a like to signal to the algorithm that it’s good. It takes 2 seconds to do, and it took us 5 hours to produce.
HOW TO GET INVOLVED
We recommend participating in the Apache Kafka mailing list.
Beyond that, you can find Greg and Ivan here:
- Ivan Yurchenko – X: @ivan0yu
- Greg Harris – X: @gharris1727
DISCLAIMERS
All views expressed in this podcast are personal opinions of the participants. They do not represent the views or positions of Aiven, the employer of Ivan and Greg, or any other entity.
Apache®, Apache Kafka®, Kafka, and the Kafka logo are either registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries. No endorsement by The Apache Software Foundation is implied by the use of these marks.
Disclaimer
This podcast’s information is provided for general reference and was obtained from publicly accessible sources. The Podcast Collaborative neither produces nor verifies the content, accuracy, or suitability of this podcast. Views and opinions belong solely to the podcast creators and guests.
For a complete disclaimer, please see our Full Disclaimer on the archive page. The Podcast Collaborative bears no responsibility for the podcast’s themes, language, or overall content. Listener discretion is advised. Read our Terms of Use and Privacy Policy for more details.