Kafka is a high-speed event streaming platform developed by Confluent (based on the Apache Kafka project). It lets systems publish and consume messages asynchronously, with high availability and horizontal scalability.
How it works
Kafka is a distributed message queue system. Producers send messages to topics, and consumers subscribe to topics to receive the messages. Each topic is split into partitions, which enables scalability and parallelism. Kafka persists messages for a configurable retention period, so consumers can replay unprocessed messages.
Practical example
In an e-commerce site, events like clicks, add to cart, payment and delivery are sent to Kafka via topics. Different systems consume them, such as an analytics dashboard, stock alerts, recommendation systems and confirmation emails. Each system consumes only the events it cares about, without overloading the core business code.
Related terms
- message queue: asynchronous messaging system
- streaming: continuous event processing
- ETL: extract, transform, load (Kafka is used in data pipelines)
How to test today
Use the Kafka documentation and run a producer with Python, sending messages to a topic. Then consume those messages with another script. In cloud environments, move Kafka to a Kafka Cloud Service.
Frequently asked questions
- What is Apache Kafka? It is a high-speed event streaming platform used to integrate systems, message queuing and real-time data processing.
- Is Kafka just a message queue? No. Kafka is an event streaming system with persistence, replay, partitioning and consumer control, plus support for stream processing (KStreams, Kafka Streams).
- When should you use Kafka? In business systems that need reliability, scalability and real-time data processing, such as dashboards, events, IoT and analytics.