Apache Kafka
Distributed event log for streaming data between systems at scale.
What it is
Apache Kafka is a distributed streaming platform used for building real-time data pipelines and streaming applications. It provides high-throughput, fault-tolerant, and scalable messaging between producers and consumers.
Kafka allows developers to produce and consume messages to topics, stream processing using Kafka Streams, and integrate with other systems using Kafka Connect. It supports distributed, fault-tolerant, and scalable messaging architecture.
- Licence
- Apache 2.0
- Watch for
- Partition count sets your maximum consumer parallelism — plan it early
When to use it
The question documentation cannot answer for you — because it cannot recommend something else.
Reach for it when
- Event-driven architectures where multiple consumers need the same stream
- You need durable, replayable history rather than fire-and-forget messaging
- Throughput is high enough that a traditional broker would struggle
Look elsewhere when
- You need a simple work queue — RabbitMQ or Redis is far less operational burden
- The team cannot support the operational complexity Kafka brings
Installation
Add org.apache.kafka:kafka-clients dependency in pom.xmlGetting started
The smallest useful thing you can do with it, and what each part means.
import org.apache.kafka.clients.producer.*;
import java.util.Properties;
Properties props = new Properties();
props.put("bootstrap.servers", "localhost:9092");
props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");
props.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer");
KafkaProducer<String, String> producer = new KafkaProducer<>(props);
producer.send(new ProducerRecord<>("test-topic", "key1", "Hello Kafka"));
producer.close();import org.apache.kafka.clients.consumer.*;
import java.util.*;
Properties props = new Properties();
props.put("bootstrap.servers", "localhost:9092");
props.put("group.id", "test-group");
props.put("key.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
props.put("value.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props);
consumer.subscribe(Arrays.asList("test-topic"));
while(true) {
for (ConsumerRecord<String, String> record : consumer.poll(Duration.ofMillis(100))) {
System.out.println("Received: " + record.value());
}
}Advanced usage
Where the library earns its place over a simpler alternative.
import org.apache.kafka.streams.*;
import org.apache.kafka.streams.kstream.*;
Properties props = new Properties();
props.put(StreamsConfig.APPLICATION_ID_CONFIG, "streams-app");
props.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
StreamsBuilder builder = new StreamsBuilder();
KStream<String, String> source = builder.stream("input-topic");
source.mapValues(value -> value.toUpperCase()).to("output-topic");
KafkaStreams streams = new KafkaStreams(builder.build(), props);
streams.start();producer.send(new ProducerRecord<>("test-topic", "key1", "Message 1"));
producer.send(new ProducerRecord<>("test-topic", "key2", "Message 2"));// Multiple consumers in the same group share messages from partitions, enabling parallel processing.consumer.commitSync();Errors and fixes
The failures you are most likely to hit, and what actually resolves them.
- TimeoutException
- Occurs when the producer cannot send a message in time. Check broker availability and network connectivity.
- SerializationException
- Ensure that the key and value serializers/deserializers match the data types.
- GroupRebalanceException
- Occurs when consumers in a group are rebalancing. Handle gracefully in the consumer application.
Best practices
- Use appropriate key selection to control partitioning and ordering.
- Configure producer retries and acknowledgments for reliability.
- Use consumer groups for scaling processing.
- Monitor consumer lag to detect processing bottlenecks.
- Leverage Kafka Streams for complex transformations and stateful processing.
Alternatives
Comparable options, and the reason you would pick one over the other.
Background
Why it exists, and what it was reacting to.
Kafka was originally developed at LinkedIn to handle real-time activity stream data and later became an open-source project under the Apache Software Foundation. It enables applications to process and react to streams of events efficiently and reliably.