Apache Kafka Cheat Sheet — Commands, Concepts, and Configuration Reference
A complete Apache Kafka cheat sheet covering CLI commands, producer/consumer code, topic management, consumer groups, offsets, and key configuration properties. Bookmark this for daily Kafka work.
kafka-topics
topic management CLI tool
kafka-console-*
producer/consumer CLI tools
__consumer_offsets
internal topic that tracks consumption progress
KafkaAdmin
programmatic administration Java API
Core Kafka Concepts
| Item | Concept | Description |
|---|---|---|
| Topic | Named stream of events | Like a database table — events are organized by topic name |
| Partition | Ordered sequence within a topic | Enables parallelism — each consumer in a group processes one partition at a time |
| Offset | Position of a message in a partition | Monotonically increasing integer — consumers track their own position per partition |
| Producer | Writes events to topics | Can specify partition key for ordering guarantees within a partition |
| Consumer | Reads events from topics | Commits offsets to Kafka to track progress and enable resume after restart |
| Consumer Group | Set of consumers sharing the work | Each partition assigned to exactly one consumer in the group — enables parallel processing |
| Broker | Kafka server node | Stores and serves partitions — a cluster has multiple brokers for fault tolerance |
| ZooKeeper/KRaft | Cluster coordination | ZooKeeper (legacy mode), KRaft (Kafka 2.8+ built-in coordinator — no ZooKeeper needed) |
| Replication Factor | Number of copies per partition | Factor of 3 means partition exists on 3 brokers — survives 2 broker failures |
Topic Management CLI
# Create a topic
kafka-topics.sh --bootstrap-server localhost:9092 \
--create --topic my-topic \
--partitions 6 \
--replication-factor 3
# List all topics
kafka-topics.sh --bootstrap-server localhost:9092 --list
# Describe a topic (shows partitions, leaders, replicas, ISR)
kafka-topics.sh --bootstrap-server localhost:9092 \
--describe --topic my-topic
# Delete a topic
kafka-topics.sh --bootstrap-server localhost:9092 \
--delete --topic my-topic
# Increase partition count (can only increase, never decrease!)
kafka-topics.sh --bootstrap-server localhost:9092 \
--alter --topic my-topic --partitions 12
# Change retention to 24 hours (86400000 ms)
kafka-configs.sh --bootstrap-server localhost:9092 \
--alter --entity-type topics --entity-name my-topic \
--add-config retention.ms=86400000
# Change max message size
kafka-configs.sh --bootstrap-server localhost:9092 \
--alter --entity-type topics --entity-name my-topic \
--add-config max.message.bytes=5242880 # 5MBProduce and Consume from CLI
# Produce messages (type messages, press Enter after each, Ctrl+C to stop)
kafka-console-producer.sh --bootstrap-server localhost:9092 \
--topic my-topic
# Produce with keys (format: key:value)
kafka-console-producer.sh --bootstrap-server localhost:9092 \
--topic my-topic \
--property "key.separator=:" \
--property "parse.key=true"
# Produce JSON messages from a file
cat messages.json | kafka-console-producer.sh \
--bootstrap-server localhost:9092 --topic my-topic
# Consume from beginning (replay all messages)
kafka-console-consumer.sh --bootstrap-server localhost:9092 \
--topic my-topic --from-beginning
# Consume with a consumer group (commits offsets — resumes on restart)
kafka-console-consumer.sh --bootstrap-server localhost:9092 \
--topic my-topic --group my-consumer-group
# Consume and show keys, timestamps, partitions
kafka-console-consumer.sh --bootstrap-server localhost:9092 \
--topic my-topic --from-beginning \
--property print.key=true \
--property print.timestamp=true \
--property print.partition=trueJava Producer and Consumer Code
Properties props = new Properties();
props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class);
props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class);
// Idempotent producer — exactly-once delivery within a session
props.put(ProducerConfig.ENABLE_IDEMPOTENCE_CONFIG, true);
props.put(ProducerConfig.ACKS_CONFIG, "all"); // Wait for all in-sync replicas
props.put(ProducerConfig.RETRIES_CONFIG, 3);
props.put(ProducerConfig.LINGER_MS_CONFIG, 5); // Wait 5ms to batch messages
props.put(ProducerConfig.BATCH_SIZE_CONFIG, 16384); // 16KB batch size
try (KafkaProducer<String, String> producer = new KafkaProducer<>(props)) {
ProducerRecord<String, String> record =
new ProducerRecord<>("my-topic", "user-123", "{"event":"login"}");
// Async send with callback
producer.send(record, (metadata, exception) -> {
if (exception != null) {
log.error("Send failed", exception);
} else {
log.info("Sent to partition={} offset={}", metadata.partition(), metadata.offset());
}
});
producer.flush(); // Ensure all buffered records are sent before close
}Properties props = new Properties();
props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
props.put(ConsumerConfig.GROUP_ID_CONFIG, "my-consumer-group");
props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
props.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest"); // Start from beginning for new groups
props.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, false); // Manual commit for reliability
props.put(ConsumerConfig.MAX_POLL_RECORDS_CONFIG, 500); // Process 500 records per poll
try (KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props)) {
consumer.subscribe(List.of("my-topic"));
while (true) {
ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(100));
for (ConsumerRecord<String, String> record : records) {
System.out.printf("partition=%d offset=%d key=%s value=%s%n",
record.partition(), record.offset(), record.key(), record.value());
// Process record here
}
if (!records.isEmpty()) {
consumer.commitSync(); // Commit after successfully processing the batch
}
}
}Consumer Group Management
# List all consumer groups
kafka-consumer-groups.sh --bootstrap-server localhost:9092 --list
# Describe group — shows lag per partition (KEY command for monitoring)
kafka-consumer-groups.sh --bootstrap-server localhost:9092 \
--describe --group my-consumer-group
# Output: TOPIC | PARTITION | CURRENT-OFFSET | LOG-END-OFFSET | LAG
# Reset offsets to beginning (replay all messages — requires group to be stopped)
kafka-consumer-groups.sh --bootstrap-server localhost:9092 \
--group my-consumer-group --topic my-topic \
--reset-offsets --to-earliest --execute
# Reset to specific offset
kafka-consumer-groups.sh --bootstrap-server localhost:9092 \
--group my-consumer-group --topic my-topic:0 \
--reset-offsets --to-offset 1000 --execute
# Reset to a specific timestamp (replay last hour)
kafka-consumer-groups.sh --bootstrap-server localhost:9092 \
--group my-consumer-group --topic my-topic \
--reset-offsets --to-datetime 2026-03-25T10:00:00.000 --execute
# Delete a consumer group (must be inactive)
kafka-consumer-groups.sh --bootstrap-server localhost:9092 \
--delete --group my-consumer-groupKey Configuration Reference
Production config essentials
Kafka has hundreds of configuration properties. These are the ones that matter most for production reliability and performance. Wrong defaults are the #1 cause of data loss and throughput problems in production Kafka deployments.
acks=all (Producer)
Producer waits for all in-sync replicas to acknowledge the write. Maximum durability guarantee. Use for critical data where loss is unacceptable. Slight latency cost.
min.insync.replicas=2 (Broker)
Require at least 2 replicas to be in-sync before acknowledging. Combined with acks=all, prevents data loss when one broker fails. Standard for production.
retention.ms=604800000 (Topic)
Keep messages for 7 days (default). Set higher for audit logs and event replay. Set lower (hours) for high-volume transient data to control disk usage.
auto.offset.reset=earliest (Consumer)
New consumer groups start from the beginning of the topic. Use "latest" to only read new messages that arrive after the consumer starts.
max.poll.interval.ms=300000 (Consumer)
Max time between poll() calls before the consumer is considered dead and its partitions reassigned. Increase for slow processing jobs (ETL, batch operations).
compression.type=snappy (Producer)
Compress message batches before sending. Snappy is fast with good compression ratio. Use lz4 for maximum throughput. Use gzip for maximum compression.
Kafka Monitoring — Key Metrics
| Item | Metric | What It Tells You + Alert Threshold |
|---|---|---|
| Consumer Lag | Messages in partition not yet consumed | Alert if growing consistently — consumer is falling behind producers |
| Under-Replicated Partitions | Partitions not fully replicated | Alert if > 0 — a broker is down or struggling |
| Active Controller Count | Should always be 1 | Alert if ≠ 1 — controller election problem |
| Produce Request Rate | Messages written per second | Capacity planning — approaching broker limits |
| Fetch Request Latency | Time for consumers to fetch batches | Alert if > 1s — broker overload or network issue |
| Log Flush Rate | Disk flush frequency | High rate indicates disk I/O bottleneck on brokers |
Common Kafka Patterns
Event Sourcing — store all state changes as events
Every state change (order placed, payment received, item shipped) is written as an immutable event to Kafka. The current state of any entity is the sum of all its events. Kafka's retention and replay make this natural — consumers can rebuild state from the beginning.
CQRS with Kafka Streams
Separate write model (events to Kafka) from read model (Kafka Streams aggregations into a state store or read database). The read model is a projection of the event stream, rebuilt by replaying events. Multiple read models can consume the same event stream independently.
Saga Pattern for Distributed Transactions
Coordinate multi-service transactions using Kafka as the event bus. Each service listens for commands, executes locally, and publishes a success or failure event. A saga orchestrator or choreography pattern coordinates the sequence and handles compensating actions on failure.
Dead Letter Queue (DLQ) for Failed Messages
When a consumer fails to process a message after N retries, publish it to a separate dead-letter topic. A separate consumer processes DLQ messages with alerting and manual intervention. Prevents one bad message from blocking the entire consumer group.