Manage Kafka topics using command-line utilities.

This blog post is based on my experience with the Ansible automation engine, but I will keep things agnostic to any particular automation software, so you can use the right approach for your own setup.

Perform these operations on a single server, like the first server in a group.

List topics

The most important thing is to know what topics are already created.

$ bin/kafka-topics.sh --bootstrap-server localhost:9092 --list --exclude-internal

cerberus
kraken

Notice, the additional parameter to exclude internal topics like __consumer_offsets.

Create topic

Add a topic if it is not created.

$ bin/kafka-topics.sh --bootstrap-server localhost:9092 --create --topic medusa --replication-factor 1

Created topic medusa.

Use multiple config parameters to define additional configuration options.

$ bin/kafka-topics.sh --bootstrap-server localhost:9092 --create --topic hydra --replication-factor 3 --config retention.ms=86400000 --config retention.bytes=1048576

Created topic hydra.

Delete topic

Delete topic if it is marked as to be removed or it is not defined, remember to exclude internal topics from a list.

$ bin/kafka-topics.sh --bootstrap-server localhost:9092 --delete --topic medusa

Update topic configuration

You should update the topic list as things could get altered in the meantime.

$ bin/kafka-topics.sh --bootstrap-server localhost:9092 --list --exclude-internal

cerberus
hydra
kraken

Extract topic configuration and store it in a temporary variable.

$ bin/kafka-topics.sh --bootstrap-server localhost:9092 --describe --topic hydra  | head -1 | tr '\t' '\n'

Topic: hydra
PartitionCount: 1
ReplicationFactor: 3
Configs: segment.bytes=1073741824,retention.ms=86400000,message.format.version=2.0-IV1,retention.bytes=1048576

Inspect the number of partitions.

$ bin/kafka-topics.sh --bootstrap-server localhost:9092 --describe --topic hydra | head -1 | tr '\t' '\n' | \
  awk -F: '$1=="PartitionCount" { print $2}' | tr -d ' '

1

Update the number of partitions, increase only.

$ bin/kafka-topics.sh --bootstrap-server localhost:9092 --alter --topic hydra --partitions 10

Inspect additional configuration.

$ bin/kafka-topics.sh --bootstrap-server localhost:9092 --describe --topic hydra | head -1 | tr '\t' '\n' | awk -F: '$1=="Configs" { print $2}' | tr ',' '\n' | sed 's/ //'

segment.bytes=1073741824
retention.ms=86400000
message.format.version=2.0-IV1
retention.bytes=1048576

Iterate over each additional configuration option. Exit code will clearly indicate if the change is needed.

$ bin/kafka-topics.sh --bootstrap-server localhost:9092 --describe --topic hydra | head -1 | tr '\t' '\n' | awk -F: '$1=="Configs" { print $2}' | tr ',' '\n' | sed 's/ //' | grep ^retention.bytes=1048576$

retention.bytes=1048576

$ echo $?

0

$ bin/kafka-topics.sh --bootstrap-server localhost:9092 --describe --topic hydra | head -1 | tr '\t' '\n' | awk -F: '$1=="Configs" { print $2}' | tr ',' '\n' | sed 's/ //' | grep ^retention.ms=43200000$

$ echo $?

1

Update or add an additional parameter.

$ bin/kafka-configs.sh --bootstrap-server localhost:9092 --alter --topic hydra --add-config retention.ms=43200000

Completed updating config for topic hydra.

Delete excessive configuration options, but use a whitelist to know what will change to the default value.

$ bin/kafka-configs.sh --bootstrap-server localhost:9092 --alter --topic hydra --delete-config retention.bytes

Completed updating config for topic hydra.

You do not need any external plugins or utilities as these operations can be easily automated using basic shell commands.

Read the documentation and help information for each command, everything important is mentioned there.