How to generate Kafka cluster ID

Generate Kafka cluster ID for Kafka Raft metadata mode.

Kafka cluster ID is expected to contain 16 bytes of a base64-encoded UUID.

You can generate cluster ID using official utilities.

$ cd /opt/kafka/kafka/

$./bin/kafka-storage.sh random-uuid

p8fFEbKGQ22B6M_Da_vCBw

Alternatively, use proc filesystem, which comes in handy when you want to dynamically generate multiple clusters without locally extracting Kafka archive to generate UUIDs.

$ cat /proc/sys/kernel/random/uuid | tr -d '-' | base64 | cut -b 1-22

ZGI1NTk0YmY3NzVjNDk5MD

There is also uuidgen program, which gives you more control over the process as you can generate random-based or time-based UUIDs.

$ apt info uuid-runtime

Package: uuid-runtime
Version: 2.36.1-7
Priority: optional
Section: utils
Source: util-linux
Maintainer: util-linux packagers <util-linux@packages.debian.org>
Installed-Size: 224 kB
Pre-Depends: libuuid1 (>= 2.25-5~), init-system-helpers (>= 1.54~)
Depends: adduser, libc6 (>= 2.17), libsmartcols1 (>= 2.27~rc1), libsystemd0, libuuid1 (>= 2.31.1)
Homepage: http://www.kernel.org/pub/linux/utils/util-linux/
Tag: role::program
Download-Size: 101 kB
APT-Manual-Installed: yes
APT-Sources: http://deb.debian.org/debian bullseye/main amd64 Packages
Description: runtime components for the Universally Unique ID library
 The libuuid library generates and parses 128-bit Universally Unique
 IDs (UUIDs). A UUID is an identifier that is unique within the space
 of all such identifiers across both space and time. It can be used for
 multiple purposes, from tagging objects with an extremely short lifetime
 to reliably identifying very persistent objects across a network.
 .
 See RFC 4122 for more information.
 .
 This package contains the uuidgen program and the uuidd daemon.
 .
 The uuidd daemon is used to generate UUIDs, especially time-based
 UUIDs, in a secure and guaranteed-unique fashion, even in the face of
 large numbers of threads trying to grab UUIDs running on different CPUs.
 It is used by libuuid as well as the uuidgen program.

Install uuid-runtime.

$ sudo apt-get install uuid-runtime

Inspect manual page.

$ man uuidgen

[...]
       -r, --random
              Generate a random-based UUID.  This method creates a UUID consisting mostly of random bits.  It requires that the operating system have a high quality random number generator, such as /dev/random.

       -t, --time
              Generate a time-based UUID.  This method creates a UUID based on the system clock plus the system's ethernet hardware address, if present.
[...]

Generate time-based UUID, it does not meet our requirements at this time as it is too long.

$ uuidgen --time

20a0978e-d10a-11eb-b689-83f5826e0c4b

Shorten it a bit to meet our requirements.

$ uuidgen --time | tr -d '-' | base64 | cut -b 1-22 

N2Q0ZTNlMWFhMTgyNDhiZj

Now it can be used to format storage.

$ sudo -u kafka /opt/kafka/kafka/bin/kafka-storage.sh format -t N2Q0ZTNlMWFhMTgyNDhiZj -c /opt/kafka/kafka/config/kraft/server.properties

Formatting /opt/kafka/kafka/kafka_data

$ sudo -u kafka cat /opt/kafka/kafka/kafka_data/meta.properties

#
#Sat Jun 19 14:06:49 UTC 2021
node.id=1
version=1
cluster.id=N2Q0ZTNlMWFhMTgyNDhiZj