Install Cluster Manager for Apache Kafka, previously known as Kafka Manager on Debian Bullseye.

I am deliberately building a package as it is a most consistent and complete solution.
Build a package
Notice, the whole process requires at least 2GB of RAM.
Update package index.
$ sudo apt update
Install basic utilities required to build a package.
$ sudo apt install git fakeroot
Install application dependencies.
$ sudo apt install openjdk-11-jre-headless nodejs
Clone source code. I will use 3.0.0.5
tag.
$ git clone --depth 1 --branch 3.0.0.5 https://github.com/yahoo/CMAK.git
Change working directory.
$ cd CMAK
Inspect default configuration file conf/application.conf
.
Create CMAK package.
$ ./sbt debian:packageBin
Inspect created package.
$ ls -lh target/cmak_3.0.0.5_all.deb
-rw-r--r-- 1 vagrant vagrant 137M Aug 3 20:53 target/cmak_3.0.0.5_all.deb
$ dpkg --info target/cmak_3.0.0.5_all.deb
new Debian package, version 2.0. size 142837952 bytes: control archive=10240 bytes. 223 bytes, 7 lines conffiles 236 bytes, 10 lines control 1240 bytes, 60 lines * postinst #!/bin/sh 109 bytes, 4 lines * postrm #!/bin/sh 109 bytes, 4 lines * preinst #!/bin/sh 1184 bytes, 59 lines * prerm #!/bin/sh Source: cmak Package: cmak Version: 3.0.0.5 Section: java Priority: optional Architecture: all Installed-Size: 138805 Maintainer: Yahoo <yahoo@example.com> Description: A tool for managing Apache Kafka A tool for managing Apache Kafka
You can now store and share this package.
Install and configure package
Install application dependencies.
$ sudo apt install openjdk-11-jre-headless
Install created package.
$ sudo dpkg --install cmak_3.0.0.5_all.deb
Edit configuration file to at least define Zookeeper hosts that will store cluster configurations and basic authentication settings.
$ vi /etc/cmak/application.conf
# Copyright 2015 Yahoo Inc. Licensed under the Apache License, Version 2.0 # See accompanying LICENSE file. # This is the main configuration file for the application. # ~~~~~ # Secret key # ~~~~~ # The secret key is used to secure cryptographics functions. # If you deploy your application to several instances be sure to use the same key! play.crypto.secret="^<csmm5Fx4d=r2HEX8pelM3iBkFVv?k[mc;IZE<_Qoq8EkX_/7@Zt6dP05Pzea3U" play.crypto.secret=${?APPLICATION_SECRET} play.http.session.maxAge="1h" # The application languages # ~~~~~ play.i18n.langs=["en"] play.http.requestHandler = "play.http.DefaultHttpRequestHandler" play.http.context = "/" play.application.loader=loader.KafkaManagerLoader # Settings prefixed with 'kafka-manager.' will be deprecated, use 'cmak.' instead. # https://github.com/yahoo/CMAK/issues/713 kafka-manager.zkhosts="kafka-manager-zookeeper:2181" kafka-manager.zkhosts=${?ZK_HOSTS} #cmak.zkhosts="kafka-manager-zookeeper:2181" cmak.zkhosts="172.16.0.101:2181,172.16.0.102:2181,172.16.0.101:2181" cmak.zkhosts=${?ZK_HOSTS} pinned-dispatcher.type="PinnedDispatcher" pinned-dispatcher.executor="thread-pool-executor" application.features=["KMClusterManagerFeature","KMTopicManagerFeature","KMPreferredReplicaElectionFeature","KMReassignPartitionsFeature", "KMScheduleLeaderElectionFeature"] akka { loggers = ["akka.event.slf4j.Slf4jLogger"] loglevel = "INFO" } akka.logger-startup-timeout = 60s #basicAuthentication.enabled=false basicAuthentication.enabled=true basicAuthentication.enabled=${?KAFKA_MANAGER_AUTH_ENABLED} basicAuthentication.ldap.enabled=false basicAuthentication.ldap.enabled=${?KAFKA_MANAGER_LDAP_ENABLED} basicAuthentication.ldap.server="" basicAuthentication.ldap.server=${?KAFKA_MANAGER_LDAP_SERVER} basicAuthentication.ldap.port=389 basicAuthentication.ldap.port=${?KAFKA_MANAGER_LDAP_PORT} basicAuthentication.ldap.username="" basicAuthentication.ldap.username=${?KAFKA_MANAGER_LDAP_USERNAME} basicAuthentication.ldap.password="" basicAuthentication.ldap.password=${?KAFKA_MANAGER_LDAP_PASSWORD} basicAuthentication.ldap.search-base-dn="" basicAuthentication.ldap.search-base-dn=${?KAFKA_MANAGER_LDAP_SEARCH_BASE_DN} basicAuthentication.ldap.search-filter="(uid=$capturedLogin$)" basicAuthentication.ldap.search-filter=${?KAFKA_MANAGER_LDAP_SEARCH_FILTER} basicAuthentication.ldap.group-filter="" basicAuthentication.ldap.group-filter=${?KAFKA_MANAGER_LDAP_GROUP_FILTER} basicAuthentication.ldap.connection-pool-size=10 basicAuthentication.ldap.connection-pool-size=${?KAFKA_MANAGER_LDAP_CONNECTION_POOL_SIZE} basicAuthentication.ldap.ssl=false basicAuthentication.ldap.ssl=${?KAFKA_MANAGER_LDAP_SSL} basicAuthentication.ldap.ssl-trust-all=false basicAuthentication.ldap.ssl-trust-all=${?KAFKA_MANAGER_LDAP_SSL_TRUST_ALL} basicAuthentication.username="admin" basicAuthentication.username=${?KAFKA_MANAGER_USERNAME} basicAuthentication.password="password" basicAuthentication.password=${?KAFKA_MANAGER_PASSWORD} basicAuthentication.realm="Kafka-Manager" basicAuthentication.excluded=["/api/health"] # ping the health of your instance without authentification kafka-manager.consumer.properties.file=${?CONSUMER_PROPERTIES_FILE}
Inspect service state.
$ sudo systemctl status cmak
● cmak.service - A tool for managing Apache Kafka Loaded: loaded (/lib/systemd/system/cmak.service; enabled; vendor preset: enabled) Active: active (running) since Tue 2021-08-03 21:03:35 UTC; 1min 42s ago Process: 2375 ExecStartPre=/bin/mkdir -p /run/cmak (code=exited, status=0/SUCCESS) Process: 2376 ExecStartPre=/bin/chown root:root /run/cmak (code=exited, status=0/SUCCESS) Process: 2377 ExecStartPre=/bin/chmod 755 /run/cmak (code=exited, status=0/SUCCESS) Main PID: 2378 (java) Tasks: 59 (limit: 2311) Memory: 250.4M CPU: 11.557s CGroup: /system.slice/cmak.service └─2378 java -Dpidfile.path=/var/run/cmak.pid -Dconfig.file=/etc/cmak/application.conf -Dlogger.file=/etc/cmak/logger.xml -Duser.dir=/usr/share/cmak -cp /> Aug 03 21:03:35 bullseye systemd[1]: Starting A tool for managing Apache Kafka... Aug 03 21:03:35 bullseye systemd[1]: Started A tool for managing Apache Kafka.
Inspect default service settings.
$ cat /etc/default/cmak
# ##################################### # ##### Environment Configuration ##### # ##################################### # This file gets sourced before the actual bashscript # gets executed. You can use this file to provide # environment variables # Available replacements # ------------------------------------------------ # Yahoo <yahoo@example.com> package author # A tool for managing Apache Kafka package description # cmak startup script name # /usr/share/cmak app directory # 0 retries for startup # 60 retry timeout # cmak normalized app name # root daemon user # ------------------------------------------------- # Setting JAVA_OPTS # ----------------- JAVA_OPTS="-Dpidfile.path=/var/run/cmak.pid -Dconfig.file=/etc/cmak/application.conf -Dlogger.file=/etc/cmak/logger.xml" # Setting PIDFILE # --------------- PIDFILE="/var/run/cmak.pid"
Inspect log file to identify potential issues.
$ tail /var/log/cmak/application.log
2021-08-03 21:03:38,190 - [INFO] - from akka.event.slf4j.Slf4jLogger in application-akka.actor.default-dispatcher-5 Slf4jLogger started 2021-08-03 21:03:38,313 - [WARN] - from application in main /etc/cmak/application.conf: 12: play.crypto.secret is deprecated, use play.http.secret.key instead 2021-08-03 21:03:38,478 - [INFO] - from akka.event.slf4j.Slf4jLogger in kafka-manager-system-akka.actor.default-dispatcher-5 Slf4jLogger started 2021-08-03 21:03:38,593 - [INFO] - from kafka.manager.actor.KafkaManagerActor in kafka-manager-system-akka.actor.default-dispatcher-3
Potential issues
Using master
branch can get you in trouble. I have encountered the Unimplemented for /kafka-manager/mutex
issue described in Failed to add cluster on 3.0.0.1 #731.
The solution is to create missing znodes, so cluster management will work.
$ bin/zookeeper-shell.sh localhost:2181 create /kafka-manager ""
$ bin/zookeeper-shell.sh localhost:2181 create /kafka-manager/mutex ""
$ bin/zookeeper-shell.sh localhost:2181 create /kafka-manager/mutex/locks ""
$ bin/zookeeper-shell.sh localhost:2181 create /kafka-manager/mutex/leases ""
Also, each cluster requires the same znodes.
$ bin/zookeeper-shell.sh localhost:2181 create /kafka-manager/clusters/kafka/mutex ""
$ bin/zookeeper-shell.sh localhost:2181 create /kafka-manager/clusters/kafka/mutex/locks ""
$ bin/zookeeper-shell.sh localhost:2181 create /kafka-manager/clusters/kafka/mutex/leases ""
Alternatively use bin/zkCli.sh
utility to perform these operations.
Additional notes
Use nginx
or a similar application to secure it further.
By default service will be available on port 9000
, alter it using using JAVA_OPTS
in /etc/default/cmak
configuration file.
[...] # Setting JAVA_OPTS # ----------------- JAVA_OPTS="-Dhttp.address=127.0.0.1 -Dhttp.port=3000 -Dpidfile.path=/var/run/cmak.pid -Dconfig.file=/etc/cmak/application.conf -Dlogger.file=/etc/cmak/logger.xml" [...]
This is really cool!