How to alter memory allocation pool for Kafka and ZooKeeper

Alter memory allocation pool for Kafka and ZooKeeper.

GC Logging for both of these services is enabled using KAFKA_HEAP_OPTS variable.

Kafka service.

$ cat bin/kafka-server-start.sh 

#!/bin/bash
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements.  See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License.  You may obtain a copy of the License at
#
#    http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

if [ $# -lt 1 ];
then
        echo "USAGE: $0 [-daemon] server.properties [--override property=value]*"
        exit 1
fi
base_dir=$(dirname $0)

if [ "x$KAFKA_LOG4J_OPTS" = "x" ]; then
    export KAFKA_LOG4J_OPTS="-Dlog4j.configuration=file:$base_dir/../config/log4j.properties"
fi

if [ "x$KAFKA_HEAP_OPTS" = "x" ]; then
    export KAFKA_HEAP_OPTS="-Xmx1G -Xms1G"
fi

EXTRA_ARGS=${EXTRA_ARGS-'-name kafkaServer -loggc'}

COMMAND=$1
case $COMMAND in
  -daemon)
    EXTRA_ARGS="-daemon "$EXTRA_ARGS
    shift
    ;;
  *)
    ;;
esac

exec $base_dir/kafka-run-class.sh $EXTRA_ARGS kafka.Kafka "$@"

ZooKeeper service.

$ cat bin/zookeeper-server-start.sh 

#!/bin/bash
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements.  See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License.  You may obtain a copy of the License at
#
#    http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

if [ $# -lt 1 ];
then
        echo "USAGE: $0 [-daemon] zookeeper.properties"
        exit 1
fi
base_dir=$(dirname $0)

if [ "x$KAFKA_LOG4J_OPTS" = "x" ]; then
    export KAFKA_LOG4J_OPTS="-Dlog4j.configuration=file:$base_dir/../config/log4j.properties"
fi

if [ "x$KAFKA_HEAP_OPTS" = "x" ]; then
    export KAFKA_HEAP_OPTS="-Xmx512M -Xms512M"
fi

EXTRA_ARGS=${EXTRA_ARGS-'-name zookeeper -loggc'}

COMMAND=$1
case $COMMAND in
  -daemon)
     EXTRA_ARGS="-daemon "$EXTRA_ARGS
     shift
     ;;
 *)
     ;;
esac

exec $base_dir/kafka-run-class.sh $EXTRA_ARGS org.apache.zookeeper.server.quorum.QuorumPeerMain "$@"

Alter KAFKA_HEAP_OPTS for the Kafka service variable to alter memory allocation pool.

$ cat /etc/systemd/system/kafka.service 

[Unit]
Description=Kafka Service
After=network-online.target 
Requires=network-online.target

[Service]
Environment="KAFKA_HEAP_OPTS=Xmx32G -Xms32G"
Type=simple
User=kafka
Group=kafka
WorkingDirectory=/opt/kafka
Restart=on-failure

ExecStart=/opt/kafka/kafka/bin/kafka-server-start.sh /opt/kafka/kafka/config/server.properties
ExecStop=/opt/kafka/kafka/bin/kafka-server-stop.sh /opt/kafka/kafka/config/server.properties

[Install]
WantedBy=multi-user.targe

Alter KAFKA_HEAP_OPTS for the ZooKeeper service variable to alter memory allocation pool.

$ cat /etc/systemd/system/zookeeper.service 

[Unit]
Description=ZooKeeper Service
After=network-online.target 
Requires=network-online.target

[Service]
Environment="KAFKA_HEAP_OPTS=Xmx1G -Xms1G"
Type=simple
User=kafka
Group=kafka
WorkingDirectory=/opt/kafka
Restart=on-failure

ExecStart=/opt/kafka/kafka/bin/zookeeper-server-start.sh /opt/kafka/kafka/config/zookeeper.properties
ExecStop=/opt/kafka/kafka/bin/zookeeper-server-stop.sh /opt/kafka/kafka/config/zookeeper.properties

[Install]
WantedBy=multi-user.target

It’s as simple as that.