Alter Log4j configuration to rotate Kafka logs.

By default Kafka uses DailyRollingFileAppender in config/log4j.properties for each configuration file.

[...]

log4j.appender.kafkaAppender=org.apache.log4j.DailyRollingFileAppender
log4j.appender.kafkaAppender.DatePattern='.'yyyy-MM-dd-HH
log4j.appender.kafkaAppender.File=${kafka.logs.dir}/server.log                                                                 
log4j.appender.kafkaAppender.layout=org.apache.log4j.PatternLayout
log4j.appender.kafkaAppender.layout.ConversionPattern=[%d] %p %m (%c)%n

[...]

It creates multiple log files according to the defined date pattern, but these files are not deleted after time passes.

controller.log                log-cleaner.log.2021-06-27-16  state-change.log.2021-06-27-16
controller.log.2021-06-27-16  server.log                     state-change.log.2021-06-27-18
controller.log.2021-06-28-06  server.log.2021-06-27-16       state-change.log.2021-06-28-06
kafka-authorizer.log          server.log.2021-06-27-18       state-change.log.2021-06-28-19
kafka-request.log             server.log.2021-06-28-06       state-change.log.2021-06-28-22
kafkaServer-gc.log            server.log.2021-06-28-19       state-change.log.2021-06-29-19
kafkaServer-gc.log.0          server.log.2021-06-30-00       state-change.log.2021-06-30-00
log-cleaner.log               state-change.log               zookeeper-gc.log

The best possible solution is to alter the Log4j configuration to send these logs to a central logging place, but this blog post focuses on a local log retention.

The first solution

Display log files modified more than (for example) 2 days ago.

$ find /opt/kafka/kafka/logs/ -maxdepth 1 -regextype egrep  -regex ".*/[a-z\-]+.log.[0-9]{4}-[0-9]{2}-[0-9]{2}-[0-9]{2}$" -mtime +2 -print

/opt/kafka/kafka/logs/state-change.log.2021-06-27-18
/opt/kafka/kafka/logs/server.log.2021-06-27-16
/opt/kafka/kafka/logs/log-cleaner.log.2021-06-27-16
/opt/kafka/kafka/logs/server.log.2021-06-28-19
/opt/kafka/kafka/logs/controller.log.2021-06-27-16
/opt/kafka/kafka/logs/state-change.log.2021-06-27-16

Delete log files modified more than 2 days ago.

$ find /opt/kafka/kafka/logs/ -maxdepth 1 -regextype egrep  -regex ".*/[a-z\-]+.log.[0-9]{4}-[0-9]{2}-[0-9]{2}-[0-9]{2}$"  -mtime +2 -delete

Log files will be not rotated automatically.

controller.log                log-cleaner.log           state-change.log.2021-06-28-06
controller.log.2021-06-28-06  server.log                state-change.log.2021-06-28-19
kafka-authorizer.log          server.log.2021-06-27-18  state-change.log.2021-06-28-22
kafka-request.log             server.log.2021-06-28-06  state-change.log.2021-06-29-19
kafkaServer-gc.log            server.log.2021-06-30-00  state-change.log.2021-06-30-00
kafkaServer-gc.log.0          state-change.log          zookeeper-gc.log

Second solution

Use RollingFileAppender to rotate log every MaxFileSize and keep MaxBackupIndex number of files.

[...]

log4j.appender.kafkaAppender=org.apache.log4j.RollingFileAppender
log4j.appender.kafkaAppender.File=${kafka.logs.dir}/server.log                                                                 
log4j.appender.kafkaAppender.layout=org.apache.log4j.PatternLayout
log4j.appender.kafkaAppender.layout.ConversionPattern=[%d] %p %m (%c)%n
log4j.appender.kafkaAppender.MaxFileSize=128MB
log4j.appender.kafkaAppender.MaxBackupIndex=10

[...]

It will rotate logs automatically.

controller.log        kafkaServer-gc.log.6  server.log.4
controller.log.1      kafkaServer-gc.log.7  server.log.5
controller.log.2      kafkaServer-gc.log.8  server.log.6
controller.log.3      kafkaServer-gc.log.9  server.log.7
controller.log.4      log-cleaner.log       server.log.8
controller.log.5      log-cleaner.log.1     server.log.9
kafka-authorizer.log  log-cleaner.log.2     state-change.log
kafka-request.log     log-cleaner.log.3     state-change.log.1
kafkaServer-gc.log    log-cleaner.log.4     state-change.log.2
kafkaServer-gc.log.1  server.log            state-change.log.3
kafkaServer-gc.log.2  server.log.1          state-change.log.4
kafkaServer-gc.log.3  server.log.10         state-change.log.5
kafkaServer-gc.log.4  server.log.2          state-change.log.6
kafkaServer-gc.log.5  server.log.3

Third solution

You can also use a FileAppender and rotate log file using an external application like logrotate, but this is not the best solution as some log entries can be lost during rotation process.

[...]

log4j.appender.kafkaAppender=org.apache.log4j.FileAppender
log4j.appender.kafkaAppender.File=${kafka.logs.dir}/server.log                                                                 
log4j.appender.kafkaAppender.layout=org.apache.log4j.PatternLayout
log4j.appender.kafkaAppender.layout.ConversionPattern=[%d] %p %m (%c)%n
log4j.appender.kafkaAppender.Append=true

[...]

Logrotate configuration.

/opt/kafka/kafka/logs/server.log {
  daily
  rotate 10

  copytruncate
  compress
  missingok
}

Logrotate will take care of everything, but there is a chance that the log file will contain additional entries before being truncated, these entries will be lost.

controller.log        kafka-request.log   log-cleaner.log  server.log.1.gz  server.log.3.gz
kafka-authorizer.log  kafkaServer-gc.log  server.log       server.log.2.gz  state-change.log