How to remove old ZooKeeper snapshots and transaction logs.
Inspect ZooKeeper snapshots and transaction logs.
$ ls data/version-2/
acceptedEpoch log.00000001 log.00000003 log.100000001 log.80000000c snapshot.0 snapshot.500000000 snapshot.700000000 snapshot.900000000 currentEpoch log.00000002 log.00000004 log.800000001 log.c00000001 snapshot.100000000 snapshot.600000000 snapshot.800000000 version-2
Use built-in shell script to keep last 3 snapshots/transaction logs.
$ bin/zkCleanup.sh -n 3
/usr/bin/java 2021-07-30 22:26:54,732 [myid:] - INFO [main:FileTxnSnapLog@124] - zookeeper.snapshot.trust.empty : false 2021-07-30 22:26:54,743 [myid:] - INFO [main:SnapStream@61] - zookeeper.snapshot.compression.method = CHECKED 2021-07-30 22:26:54,755 [myid:] - INFO [main:PurgeTxnLog@157] - Removing file: Jul 30, 2021, 10:25:19 PM /opt/zookeeper/zookeeper/data/version-2/log.00000003 Removing file: Jul 30, 2021, 10:25:19 PM /opt/zookeeper/zookeeper/data/version-2/log.00000003 2021-07-30 22:26:54,756 [myid:] - INFO [main:PurgeTxnLog@157] - Removing file: Jul 30, 2021, 10:25:19 PM /opt/zookeeper/zookeeper/data/version-2/log.00000001 Removing file: Jul 30, 2021, 10:25:19 PM /opt/zookeeper/zookeeper/data/version-2/log.00000001 2021-07-30 22:26:54,765 [myid:] - INFO [main:PurgeTxnLog@157] - Removing file: Jul 30, 2021, 10:25:19 PM /opt/zookeeper/zookeeper/data/version-2/log.00000002 Removing file: Jul 30, 2021, 10:25:19 PM /opt/zookeeper/zookeeper/data/version-2/log.00000002 2021-07-30 22:26:54,766 [myid:] - INFO [main:PurgeTxnLog@157] - Removing file: Jul 30, 2021, 10:25:19 PM /opt/zookeeper/zookeeper/data/version-2/log.00000004 Removing file: Jul 30, 2021, 10:25:19 PM /opt/zookeeper/zookeeper/data/version-2/log.00000004 2021-07-30 22:26:54,787 [myid:] - INFO [main:PurgeTxnLog@157] - Removing file: Jul 30, 2021, 10:25:19 PM /opt/zookeeper/zookeeper/data/version-2/snapshot.0 Removing file: Jul 30, 2021, 10:25:19 PM /opt/zookeeper/zookeeper/data/version-2/snapshot.0 2021-07-30 22:26:54,789 [myid:] - INFO [main:PurgeTxnLog@157] - Removing file: Jul 30, 2021, 10:25:19 PM /opt/zookeeper/zookeeper/data/version-2/snapshot.600000000 Removing file: Jul 30, 2021, 10:25:19 PM /opt/zookeeper/zookeeper/data/version-2/snapshot.600000000 2021-07-30 22:26:54,790 [myid:] - INFO [main:PurgeTxnLog@157] - Removing file: Jul 30, 2021, 10:25:19 PM /opt/zookeeper/zookeeper/data/version-2/snapshot.500000000 Removing file: Jul 30, 2021, 10:25:19 PM /opt/zookeeper/zookeeper/data/version-2/snapshot.500000000 2021-07-30 22:26:54,791 [myid:] - INFO [main:PurgeTxnLog@157] - Removing file: Jul 30, 2021, 10:25:19 PM /opt/zookeeper/zookeeper/data/version-2/snapshot.100000000 Removing file: Jul 30, 2021, 10:25:19 PM /opt/zookeeper/zookeeper/data/version-2/snapshot.100000000
This operation is equivalent to a direct call.
$ java -cp lib/zookeeper-3.7.0.jar:lib/log4j-1.2.17.jar:lib/zookeeper-jute-3.7.0.jar:lib/slf4j-api-1.7.30.jar:lib/slf4j-log4j12-1.7.30.jar:lib/snappy-java-1.1.7.7.jar:conf org.apache.zookeeper.server.PurgeTxnLog /opt/zookeeper/zookeeper/data -n 3
2021-07-30 22:29:00,611 [myid:] - INFO [main:FileTxnSnapLog@124] - zookeeper.snapshot.trust.empty : false 2021-07-30 22:29:00,619 [myid:] - INFO [main:SnapStream@61] - zookeeper.snapshot.compression.method = CHECKED 2021-07-30 22:29:00,630 [myid:] - INFO [main:PurgeTxnLog@157] - Removing file: Jul 30, 2021, 10:28:15 PM /opt/zookeeper/zookeeper/data/version-2/log.00000003 Removing file: Jul 30, 2021, 10:28:15 PM /opt/zookeeper/zookeeper/data/version-2/log.00000003 2021-07-30 22:29:00,633 [myid:] - INFO [main:PurgeTxnLog@157] - Removing file: Jul 30, 2021, 10:28:15 PM /opt/zookeeper/zookeeper/data/version-2/log.00000001 Removing file: Jul 30, 2021, 10:28:15 PM /opt/zookeeper/zookeeper/data/version-2/log.00000001 2021-07-30 22:29:00,636 [myid:] - INFO [main:PurgeTxnLog@157] - Removing file: Jul 30, 2021, 10:28:15 PM /opt/zookeeper/zookeeper/data/version-2/log.00000002 Removing file: Jul 30, 2021, 10:28:15 PM /opt/zookeeper/zookeeper/data/version-2/log.00000002 2021-07-30 22:29:00,638 [myid:] - INFO [main:PurgeTxnLog@157] - Removing file: Jul 30, 2021, 10:28:15 PM /opt/zookeeper/zookeeper/data/version-2/log.00000004 Removing file: Jul 30, 2021, 10:28:15 PM /opt/zookeeper/zookeeper/data/version-2/log.00000004 2021-07-30 22:29:00,639 [myid:] - INFO [main:PurgeTxnLog@157] - Removing file: Jul 30, 2021, 10:28:15 PM /opt/zookeeper/zookeeper/data/version-2/snapshot.0 Removing file: Jul 30, 2021, 10:28:15 PM /opt/zookeeper/zookeeper/data/version-2/snapshot.0 2021-07-30 22:29:00,645 [myid:] - INFO [main:PurgeTxnLog@157] - Removing file: Jul 30, 2021, 10:28:15 PM /opt/zookeeper/zookeeper/data/version-2/snapshot.600000000 Removing file: Jul 30, 2021, 10:28:15 PM /opt/zookeeper/zookeeper/data/version-2/snapshot.600000000 2021-07-30 22:29:00,646 [myid:] - INFO [main:PurgeTxnLog@157] - Removing file: Jul 30, 2021, 10:28:15 PM /opt/zookeeper/zookeeper/data/version-2/snapshot.500000000 Removing file: Jul 30, 2021, 10:28:15 PM /opt/zookeeper/zookeeper/data/version-2/snapshot.500000000 2021-07-30 22:29:00,647 [myid:] - INFO [main:PurgeTxnLog@157] - Removing file: Jul 30, 2021, 10:28:15 PM /opt/zookeeper/zookeeper/data/version-2/snapshot.100000000 Removing file: Jul 30, 2021, 10:28:15 PM /opt/zookeeper/zookeeper/data/version-2/snapshot.100000000
Alternatively, set the autopurge.snapRetainCount to the number of recent snapshots you want to keep and the autopurge.purgeInterval to the purge interval in hours.
autopurge.snapRetainCount=3 autopurge.purgeInterval=1
Restart ZooKeeper service and sleep well.
$ grep PurgeTask logs/zookeeper-zookeeper-server-zookeeper1.out
2021-07-30 22:33:14,521 [myid:1] - INFO [PurgeTask:DatadirCleanupManager$PurgeTask@139] - Purge task started. 2021-07-30 22:33:14,543 [myid:1] - INFO [PurgeTask:FileTxnSnapLog@124] - zookeeper.snapshot.trust.empty : false 2021-07-30 22:33:14,567 [myid:1] - INFO [PurgeTask:SnapStream@61] - zookeeper.snapshot.compression.method = CHECKED 2021-07-30 22:33:14,580 [myid:1] - INFO [PurgeTask:PurgeTxnLog@157] - Removing file: Jul 30, 2021, 10:29:35 PM /opt/zookeeper/zookeeper/data/version-2/log.00000003 2021-07-30 22:33:14,581 [myid:1] - INFO [PurgeTask:PurgeTxnLog@157] - Removing file: Jul 30, 2021, 10:29:35 PM /opt/zookeeper/zookeeper/data/version-2/log.00000001 2021-07-30 22:33:14,581 [myid:1] - INFO [PurgeTask:PurgeTxnLog@157] - Removing file: Jul 30, 2021, 10:29:35 PM /opt/zookeeper/zookeeper/data/version-2/log.00000002 2021-07-30 22:33:14,582 [myid:1] - INFO [PurgeTask:PurgeTxnLog@157] - Removing file: Jul 30, 2021, 10:29:35 PM /opt/zookeeper/zookeeper/data/version-2/log.00000004 2021-07-30 22:33:14,582 [myid:1] - INFO [PurgeTask:PurgeTxnLog@157] - Removing file: Jul 30, 2021, 10:29:35 PM /opt/zookeeper/zookeeper/data/version-2/snapshot.0 2021-07-30 22:33:14,583 [myid:1] - INFO [PurgeTask:PurgeTxnLog@157] - Removing file: Jul 30, 2021, 10:29:35 PM /opt/zookeeper/zookeeper/data/version-2/snapshot.600000000 2021-07-30 22:33:14,583 [myid:1] - INFO [PurgeTask:PurgeTxnLog@157] - Removing file: Jul 30, 2021, 10:29:35 PM /opt/zookeeper/zookeeper/data/version-2/snapshot.500000000 2021-07-30 22:33:14,583 [myid:1] - INFO [PurgeTask:PurgeTxnLog@157] - Removing file: Jul 30, 2021, 10:29:35 PM /opt/zookeeper/zookeeper/data/version-2/snapshot.100000000 2021-07-30 22:33:14,583 [myid:1] - INFO [PurgeTask:DatadirCleanupManager$PurgeTask@145] - Purge task completed.