Categories
SysOps

How to save namespace and perform checkpoint

Save namespace on namenode and perform checkpoint on secondary namenode.

Namenode

Enter safe mode.

$ sudo -u hadoop -i  hdfs dfsadmin -safemode enter
Safe mode is ON

Ensure that safe mode is active.

$ sudo -u hadoop -i  hdfs dfsadmin -safemode get
Safe mode is ON

Save current namespace into storage directories and reset edits log.

$ sudo -u hadoop -i  hdfs dfsadmin -saveNamespace
Save namespace successful

Leave safe mode.

$ sudo -u hadoop -i  hdfs dfsadmin -safemode leave
Safe mode is OFF

Inspect namenode logs.

2021-06-08 23:30:25,296 INFO org.apache.hadoop.hdfs.StateChange: STATE* Safe mode is ON.
It was turned on manually. Use "hdfs dfsadmin -safemode leave" to turn safe mode off.
2021-06-08 23:30:25,296 INFO org.apache.hadoop.hdfs.server.namenode.FSImage: Save namespace ...
2021-06-08 23:30:25,296 INFO org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log segment 76693, 76693
2021-06-08 23:30:25,297 INFO org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2 Total time for transactions(ms): 0 Number of transactions batched in Syncs: 76692 Number of syncs: 3 SyncTimes(ms): 18 
2021-06-08 23:30:25,298 INFO org.apache.hadoop.hdfs.server.namenode.FileJournalManager: Finalizing edits file /opt/hadoop/local_data/namenode/current/edits_inprogress_0000000000000076693 -> /opt/hadoop/local_data/namenode/current/edits_0000000000000076693-0000000000000076694
2021-06-08 23:30:25,334 INFO org.apache.hadoop.hdfs.server.namenode.FSImageFormatProtobuf: Saving image file /opt/hadoop/local_data/namenode/current/fsimage.ckpt_0000000000000076694 using no compression
2021-06-08 23:30:25,492 INFO org.apache.hadoop.hdfs.server.namenode.FSImageFormatProtobuf: Image file /opt/hadoop/local_data/namenode/current/fsimage.ckpt_0000000000000076694 of size 765058 bytes saved in 0 seconds .
2021-06-08 23:30:25,509 INFO org.apache.hadoop.hdfs.server.namenode.NNStorageRetentionManager: Going to retain 2 images with txid >= 76675
2021-06-08 23:30:25,509 INFO org.apache.hadoop.hdfs.server.namenode.NNStorageRetentionManager: Purging old image FSImageFile(file=/opt/hadoop/local_data/namenode/current/fsimage_0000000000000076671, cpktTxId=0000000000000076671)
2021-06-08 23:30:25,568 INFO org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at 76695
2021-06-08 23:30:25,592 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: New namespace image has been created
2021-06-08 23:30:31,286 INFO org.apache.hadoop.hdfs.StateChange: STATE* Leaving safe mode after 1031 secs
2021-06-08 23:30:31,286 INFO org.apache.hadoop.hdfs.StateChange: STATE* Network topology has 1 racks and 3 datanodes
2021-06-08 23:30:31,286 INFO org.apache.hadoop.hdfs.StateChange: STATE* UnderReplicatedBlocks has 0 blocks

Secondary namenode

Determine checkpoint directory.

$ hdfs getconf -confKey dfs.namenode.checkpoint.dir
/opt/hadoop/local_data/secondarynamenode

Stop secondary namenode.

$ systemctl stop hadoop-secondarynamenode.service

Rename current checkpoint directory.

$ mv /opt/hadoop/local_data/secondarynamenode /opt/hadoop/local_data/secondarynamenode.$(date +"%Y%m%d_%H%M")

Force checkpoint operation.

$ sudo -u hadoop -i hdfs secondarynamenode -checkpoint force
2021-06-08 23:38:42,104 INFO namenode.SecondaryNameNode: STARTUP_MSG: 
/************************************************************
STARTUP_MSG: Starting SecondaryNameNode
STARTUP_MSG:   host = secondarynamenode/192.168.8.171
STARTUP_MSG:   args = [-checkpoint, force]
STARTUP_MSG:   version = 3.2.2
STARTUP_MSG:   classpath = /opt/hadoop/hadoop-3.2.2/etc/hadoop:/opt/hadoop/hadoop-3.2.2/share/hadoop/common/lib/jsr305-3.0.2.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/common/lib/commons-io-2.5.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/common/lib/paranamer-2.3.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/common/lib/kerb-simplekdc-1.0.1.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/common/lib/jetty-security-9.4.20.v20190813.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/common/lib/curator-recipes-2.13.0.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/common/lib/jersey-core-1.19.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/common/lib/jsr311-api-1.1.1.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/common/lib/jaxb-impl-2.2.3-1.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/common/lib/gson-2.2.4.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/common/lib/slf4j-api-1.7.25.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/common/lib/avro-1.7.7.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/common/lib/commons-cli-1.2.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/common/lib/jettison-1.1.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/common/lib/jackson-databind-2.9.10.4.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/common/lib/kerb-admin-1.0.1.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/common/lib/jackson-core-2.9.10.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/common/lib/kerb-common-1.0.1.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/common/lib/kerb-client-1.0.1.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/common/lib/commons-lang3-3.7.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/common/lib/commons-beanutils-1.9.4.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/common/lib/asm-5.0.4.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/common/lib/netty-3.10.6.Final.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/common/lib/json-smart-2.3.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/common/lib/snappy-java-1.0.5.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/common/lib/commons-text-1.4.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/common/lib/stax2-api-3.1.4.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/common/lib/kerby-xdr-1.0.1.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/common/lib/jackson-core-asl-1.9.13.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/common/lib/jaxb-api-2.2.11.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/common/lib/zookeeper-3.4.13.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/common/lib/log4j-1.2.17.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/common/lib/dnsjava-2.1.7.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/common/lib/jersey-json-1.19.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/common/lib/jul-to-slf4j-1.7.25.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/common/lib/nimbus-jose-jwt-7.9.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/common/lib/jetty-xml-9.4.20.v20190813.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/common/lib/jetty-http-9.4.20.v20190813.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/common/lib/kerb-crypto-1.0.1.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/common/lib/j2objc-annotations-1.1.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/common/lib/httpclient-4.5.13.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/common/lib/jersey-server-1.19.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/common/lib/animal-sniffer-annotations-1.17.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/common/lib/kerby-asn1-1.0.1.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/common/lib/jetty-webapp-9.4.20.v20190813.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/common/lib/kerby-util-1.0.1.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/common/lib/commons-collections-3.2.2.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/common/lib/jersey-servlet-1.19.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/common/lib/commons-configuration2-2.1.1.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/common/lib/jackson-jaxrs-1.9.13.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/common/lib/kerby-config-1.0.1.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/common/lib/checker-qual-2.5.2.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/common/lib/hadoop-auth-3.2.2.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/common/lib/jetty-server-9.4.20.v20190813.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/common/lib/kerb-util-1.0.1.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/common/lib/metrics-core-3.2.4.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/common/lib/jetty-util-9.4.20.v20190813.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/common/lib/jackson-mapper-asl-1.9.13.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/common/lib/jackson-annotations-2.9.10.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/common/lib/guava-27.0-jre.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/common/lib/jsch-0.1.55.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/common/lib/kerb-server-1.0.1.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/common/lib/commons-net-3.6.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/common/lib/commons-math3-3.1.1.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/common/lib/jcip-annotations-1.0-1.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/common/lib/htrace-core4-4.1.0-incubating.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/common/lib/commons-compress-1.19.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/common/lib/jsp-api-2.1.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/common/lib/curator-framework-2.13.0.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/common/lib/token-provider-1.0.1.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/common/lib/accessors-smart-1.2.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/common/lib/audience-annotations-0.5.0.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/common/lib/javax.servlet-api-3.1.0.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/common/lib/failureaccess-1.0.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/common/lib/kerb-core-1.0.1.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/common/lib/woodstox-core-5.0.3.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/common/lib/listenablefuture-9999.0-empty-to-avoid-conflict-with-guava.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/common/lib/error_prone_annotations-2.2.0.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/common/lib/commons-codec-1.11.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/common/lib/hadoop-annotations-3.2.2.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/common/lib/kerby-pkix-1.0.1.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/common/lib/kerb-identity-1.0.1.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/common/lib/protobuf-java-2.5.0.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/common/lib/httpcore-4.4.13.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/common/lib/re2j-1.1.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/common/lib/jetty-servlet-9.4.20.v20190813.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/common/lib/jetty-io-9.4.20.v20190813.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/common/lib/javax.activation-api-1.2.0.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/common/lib/commons-logging-1.1.3.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/common/lib/jackson-xc-1.9.13.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/common/lib/curator-client-2.13.0.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/common/hadoop-common-3.2.2-tests.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/common/hadoop-nfs-3.2.2.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/common/hadoop-kms-3.2.2.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/common/hadoop-common-3.2.2.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/hdfs:/opt/hadoop/hadoop-3.2.2/share/hadoop/hdfs/lib/jsr305-3.0.2.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/hdfs/lib/commons-io-2.5.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/hdfs/lib/paranamer-2.3.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/hdfs/lib/kerb-simplekdc-1.0.1.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/hdfs/lib/jetty-security-9.4.20.v20190813.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/hdfs/lib/curator-recipes-2.13.0.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/hdfs/lib/jetty-util-ajax-9.4.20.v20190813.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/hdfs/lib/json-simple-1.1.1.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/hdfs/lib/jersey-core-1.19.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/hdfs/lib/jsr311-api-1.1.1.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/hdfs/lib/jaxb-impl-2.2.3-1.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/hdfs/lib/gson-2.2.4.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/hdfs/lib/avro-1.7.7.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/hdfs/lib/commons-cli-1.2.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/hdfs/lib/jettison-1.1.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/hdfs/lib/jackson-databind-2.9.10.4.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/hdfs/lib/kerb-admin-1.0.1.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/hdfs/lib/jackson-core-2.9.10.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/hdfs/lib/kerb-common-1.0.1.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/hdfs/lib/kerb-client-1.0.1.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/hdfs/lib/commons-lang3-3.7.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/hdfs/lib/leveldbjni-all-1.8.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/hdfs/lib/commons-beanutils-1.9.4.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/hdfs/lib/asm-5.0.4.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/hdfs/lib/netty-3.10.6.Final.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/hdfs/lib/json-smart-2.3.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/hdfs/lib/snappy-java-1.0.5.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/hdfs/lib/commons-text-1.4.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/hdfs/lib/stax2-api-3.1.4.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/hdfs/lib/kerby-xdr-1.0.1.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/hdfs/lib/jackson-core-asl-1.9.13.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/hdfs/lib/jaxb-api-2.2.11.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/hdfs/lib/zookeeper-3.4.13.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/hdfs/lib/netty-all-4.1.48.Final.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/hdfs/lib/log4j-1.2.17.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/hdfs/lib/dnsjava-2.1.7.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/hdfs/lib/jersey-json-1.19.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/hdfs/lib/okhttp-2.7.5.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/hdfs/lib/nimbus-jose-jwt-7.9.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/hdfs/lib/jetty-xml-9.4.20.v20190813.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/hdfs/lib/jetty-http-9.4.20.v20190813.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/hdfs/lib/kerb-crypto-1.0.1.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/hdfs/lib/j2objc-annotations-1.1.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/hdfs/lib/httpclient-4.5.13.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/hdfs/lib/jersey-server-1.19.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/hdfs/lib/animal-sniffer-annotations-1.17.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/hdfs/lib/kerby-asn1-1.0.1.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/hdfs/lib/jetty-webapp-9.4.20.v20190813.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/hdfs/lib/kerby-util-1.0.1.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/hdfs/lib/okio-1.6.0.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/hdfs/lib/commons-collections-3.2.2.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/hdfs/lib/jersey-servlet-1.19.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/hdfs/lib/commons-configuration2-2.1.1.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/hdfs/lib/jackson-jaxrs-1.9.13.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/hdfs/lib/kerby-config-1.0.1.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/hdfs/lib/checker-qual-2.5.2.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/hdfs/lib/hadoop-auth-3.2.2.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/hdfs/lib/jetty-server-9.4.20.v20190813.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/hdfs/lib/kerb-util-1.0.1.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/hdfs/lib/jetty-util-9.4.20.v20190813.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/hdfs/lib/jackson-mapper-asl-1.9.13.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/hdfs/lib/jackson-annotations-2.9.10.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/hdfs/lib/guava-27.0-jre.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/hdfs/lib/jsch-0.1.55.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/hdfs/lib/commons-daemon-1.0.13.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/hdfs/lib/kerb-server-1.0.1.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/hdfs/lib/commons-net-3.6.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/hdfs/lib/commons-math3-3.1.1.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/hdfs/lib/jcip-annotations-1.0-1.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/hdfs/lib/htrace-core4-4.1.0-incubating.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/hdfs/lib/commons-compress-1.19.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/hdfs/lib/curator-framework-2.13.0.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/hdfs/lib/token-provider-1.0.1.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/hdfs/lib/accessors-smart-1.2.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/hdfs/lib/audience-annotations-0.5.0.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/hdfs/lib/javax.servlet-api-3.1.0.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/hdfs/lib/failureaccess-1.0.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/hdfs/lib/kerb-core-1.0.1.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/hdfs/lib/woodstox-core-5.0.3.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/hdfs/lib/listenablefuture-9999.0-empty-to-avoid-conflict-with-guava.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/hdfs/lib/error_prone_annotations-2.2.0.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/hdfs/lib/commons-codec-1.11.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/hdfs/lib/hadoop-annotations-3.2.2.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/hdfs/lib/kerby-pkix-1.0.1.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/hdfs/lib/kerb-identity-1.0.1.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/hdfs/lib/protobuf-java-2.5.0.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/hdfs/lib/httpcore-4.4.13.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/hdfs/lib/re2j-1.1.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/hdfs/lib/jetty-servlet-9.4.20.v20190813.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/hdfs/lib/jetty-io-9.4.20.v20190813.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/hdfs/lib/javax.activation-api-1.2.0.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/hdfs/lib/commons-logging-1.1.3.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/hdfs/lib/jackson-xc-1.9.13.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/hdfs/lib/curator-client-2.13.0.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/hdfs/hadoop-hdfs-native-client-3.2.2.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/hdfs/hadoop-hdfs-3.2.2-tests.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/hdfs/hadoop-hdfs-client-3.2.2-tests.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/hdfs/hadoop-hdfs-nfs-3.2.2.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/hdfs/hadoop-hdfs-rbf-3.2.2-tests.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/hdfs/hadoop-hdfs-httpfs-3.2.2.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/hdfs/hadoop-hdfs-3.2.2.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/hdfs/hadoop-hdfs-rbf-3.2.2.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/hdfs/hadoop-hdfs-client-3.2.2.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/hdfs/hadoop-hdfs-native-client-3.2.2-tests.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/mapreduce/lib/hamcrest-core-1.3.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/mapreduce/lib/junit-4.11.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-3.2.2.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/mapreduce/hadoop-mapreduce-client-nativetask-3.2.2.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/mapreduce/hadoop-mapreduce-client-app-3.2.2.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-3.2.2-tests.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/mapreduce/hadoop-mapreduce-client-shuffle-3.2.2.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-3.2.2.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/mapreduce/hadoop-mapreduce-client-uploader-3.2.2.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-plugins-3.2.2.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/mapreduce/hadoop-mapreduce-client-core-3.2.2.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.2.2.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/mapreduce/hadoop-mapreduce-client-common-3.2.2.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/yarn:/opt/hadoop/hadoop-3.2.2/share/hadoop/yarn/lib/json-io-2.5.1.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/yarn/lib/jackson-jaxrs-base-2.9.10.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/yarn/lib/snakeyaml-1.16.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/yarn/lib/javax.inject-1.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/yarn/lib/HikariCP-java7-2.4.12.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/yarn/lib/ehcache-3.3.1.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/yarn/lib/bcprov-jdk15on-1.60.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/yarn/lib/jersey-client-1.19.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/yarn/lib/fst-2.50.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/yarn/lib/objenesis-1.0.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/yarn/lib/mssql-jdbc-6.2.1.jre7.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/yarn/lib/swagger-annotations-1.5.4.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/yarn/lib/jackson-jaxrs-json-provider-2.9.10.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/yarn/lib/guice-servlet-4.0.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/yarn/lib/jackson-module-jaxb-annotations-2.9.10.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/yarn/lib/jersey-guice-1.19.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/yarn/lib/metrics-core-3.2.4.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/yarn/lib/java-util-1.9.0.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/yarn/lib/bcpkix-jdk15on-1.60.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/yarn/lib/aopalliance-1.0.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/yarn/lib/guice-4.0.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/yarn/lib/geronimo-jcache_1.0_spec-1.0-alpha-1.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/yarn/hadoop-yarn-client-3.2.2.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/yarn/hadoop-yarn-server-applicationhistoryservice-3.2.2.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/yarn/hadoop-yarn-applications-distributedshell-3.2.2.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/yarn/hadoop-yarn-server-common-3.2.2.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/yarn/hadoop-yarn-applications-unmanaged-am-launcher-3.2.2.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/yarn/hadoop-yarn-server-resourcemanager-3.2.2.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/yarn/hadoop-yarn-server-web-proxy-3.2.2.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/yarn/hadoop-yarn-registry-3.2.2.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/yarn/hadoop-yarn-services-api-3.2.2.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/yarn/hadoop-yarn-server-nodemanager-3.2.2.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/yarn/hadoop-yarn-common-3.2.2.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/yarn/hadoop-yarn-server-sharedcachemanager-3.2.2.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/yarn/hadoop-yarn-api-3.2.2.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/yarn/hadoop-yarn-server-router-3.2.2.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/yarn/hadoop-yarn-server-timeline-pluginstorage-3.2.2.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/yarn/hadoop-yarn-services-core-3.2.2.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/yarn/hadoop-yarn-submarine-3.2.2.jar:/opt/hadoop/hadoop-3.2.2/share/hadoop/yarn/hadoop-yarn-server-tests-3.2.2.jar
STARTUP_MSG:   build = Unknown -r 7a3bc90b05f257c8ace2f76d74264906f0f7a932; compiled by 'hexiaoqiao' on 2021-01-03T09:26Z
STARTUP_MSG:   java = 11.0.11
************************************************************/
2021-06-08 23:38:42,128 INFO namenode.SecondaryNameNode: registered UNIX signal handlers for [TERM, HUP, INT]
2021-06-08 23:38:42,678 INFO impl.MetricsConfig: Loaded properties from hadoop-metrics2.properties
2021-06-08 23:38:42,856 INFO impl.MetricsSystemImpl: Scheduled Metric snapshot period at 10 second(s).
2021-06-08 23:38:42,856 INFO impl.MetricsSystemImpl: SecondaryNameNode metrics system started
2021-06-08 23:38:43,427 INFO common.Util: Assuming 'file' scheme for path /opt/hadoop/local_data/secondarynamenode in configuration.
2021-06-08 23:38:43,438 INFO common.Util: Assuming 'file' scheme for path /opt/hadoop/local_data/secondarynamenode in configuration.
2021-06-08 23:38:43,505 INFO namenode.FSEditLog: Edit logging is async:true
2021-06-08 23:38:43,535 INFO common.Storage: Lock on /opt/hadoop/local_data/secondarynamenode/in_use.lock acquired by nodename 10567@secondarynamenode
2021-06-08 23:38:43,570 INFO namenode.FSNamesystem: KeyProvider: null
2021-06-08 23:38:43,571 INFO namenode.FSNamesystem: fsLock is fair: true
2021-06-08 23:38:43,571 INFO namenode.FSNamesystem: Detailed lock hold time metrics enabled: false
2021-06-08 23:38:43,582 INFO namenode.FSNamesystem: fsOwner             = hadoop (auth:SIMPLE)
2021-06-08 23:38:43,582 INFO namenode.FSNamesystem: supergroup          = supergroup
2021-06-08 23:38:43,582 INFO namenode.FSNamesystem: isPermissionEnabled = true
2021-06-08 23:38:43,582 INFO namenode.FSNamesystem: HA Enabled: false
2021-06-08 23:38:43,627 INFO common.Util: dfs.datanode.fileio.profiling.sampling.percentage set to 0. Disabling file IO profiling
2021-06-08 23:38:43,633 INFO blockmanagement.DatanodeManager: dfs.block.invalidate.limit: configured=1000, counted=60, effected=1000
2021-06-08 23:38:43,633 INFO blockmanagement.DatanodeManager: dfs.namenode.datanode.registration.ip-hostname-check=true
2021-06-08 23:38:43,638 INFO blockmanagement.BlockManager: dfs.namenode.startup.delay.block.deletion.sec is set to 000:00:00:00.000
2021-06-08 23:38:43,638 INFO blockmanagement.BlockManager: The block deletion will start around 2021 Jun 08 23:38:43
2021-06-08 23:38:43,640 INFO util.GSet: Computing capacity for map BlocksMap
2021-06-08 23:38:43,640 INFO util.GSet: VM type       = 64-bit
2021-06-08 23:38:43,641 INFO util.GSet: 2.0% max memory 247.5 MB = 4.9 MB
2021-06-08 23:38:43,641 INFO util.GSet: capacity      = 2^19 = 524288 entries
2021-06-08 23:38:43,653 INFO blockmanagement.BlockManager: Storage policy satisfier is disabled
2021-06-08 23:38:43,653 INFO blockmanagement.BlockManager: dfs.block.access.token.enable = false
2021-06-08 23:38:43,667 INFO Configuration.deprecation: No unit for dfs.namenode.safemode.extension(30000) assuming MILLISECONDS
2021-06-08 23:38:43,668 INFO blockmanagement.BlockManagerSafeMode: dfs.namenode.safemode.threshold-pct = 0.9990000128746033
2021-06-08 23:38:43,668 INFO blockmanagement.BlockManagerSafeMode: dfs.namenode.safemode.min.datanodes = 0
2021-06-08 23:38:43,668 INFO blockmanagement.BlockManagerSafeMode: dfs.namenode.safemode.extension = 30000
2021-06-08 23:38:43,669 INFO blockmanagement.BlockManager: defaultReplication         = 3
2021-06-08 23:38:43,669 INFO blockmanagement.BlockManager: maxReplication             = 512
2021-06-08 23:38:43,669 INFO blockmanagement.BlockManager: minReplication             = 1
2021-06-08 23:38:43,669 INFO blockmanagement.BlockManager: maxReplicationStreams      = 2
2021-06-08 23:38:43,669 INFO blockmanagement.BlockManager: redundancyRecheckInterval  = 3000ms
2021-06-08 23:38:43,669 INFO blockmanagement.BlockManager: encryptDataTransfer        = false
2021-06-08 23:38:43,669 INFO blockmanagement.BlockManager: maxNumBlocksToLog          = 1000
2021-06-08 23:38:43,696 INFO namenode.FSDirectory: GLOBAL serial map: bits=29 maxEntries=536870911
2021-06-08 23:38:43,696 INFO namenode.FSDirectory: USER serial map: bits=24 maxEntries=16777215
2021-06-08 23:38:43,696 INFO namenode.FSDirectory: GROUP serial map: bits=24 maxEntries=16777215
2021-06-08 23:38:43,696 INFO namenode.FSDirectory: XATTR serial map: bits=24 maxEntries=16777215
2021-06-08 23:38:43,704 INFO util.GSet: Computing capacity for map INodeMap
2021-06-08 23:38:43,704 INFO util.GSet: VM type       = 64-bit
2021-06-08 23:38:43,704 INFO util.GSet: 1.0% max memory 247.5 MB = 2.5 MB
2021-06-08 23:38:43,704 INFO util.GSet: capacity      = 2^18 = 262144 entries
2021-06-08 23:38:43,709 INFO namenode.FSDirectory: ACLs enabled? false
2021-06-08 23:38:43,709 INFO namenode.FSDirectory: POSIX ACL inheritance enabled? true
2021-06-08 23:38:43,709 INFO namenode.FSDirectory: XAttrs enabled? true
2021-06-08 23:38:43,710 INFO namenode.NameNode: Caching file names occurring more than 10 times
2021-06-08 23:38:43,713 INFO snapshot.SnapshotManager: Loaded config captureOpenFiles: false, skipCaptureAccessTimeOnlyChange: false, snapshotDiffAllowSnapRootDescendant: true, maxSnapshotLimit: 65536
2021-06-08 23:38:43,715 INFO snapshot.SnapshotManager: SkipList is disabled
2021-06-08 23:38:43,725 INFO util.GSet: Computing capacity for map cachedBlocks
2021-06-08 23:38:43,725 INFO util.GSet: VM type       = 64-bit
2021-06-08 23:38:43,725 INFO util.GSet: 0.25% max memory 247.5 MB = 633.6 KB
2021-06-08 23:38:43,725 INFO util.GSet: capacity      = 2^16 = 65536 entries
2021-06-08 23:38:43,732 INFO metrics.TopMetrics: NNTop conf: dfs.namenode.top.window.num.buckets = 10
2021-06-08 23:38:43,734 INFO metrics.TopMetrics: NNTop conf: dfs.namenode.top.num.users = 10
2021-06-08 23:38:43,734 INFO metrics.TopMetrics: NNTop conf: dfs.namenode.top.windows.minutes = 1,5,25
2021-06-08 23:38:43,758 INFO namenode.SecondaryNameNode: Checkpoint Period   :3600 secs (60 min)
2021-06-08 23:38:43,759 INFO namenode.SecondaryNameNode: Log Size Trigger    :1000000 txns
2021-06-08 23:38:43,896 INFO namenode.SecondaryNameNode: Image has changed. Downloading updated image from NN.
2021-06-08 23:38:43,920 INFO namenode.TransferFsImage: Opening connection to http://namenode.example.org:9870/imagetransfer?getimage=1&txid=76694&storageInfo=-65:1373659126:1620683988326:CID-5d1164fe-f8a5-4982-a503-096898971b41&bootstrapstandby=false
2021-06-08 23:38:43,986 INFO common.Util: Combined time for file download and fsync to all disks took 0.03s. The file download took 0.03s at 29880.00 KB/s. Synchronous (fsync) write to disk of /opt/hadoop/local_data/secondarynamenode/current/fsimage.ckpt_0000000000000076694 took 0.00s.
2021-06-08 23:38:43,986 INFO namenode.TransferFsImage: Downloaded file fsimage.ckpt_0000000000000076694 size 765058 bytes.
2021-06-08 23:38:43,994 INFO namenode.TransferFsImage: Opening connection to http://namenode.example.org:9870/imagetransfer?getedit=1&startTxId=76695&endTxId=76695&storageInfo=-65:1373659126:1620683988326:CID-5d1164fe-f8a5-4982-a503-096898971b41
2021-06-08 23:38:44,006 INFO common.Util: Combined time for file download and fsync to all disks took 0.00s. The file download took 0.00s at 512000.00 KB/s. Synchronous (fsync) write to disk of /opt/hadoop/local_data/secondarynamenode/current/edits_tmp_0000000000000076695-0000000000000076695_0000000001312727778 took 0.00s.
2021-06-08 23:38:44,007 INFO namenode.TransferFsImage: Downloaded file edits_tmp_0000000000000076695-0000000000000076695_0000000001312727778 size 0 bytes.
2021-06-08 23:38:44,007 INFO namenode.TransferFsImage: Opening connection to http://namenode.example.org:9870/imagetransfer?getedit=1&startTxId=76696&endTxId=76697&storageInfo=-65:1373659126:1620683988326:CID-5d1164fe-f8a5-4982-a503-096898971b41
2021-06-08 23:38:44,013 INFO common.Util: Combined time for file download and fsync to all disks took 0.00s. The file download took 0.00s at 0.00 KB/s. Synchronous (fsync) write to disk of /opt/hadoop/local_data/secondarynamenode/current/edits_tmp_0000000000000076696-0000000000000076697_0000000001312727791 took 0.00s.
2021-06-08 23:38:44,013 INFO namenode.TransferFsImage: Downloaded file edits_tmp_0000000000000076696-0000000000000076697_0000000001312727791 size 0 bytes.
2021-06-08 23:38:44,014 INFO namenode.TransferFsImage: Opening connection to http://namenode.example.org:9870/imagetransfer?getedit=1&startTxId=76698&endTxId=76699&storageInfo=-65:1373659126:1620683988326:CID-5d1164fe-f8a5-4982-a503-096898971b41
2021-06-08 23:38:44,019 INFO common.Util: Combined time for file download and fsync to all disks took 0.00s. The file download took 0.00s at 0.00 KB/s. Synchronous (fsync) write to disk of /opt/hadoop/local_data/secondarynamenode/current/edits_tmp_0000000000000076698-0000000000000076699_0000000001312727798 took 0.00s.
2021-06-08 23:38:44,019 INFO namenode.TransferFsImage: Downloaded file edits_tmp_0000000000000076698-0000000000000076699_0000000001312727798 size 0 bytes.
2021-06-08 23:38:44,102 INFO namenode.FSImageFormatPBINode: Loading 10962 INodes.
2021-06-08 23:38:44,322 INFO namenode.FSImageFormatProtobuf: Loaded FSImage in 0 seconds.
2021-06-08 23:38:44,322 INFO namenode.FSImage: Loaded image for txid 76694 from /opt/hadoop/local_data/secondarynamenode/current/fsimage_0000000000000076694
2021-06-08 23:38:44,323 INFO namenode.NameCache: initialized with 287 entries 10879 lookups
2021-06-08 23:38:44,339 INFO namenode.Checkpointer: Checkpointer about to load edits from 3 stream(s).
2021-06-08 23:38:44,342 INFO namenode.FSImage: Reading /opt/hadoop/local_data/secondarynamenode/current/edits_0000000000000076695-0000000000000076695 expecting start txid #76695
2021-06-08 23:38:44,343 INFO namenode.FSImage: Start loading edits file /opt/hadoop/local_data/secondarynamenode/current/edits_0000000000000076695-0000000000000076695 maxTxnsToRead = 9223372036854775807
2021-06-08 23:38:44,386 INFO namenode.FSImage: Loaded 1 edits file(s) (the last named /opt/hadoop/local_data/secondarynamenode/current/edits_0000000000000076695-0000000000000076695) of total size 1048576.0, total edits 1.0, total load time 28.0 ms
2021-06-08 23:38:44,430 INFO namenode.FSImageFormatProtobuf: Saving image file /opt/hadoop/local_data/secondarynamenode/current/fsimage.ckpt_0000000000000076699 using no compression
2021-06-08 23:38:44,618 INFO namenode.FSImageFormatProtobuf: Image file /opt/hadoop/local_data/secondarynamenode/current/fsimage.ckpt_0000000000000076699 of size 765058 bytes saved in 0 seconds .
2021-06-08 23:38:44,623 INFO namenode.FSImageTransactionalStorageInspector: No version file in /opt/hadoop/local_data/secondarynamenode
2021-06-08 23:38:44,627 INFO namenode.FSImageTransactionalStorageInspector: No version file in /opt/hadoop/local_data/secondarynamenode
2021-06-08 23:38:44,668 INFO namenode.TransferFsImage: Image Transfer timeout configured to 60000 milliseconds
2021-06-08 23:38:44,672 INFO namenode.TransferFsImage: Sending fileName: /opt/hadoop/local_data/secondarynamenode/current/fsimage_0000000000000076699, fileSize: 765058. Sent total: 765058 bytes. Size of last segment intended to send: -1 bytes.
2021-06-08 23:38:44,674 WARN namenode.SecondaryNameNode: Checkpoint done. New Image Size: 765058
2021-06-08 23:38:44,688 INFO namenode.FSImage: FSImageSaver clean checkpoint: txid=76699 when meet shutdown.
2021-06-08 23:38:44,688 INFO namenode.SecondaryNameNode: SHUTDOWN_MSG: 
/************************************************************
SHUTDOWN_MSG: Shutting down SecondaryNameNode at secondarynamenode/192.168.8.171
************************************************************/

Start secondary namenode.

$ systemctl start hadoop-secondarynamenode.service