Categories
SysOps

How to leave Hadoop safe mode

The Hadoop cluster enters safe mode during the name node startup till the basic indicators are met and later in case of emergency, which means that the cluster enters read-only mode.

The client will immediately notice that the name node is in safe mode.

$ hdfs dfs -copyFromLocal -f /var/log/syslog /logs/backup/
copyFromLocal: Cannot create file/logs/backup/syslog._COPYING_. Name node is in safe mode.

Inspect log file on the name node.

$ tail -100  /opt/hadoop/hadoop-3.2.2/logs/hadoop-hadoop-namenode-namenode.log 
2021-05-18 22:53:04,014 WARN org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker: Space available on volume '/dev/mapper/cerberus--ssd-vm--100--disk--0' is 0, which is below the configured reserved amount 104857600
2021-05-18 22:53:04,025 WARN org.apache.hadoop.hdfs.server.namenode.FSNamesystem: NameNode low on available disk space. Entering safe mode.

Focus on resolving the issue before any other action.

Determine current safe mode status.

$ hdfs dfsadmin -safemode get
Safe mode is ON

Disable safe mode.

$ hdfs dfsadmin -safemode leave
Safe mode is OFF

Additional notes

There are other useful options like enter to deliberately enter safe mode, wait which is useful inside a shell script as it just waits till safe mode is off.

$ hdfs dfsadmin -safemode enter|leave|get|wait|forceExit 

Inspect mentioned earlier dfs.namenode.resource.du.reserved option.

Also dfs.namenode.safemode.threshold-pct, dfs.namenode.safemode.min.datanodes and dfs.namenode.safemode.extension options which are important during name node startup.

2021-05-18 23:35:12,479 INFO org.apache.hadoop.hdfs.StateChange: STATE* Safe mode ON, in safe mode extension. 
The reported blocks 5836 has reached the threshold 0.9990 of total blocks 5836. The number of live datanodes 3 has reached the minimum number 3. In safe mode extension. Safe mode will be turned off automatically in 9 seconds.