Categories
SysOps

How to validate Hadoop configuration files

Validate Hadoop configuration XML files.

Use conftest Hadoop client command to validate configuration XML files.

$ sudo -i -u hadoop hadoop conftest 
/opt/hadoop/hadoop-3.2.2/etc/hadoop/yarn-site.xml:
        Line 19: <property> has no <value>
/opt/hadoop/hadoop-3.2.2/etc/hadoop/hdfs-site.xml:
        bad conf file: Unexpected EOF; was expecting a close tag for element <configuration>
 at [row,col {unknown-source}]: [23,0]
/opt/hadoop/hadoop-3.2.2/etc/hadoop/kms-acls.xml: valid
/opt/hadoop/hadoop-3.2.2/etc/hadoop/kms-site.xml: valid
/opt/hadoop/hadoop-3.2.2/etc/hadoop/hadoop-policy.xml: valid
/opt/hadoop/hadoop-3.2.2/etc/hadoop/core-site.xml: valid
/opt/hadoop/hadoop-3.2.2/etc/hadoop/capacity-scheduler.xml: valid
/opt/hadoop/hadoop-3.2.2/etc/hadoop/mapred-site.xml:
        bad conf file: Unexpected close tag </configuration>; expected </property>.
 at [row,col {unknown-source}]: [36,15]
/opt/hadoop/hadoop-3.2.2/etc/hadoop/httpfs-site.xml: valid
Invalid file exists

Exit code will indicate success or failure.

$ echo $?
1

Validate a specific configuration XML file.

$ sudo -i -u hadoop hadoop conftest -conffile /opt/hadoop/hadoop-3.2.2/etc/hadoop/hdfs-site.xml
/opt/hadoop/hadoop-3.2.2/etc/hadoop/hdfs-site.xml:
        bad conf file: Unexpected EOF; was expecting a close tag for element <configuration>
 at [row,col {unknown-source}]: [23,0]
Invalid file exists

Exit code will indicate success or failure.

$ echo $?
1

Validate after fixing every individual issue found in configuration files.

$ sudo -i -u hadoop hadoop conftest
/opt/hadoop/hadoop-3.2.2/etc/hadoop/yarn-site.xml: valid
/opt/hadoop/hadoop-3.2.2/etc/hadoop/hdfs-site.xml: valid
/opt/hadoop/hadoop-3.2.2/etc/hadoop/kms-acls.xml: valid
/opt/hadoop/hadoop-3.2.2/etc/hadoop/kms-site.xml: valid
/opt/hadoop/hadoop-3.2.2/etc/hadoop/hadoop-policy.xml: valid
/opt/hadoop/hadoop-3.2.2/etc/hadoop/core-site.xml: valid
/opt/hadoop/hadoop-3.2.2/etc/hadoop/capacity-scheduler.xml: valid
/opt/hadoop/hadoop-3.2.2/etc/hadoop/mapred-site.xml: valid
/opt/hadoop/hadoop-3.2.2/etc/hadoop/httpfs-site.xml: valid
OK

Exit code will indicate success or failure.

$ echo $?
0

This command will validate configuration XML files only, so it does not mean that every value is correct.