Categories
SysOps

How to load native Hadoop libraries

Hadoop has a native implementation for certain components, but sometimes it does not load automatically.

This is mainly the case of Hadoop 2.

The issue

The message that indicates an issue.

$ hdfs dfsadmin -printTopology          
21/05/13 21:35:08 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... 
using builtin-java classes where applicable
Rack: /default-rack
   192.168.8.173:9866 (datanode1.example.org)
   192.168.8.174:9866 (datanode2.example.org)
   192.168.8.175:9866 (datanode3.example.org)

Check all available libraries.

$ hdfs checknative -a
21/05/13 21:41:20 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... 
using builtin-java classes where applicable
Native library checking:
hadoop:  false
zlib:    false
snappy:  false 
lz4:     false
bzip2:   false
openssl: false Cannot load libcrypto.so (libcrypto.so: cannot open shared object file: No such file or directory)!
2021-05-13 21:41:20,958 INFO util.ExitUtil: Exiting with status 1: ExitException

The solution

The solution is to provide a path for the native libraries using HADOOP_OPTS variable. Note, the path must match your installation.

$ export HADOOP_OPTS="-Djava.library.path=/opt/hadoop/hadoop-2.7.3/lib/native/"
$ hdfs dfsadmin -printTopology          
Rack: /default-rack
   192.168.8.173:9866 (datanode1.example.org)
   192.168.8.174:9866 (datanode2.example.org)
   192.168.8.175:9866 (datanode3.example.org)

Ensure that HADOOP_OPTS variable is defined at login.

$ echo 'export HADOOP_OPTS="$HADOOP_OPTS -Djava.library.path=/opt/hadoop/hadoop-2.7.3/lib/native/"' | sudo tee /etc/profile.d/hadoop_opts_native.sh

Check all available libraries.

$ hdfs checknative -a
Native library checking:
hadoop:  true /opt/hadoop/hadoop-2.7.3/lib/native/libhadoop.so.1.0.0
zlib:    true /lib/x86_64-linux-gnu/libz.so.1
snappy:  true /lib/x86_64-linux-gnu/libsnappy.so.1 
lz4:     true revision:99
bzip2:   false
openssl: false Cannot load libcrypto.so (libcrypto.so: cannot open shared object file: No such file or directory)!
2021-05-13 21:43:47,019 INFO util.ExitUtil: Exiting with status 1: ExitException

Now everything is clear, so you can fix the remaining issues.

Additional notes

Native Libraries Guide