Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

HDP 2.4.0 Sandbox: MR AMs warn about "Unable to load native-hadoop" / impact on SCR/ZCR?

Highlighted

HDP 2.4.0 Sandbox: MR AMs warn about "Unable to load native-hadoop" / impact on SCR/ZCR?

New Contributor

When running a simple MR job (e.g., teragen) on HDP 2.4.0 Sandbox the logs always containg warnings about being "Unable to load native-hadoop":

[root@sandbox ~]# yarn jar /usr/hdp/current/hadoop-mapreduce-client/hadoop-mapreduce-examples-*.jar teragen 10000 tmp/teragenout

This results in the following error in the AM syslog:

2016-06-01 13:13:39,796 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Created MRAppMaster for application appattempt_1464782258427_0003_000001
2016-06-01 13:13:40,114 WARN [main] org.apache.hadoop.util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2016-06-01 13:13:40,161 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Executing with tokens:
2016-06-01 13:13:40,161 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Kind: YARN_AM_RM_TOKEN, Service: , Ident: (appAttemptId { application_id { id: 3 cluster_timestamp: 1464782258427 } attemptId: 1 } keyId: 832873100)
2016-06-01 13:13:40,285 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Using mapred newApiCommitter.
2016-06-01 13:13:40,286 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: OutputCommitter set in config null
2016-06-01 13:13:40,322 INFO [main] org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter: File Output Committer Algorithm version is 1
2016-06-01 13:13:40,322 INFO [main] org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
2016-06-01 13:13:40,865 WARN [main] org.apache.hadoop.hdfs.shortcircuit.DomainSocketFactory: The short-circuit local reads feature cannot be used because libhadoop cannot be loaded.

As I am interested in testing short-circuit reads (SCR) / cached zero copy reads (ZCR), I am worried about the two warnings above:

  1. Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
  2. The short-circuit local reads feature cannot be used because libhadoop cannot be loaded

Any hints how to solve this warnings?

Or any experiences whether that affects SCR / ZCR?

4 REPLIES 4

Re: HDP 2.4.0 Sandbox: MR AMs warn about "Unable to load native-hadoop" / impact on SCR/ZCR?

@Michael Häusler

Can you please share the output of below command?

hadoop checknative -a 

Also do you see same WARN when running job with "hadoop" instead of "yarn" command?

Re: HDP 2.4.0 Sandbox: MR AMs warn about "Unable to load native-hadoop" / impact on SCR/ZCR?

New Contributor

thanks @Jitendra Yadav for having a look at this.

results from checknative look good:

[root@sandbox ~]# hadoop checknative -a
16/06/01 13:58:36 INFO bzip2.Bzip2Factory: Successfully loaded & initialized native-bzip2 library system-native
16/06/01 13:58:36 INFO zlib.ZlibFactory: Successfully loaded & initialized native-zlib library
Native library checking:
hadoop:  true /usr/hdp/2.4.0.0-169/hadoop/lib/native/libhadoop.so.1.0.0
zlib:    true /lib64/libz.so.1
snappy:  true /usr/hdp/2.4.0.0-169/hadoop/lib/native/libsnappy.so.1
lz4:     true revision:99
bzip2:   true /lib64/libbz2.so.1
openssl: true /usr/lib64/libcrypto.so

Yes, I get exactly the same WARNs when running teragen with "hadoop".

Re: HDP 2.4.0 Sandbox: MR AMs warn about "Unable to load native-hadoop" / impact on SCR/ZCR?

New Contributor

thanks @Jitendra Yadav for your comments so far. Did you have a chance to look at the output of checknative? Should we check anywhere else? Any hints would be highly appreciated.

Re: HDP 2.4.0 Sandbox: MR AMs warn about "Unable to load native-hadoop" / impact on SCR/ZCR?

Hi @Michael Häusler Sorry I missed that post. However I tested this on my cluster couple of days back where I have hbase installed also and found out that these error messages are coming in AM container logs only but I have also enabled Hbase short circuit which doesn't have any issue so i think we can ignore this WARN in yarn.

Don't have an account?
Coming from Hortonworks? Activate your account here