Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

ALL hadoop-mapreduce-examples.jar fail cdh6

Solved Go to solution
Highlighted

Re: ALL hadoop-mapreduce-examples.jar fail cdh6

Super Guru

@HadoopBD,

 

I was able to reproduce your symptoms based on what I saw in the debug on my successful run.

Although I am sure there are a few ways this could happen, here is how I was able to get the same failure:

[2020-03-02 18:56:45.154]Container exited with a non-zero exit code 1. Error file: prelaunch.err.
Last 4096 bytes of prelaunch.err :
Last 4096 bytes of stderr :
Error: Could not find or load main class NOT

 

HOW I REPRODUCED THE ERROR:

 

(1)

 

In Cloudera Manager, open YARN configuration

 

(2)

 

Search for Map Task Java Opts Base

 

(3)

 

Update Map Task Java Opts Base by adding a space and then the word "NOT"

For example:


-Djava.net.preferIPv4Stack=true NOT

 

(4)

 

Save change.
Deploy Client Config
Restarted YARN (probably not necessary)

 

(5)

 

Ran hadoop jar /opt/cloudera/parcels/CDH/lib/hadoop-mapreduce/hadoop-mapreduce-examples.jar pi 4 4

 

(6)

 

Result was the error.  When I captured the application logs with the debugging I mentioned enabled, I could see that launch_container.sh issued the following Java command:

exec /bin/bash -c "$JAVA_HOME/bin/java -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -Djava.net.preferIPv4Stack=true NOT -Xmx820m ...

Since the word "NOT" does not have an option in front of it, Java interprets this as the class that should be run.

 

Based on the above example, I would say that increasing container log deletion delay and enabling debug as I described in previous posts will show us the problem.

 

Cheers,

 

Ben

 

Highlighted

Re: ALL hadoop-mapreduce-examples.jar fail cdh6

Explorer

@bgooley 

You might be onto something there. Since I was not running 6.3...I tore down the cluster and rebuilt from scratch this time using CDH 6.3.2, enabled HA for yarn and hdfs, deployed yarn framework, created job history directory and created nodemanager application log directory, then set  Localized Dir Deletion Delay to 1800 and
Enable Container Launch Debug Information..  Then ran this:

 

 

sudo -u hdfs hadoop jar /opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/lib/hadoop-mapreduce/hadoop-mapreduce-examples.jar pi 4 4

 

then pulled the logs and looked for the bash line  and here is what I found:

echo "ls -l:" 1>"/var/log/cloudera/hadoop-yarn/container/application_1583199702433_0138/container_e03_1583199702433_0138_01_000014/directory.info"
ls -l 1>>"/var/log/cloudera/hadoop-yarn/container/application_1583199702433_0138/container_e03_1583199702433_0138_01_000014/directory.info"
echo "find -L . -maxdepth 5 -ls:" 1>>"/var/log/cloudera/hadoop-yarn/container/application_1583199702433_0138/container_e03_1583199702433_0138_01_000014/directory.info"
find -L . -maxdepth 5 -ls 1>>"/var/log/cloudera/hadoop-yarn/container/application_1583199702433_0138/container_e03_1583199702433_0138_01_000014/directory.info"
echo "broken symlinks(find -L . -maxdepth 5 -type l -ls):" 1>>"/var/log/cloudera/hadoop-yarn/container/application_1583199702433_0138/container_e03_1583199702433_0138_01_000014/directory.info"
find -L . -maxdepth 5 -type l -ls 1>>"/var/log/cloudera/hadoop-yarn/container/application_1583199702433_0138/container_e03_1583199702433_0138_01_000014/directory.info"
echo "Launching container"
exec /bin/bash -c "$JAVA_HOME/bin/java -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN NOT FOUND -Xmx820m -Djava.io.tmpdir=$PWD/tmp -Dlog4j.configuration=container-log4j.properties -Dyarn.app.container.log.dir=/var/log/cloudera/hadoop-yarn/container/application_1583199702433_0138/container_e03_1583199702433_0138_01_000014 -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA -Dhadoop.root.logfile=syslog org.apache.hadoop.mapred.YarnChild 10.0.2.7 33307 attempt_1583199702433_0138_m_000002_3 3298534883342 1>/var/log/cloudera/hadoop-yarn/container/application_1583199702433_0138/container_e03_1583199702433_0138_01_000014/stdout 2>/var/log/cloudera/hadoop-yarn/container/application_1583199702433_0138/container_e03_1583199702433_0138_01_000014/stderr "

End of LogType:launch_container.sh
************************************************************************************

 

so I then went into Yarn and looked for Map Task Java Opts Base

and found:

YarnErrorFound.png

 

BINGO.  You hit the nail on the head.  Great find.  

Highlighted

Re: ALL hadoop-mapreduce-examples.jar fail cdh6

Super Guru

That's great!  You should be able to replace the "NOT FOUND" values for those two fields with:

 

-Djava.net.preferIPv4Stack=true

This will configure it as CM usually has by default.

Not sure how the NOT FOUND ended up there.

View solution in original post

Re: ALL hadoop-mapreduce-examples.jar fail cdh6

Explorer

yep already did and even looked for a tuning parameter based on the cloud shape I'm using and so ended up setting both of them to:

-Djava.net.preferIPv4Stack=true -Xms2368m -Xmx2368m

and now pi works.  Thanks a bunch.

 

 

 

Highlighted

Re: ALL hadoop-mapreduce-examples.jar fail cdh6

Guru
Great catch @bgooley!!
Don't have an account?
Coming from Hortonworks? Activate your account here