ALL hadoop-mapreduce-examples.jar fail cdh6

[#1]

Please help I've spent 3 days and still can't get an example to run including pi, terasort or even wordcount

New installation and have turned on HA for yarn and also hdfs, and installed the yarn MapReduce Framework JARs as well. Cluster looks health as seen below:

If I try and run any of the examples ...they fail.

[hdfs@cdh-utility-1 opc]$ hadoop jar /opt/cloudera/parcels/CDH/lib/hadoop-mapreduce/hadoop-mapreduce-examples.jar pi 4 4
WARNING: Use "yarn jar" to launch YARN applications.
Number of Maps = 4
Samples per Map example.com= 4
Wrote input for Map #0
Wrote input for Map #1
Wrote input for Map #2
Wrote input for Map #3
Starting Job
20/02/26 17:09:47 INFO mapreduce.JobResourceUploader: Disabling Erasure Coding for path: /user/hdfs/.staging/job_1582677468069_0009
20/02/26 17:09:47 INFO input.FileInputFormat: Total input files to process : 4
20/02/26 17:09:47 INFO mapreduce.JobSubmitter: number of splits:4
20/02/26 17:09:47 INFO Configuration.deprecation: yarn.resourcemanager.zk-address is deprecated. Instead, use hadoop.zk.address
20/02/26 17:09:47 INFO Configuration.deprecation: yarn.resourcemanager.system-metrics-publisher.enabled is deprecated. Instead, use yarn.system-metrics-publisher.enabled
20/02/26 17:09:48 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1582677468069_0009
20/02/26 17:09:48 INFO mapreduce.JobSubmitter: Executing with tokens: []
20/02/26 17:09:48 INFO conf.Configuration: resource-types.xml not found
20/02/26 17:09:48 INFO resource.ResourceUtils: Unable to find 'resource-types.xml'.
20/02/26 17:09:48 INFO impl.YarnClientImpl: Submitted application application_1582677468069_0009
20/02/26 17:09:48 INFO mapreduce.Job: The url to track the job: http://cdh-master-1.private.cdhvcn.examplevcn.com:8088/proxy/application_1582677468069_0009/
20/02/26 17:09:48 INFO mapreduce.Job: Running job: job_1582677468069_0009
20/02/26 17:09:54 INFO mapreduce.Job: Job job_1582677468069_0009 running in uber mode : false
20/02/26 17:09:54 INFO mapreduce.Job: map 0% reduce 0%
20/02/26 17:09:56 INFO mapreduce.Job: Task Id : attempt_1582677468069_0009_m_000002_0, Status : FAILED
[2020-02-26 17:09:55.814]Exception from container-launch.
Container id: container_e02_1582677468069_0009_01_000004
Exit code: 1

[2020-02-26 17:09:55.815]Container exited with a non-zero exit code 1. Error file: prelaunch.err.
Last 4096 bytes of prelaunch.err :
Last 4096 bytes of stderr :
Error: Could not find or load main class NOT

[2020-02-26 17:09:55.816]Container exited with a non-zero exit code 1. Error file: prelaunch.err.
Last 4096 bytes of prelaunch.err :
Last 4096 bytes of stderr :
Error: Could not find or load main class NOT

20/02/26 17:09:56 INFO mapreduce.Job: Task Id : attempt_1582677468069_0009_m_000001_0, Status : FAILED
[2020-02-26 17:09:55.826]Exception from container-launch.
Container id: container_e02_1582677468069_0009_01_000003
Exit code: 1

[2020-02-26 17:09:55.853]Container exited with a non-zero exit code 1. Error file: prelaunch.err.
Last 4096 bytes of prelaunch.err :
Last 4096 bytes of stderr :
Error: Could not find or load main class NOT

[2020-02-26 17:09:55.853]Container exited with a non-zero exit code 1. Error file: prelaunch.err.
Last 4096 bytes of prelaunch.err :
Last 4096 bytes of stderr :
Error: Could not find or load main class NOT

20/02/26 17:09:56 INFO mapreduce.Job: Task Id : attempt_1582677468069_0009_m_000000_0, Status : FAILED
[2020-02-26 17:09:55.813]Exception from container-launch.
Container id: container_e02_1582677468069_0009_01_000002
Exit code: 1

[2020-02-26 17:09:55.816]Container exited with a non-zero exit code 1. Error file: prelaunch.err.
Last 4096 bytes of prelaunch.err :
Last 4096 bytes of stderr :
Error: Could not find or load main class NOT

[2020-02-26 17:09:55.816]Container exited with a non-zero exit code 1. Error file: prelaunch.err.
Last 4096 bytes of prelaunch.err :
Last 4096 bytes of stderr :
Error: Could not find or load main class NOT

20/02/26 17:09:56 INFO mapreduce.Job: Task Id : attempt_1582677468069_0009_m_000003_0, Status : FAILED
[2020-02-26 17:09:55.819]Exception from container-launch.
Container id: container_e02_1582677468069_0009_01_000005
Exit code: 1

[2020-02-26 17:09:55.852]Container exited with a non-zero exit code 1. Error file: prelaunch.err.
Last 4096 bytes of prelaunch.err :
Last 4096 bytes of stderr :
Error: Could not find or load main class NOT

[2020-02-26 17:09:55.853]Container exited with a non-zero exit code 1. Error file: prelaunch.err.
Last 4096 bytes of prelaunch.err :
Last 4096 bytes of stderr :
Error: Could not find or load main class NOT

[#2]

@HadoopBD,

I was able to reproduce your symptoms based on what I saw in the debug on my successful run.

Although I am sure there are a few ways this could happen, here is how I was able to get the same failure:

[2020-03-02 18:56:45.154]Container exited with a non-zero exit code 1. Error file: prelaunch.err.
Last 4096 bytes of prelaunch.err :
Last 4096 bytes of stderr :
Error: Could not find or load main class NOT

HOW I REPRODUCED THE ERROR:

(1)

In Cloudera Manager, open YARN configuration

(2)

Search for Map Task Java Opts Base

(3)

Update Map Task Java Opts Base by adding a space and then the word "NOT"

For example:

-Djava.net.preferIPv4Stack=true NOT

(4)

Save change.
Deploy Client Config
Restarted YARN (probably not necessary)

(5)

Ran hadoop jar /opt/cloudera/parcels/CDH/lib/hadoop-mapreduce/hadoop-mapreduce-examples.jar pi 4 4

(6)

Result was the error.  When I captured the application logs with the debugging I mentioned enabled, I could see that launch_container.sh issued the following Java command:

exec /bin/bash -c "$JAVA_HOME/bin/java -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -Djava.net.preferIPv4Stack=true NOT -Xmx820m ...

Since the word "NOT" does not have an option in front of it, Java interprets this as the class that should be run.

Based on the above example, I would say that increasing container log deletion delay and enabling debug as I described in previous posts will show us the problem.

Cheers,

Ben

[#3]

@bgooley 

You might be onto something there. Since I was not running 6.3...I tore down the cluster and rebuilt from scratch this time using CDH 6.3.2, enabled HA for yarn and hdfs, deployed yarn framework, created job history directory and created nodemanager application log directory, then set  Localized Dir Deletion Delay to 1800 and
Enable Container Launch Debug Information..  Then ran this:

sudo -u hdfs hadoop jar /opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/lib/hadoop-mapreduce/hadoop-mapreduce-examples.jar pi 4 4

then pulled the logs and looked for the bash line  and here is what I found:

echo "ls -l:" 1>"/var/log/cloudera/hadoop-yarn/container/application_1583199702433_0138/container_e03_1583199702433_0138_01_000014/directory.info"
ls -l 1>>"/var/log/cloudera/hadoop-yarn/container/application_1583199702433_0138/container_e03_1583199702433_0138_01_000014/directory.info"
echo "find -L . -maxdepth 5 -ls:" 1>>"/var/log/cloudera/hadoop-yarn/container/application_1583199702433_0138/container_e03_1583199702433_0138_01_000014/directory.info"
find -L . -maxdepth 5 -ls 1>>"/var/log/cloudera/hadoop-yarn/container/application_1583199702433_0138/container_e03_1583199702433_0138_01_000014/directory.info"
echo "broken symlinks(find -L . -maxdepth 5 -type l -ls):" 1>>"/var/log/cloudera/hadoop-yarn/container/application_1583199702433_0138/container_e03_1583199702433_0138_01_000014/directory.info"
find -L . -maxdepth 5 -type l -ls 1>>"/var/log/cloudera/hadoop-yarn/container/application_1583199702433_0138/container_e03_1583199702433_0138_01_000014/directory.info"
echo "Launching container"
exec /bin/bash -c "$JAVA_HOME/bin/java -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN NOT FOUND -Xmx820m -Djava.io.tmpdir=$PWD/tmp -Dlog4j.configuration=container-log4j.properties -Dyarn.app.container.log.dir=/var/log/cloudera/hadoop-yarn/container/application_1583199702433_0138/container_e03_1583199702433_0138_01_000014 -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA -Dhadoop.root.logfile=syslog org.apache.hadoop.mapred.YarnChild 10.0.2.7 33307 attempt_1583199702433_0138_m_000002_3 3298534883342 1>/var/log/cloudera/hadoop-yarn/container/application_1583199702433_0138/container_e03_1583199702433_0138_01_000014/stdout 2>/var/log/cloudera/hadoop-yarn/container/application_1583199702433_0138/container_e03_1583199702433_0138_01_000014/stderr "

End of LogType:launch_container.sh
************************************************************************************

so I then went into Yarn and looked for Map Task Java Opts Base

and found:

BINGO.  You hit the nail on the head.  Great find.  

[#4 · Accepted solution]

That's great!  You should be able to replace the "NOT FOUND" values for those two fields with:

-Djava.net.preferIPv4Stack=true

This will configure it as CM usually has by default.

Not sure how the NOT FOUND ended up there.

[#5]

yep already did and even looked for a tuning parameter based on the cloud shape I'm using and so ended up setting both of them to:

-Djava.net.preferIPv4Stack=true -Xms2368m -Xmx2368m

and now pi works.  Thanks a bunch.

[#6]

Great catch @bgooley!!