Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

cdh5.3.3. yarn map reduce application fail at container launch

cdh5.3.3. yarn map reduce application fail at container launch

Explorer

Hello - I am trying to make a remote job submission from websphere to cdh5.3.3. 

 

My job is simple map-reduce build originally for mr1 cdh4.2 which is recomplied with mr2 jars and works fine when tested on cluster with hadoop jar ... command. No issue till this point.

 

Things become difficult when the same job is tried to submit via remote application running into websphere application server via RPC. After compiling the code with all the new jars and java 1.7 I am able to submit the job to Yarn RM. Which in return gives back job-id(application-id). But after that when yarn try to execute the container it throws less informative error "Exit code from container is : 1" and stack trace with the error on ShellCommandExecutor fail.

 

There is no more information even when I run the yarn in debug mode.

 

Sort of stuck at this point. 

 

My development environment is windows ( eclipse and websphere is running on windows) and cluster cdh5.3.3 is depoyed on redhat with kerberos enabled. while reading on internet I saw the need for WindowsSecureContainerExecutor (WCE) https://hadoop.apache.org/docs/r2.6.0/hadoop-yarn/hadoop-yarn-site/SecureContainer.html#Overview. Does application submission behave differently when submit from windows or from linux. 

 

 

Any clue in this regard will be helpful.

 

Thanks

MG

7 REPLIES 7

Re: cdh5.3.3. yarn map reduce application fail at container launch

Master Guru

Is your application server running on Windows? (Edit: Sorry, missed your bottom note).

 

The https://issues.apache.org/jira/browse/YARN-1824 should resolve the issue w.r.t. path separator characters being the cause, and this is present in CDH 5.3.3, so as long as you are referencing the same version of libraries in your application server, that shouldn't be a problem.

 

Have you looked further at the stdout/stderr of the failing jobs to see what part of its environment setup may be failing, coming from Windows?

 

P.s. Note that Windows is not a fully supported client or server platform for Hadoop in CDH yet.

Re: cdh5.3.3. yarn map reduce application fail at container launch

Explorer

Harsh - Thanks for the reply. As per this link this bug was fixed in version 2.4.0 and cdh5.3.3 is based on 2.5.0 so this fix should be in cdh5.3.3. If my understanding is wrong let me know.

Regarding stdout/stderr there is no error stack other then this error.


Container id: container_1439576847667_0021_02_000001
Exit code: 1
Stack trace: ExitCodeException exitCode=1:
at org.apache.hadoop.util.Shell.runCommand(Shell.java:538)
at org.apache.hadoop.util.Shell.run(Shell.java:455)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:702)
at org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.launchContainer(LinuxContainerExecutor.java:290)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:299)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:81)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Shell output: main : command provided 1
main : user is devuser1
main : requested yarn user is devuser1
Container exited with a non-zero exit code 1
.Failing this attempt.. Failing the application.
 


Container just fail on launch with this error. I looked around for other logs but did not find any other info. let me know if there is specify place in can look for stdout/stderr logs.

 

Is there any place we can find this error documented with in cloudera to show its a know issue with windows in CDH5.3.3, also this give me another question our WebSphere runs on Solaris in higher environments will this problem persist with Solaris too.

 

 

Thanks, MG

Re: cdh5.3.3. yarn map reduce application fail at container launch

Super Collaborator

The exit codes of the executor are documented here. Based on your exit code you have the wrong number of arguments or the localiser fails.

Do you have full HDFS access from the webspahere container and are all the configurations on the classpath of the websphere container?

 

What happens at this point is that the shell starts an executable (setuid etc) and then the next step is starting the JVM from that.

 

It might throw more information and we normally just write to stdout/stderr. I suspect that the issue is the localisation and thus the local directories that need to exist on the node. Is the host also a nodemanager or not?

 

Wilfred

Re: cdh5.3.3. yarn map reduce application fail at container launch

Explorer

I have exactly the same error.

Despite full access to HDFS I have no additional information. While accessing node manager I find empty container's directory (no stdout or stderr files)

Re: cdh5.3.3. yarn map reduce application fail at container launch

Explorer

I found a solution.

If you launch map reduce job from windows client you have to specify parameter in mapred-site.xml

  <property>
    <name>mapreduce.app-submission.cross-platform</name>
    <value>true</value>
  </property>

And in the same file update mapreduce.application.classpath value

  <property>
    <name>mapreduce.application.classpath</name>
    <value>{{HADOOP_MAPRED_HOME}}/*,{{HADOOP_MAPRED_HOME}}/lib/*,{{MR2_CLASSPATH}}</value>
  </property>

 

Re: cdh5.3.3. yarn map reduce application fail at container launch

Explorer
Dubisiv - Thanks for sharing the information. I tried by setting both the values as explained by you but still no success. Wonder if you can share your environment configuration. Mine client is running in IBM WAS on windows 7 32 bit. CDH cluster is 5.3.3 on redhat. I will appreciate if you can share how you are submitting you job from client. What all properties you are setting up. Thanks in Advance.
Highlighted

Re: cdh5.3.3. yarn map reduce application fail at container launch

Explorer

Just an update. It worked for me now. In addition to above two properties I add one more property yarn.application.classpath with {{... }} option in place of $. It did not work for me just by adding at cluster so I sendup adding both place( Cluster and Client)  and it worked.

Don't have an account?
Coming from Hortonworks? Activate your account here