28730
DISCUSSIONS
101732
MEMBERS
3157
ARTICLES
Created 08-20-2015 08:26 PM
Hello - I am trying to make a remote job submission from websphere to cdh5.3.3.
My job is simple map-reduce build originally for mr1 cdh4.2 which is recomplied with mr2 jars and works fine when tested on cluster with hadoop jar ... command. No issue till this point.
Things become difficult when the same job is tried to submit via remote application running into websphere application server via RPC. After compiling the code with all the new jars and java 1.7 I am able to submit the job to Yarn RM. Which in return gives back job-id(application-id). But after that when yarn try to execute the container it throws less informative error "Exit code from container is : 1" and stack trace with the error on ShellCommandExecutor fail.
There is no more information even when I run the yarn in debug mode.
Sort of stuck at this point.
My development environment is windows ( eclipse and websphere is running on windows) and cluster cdh5.3.3 is depoyed on redhat with kerberos enabled. while reading on internet I saw the need for WindowsSecureContainerExecutor (WCE) https://hadoop.apache.org/docs/r2.6.0/hadoop-yarn/hadoop-yarn-site/SecureContainer.html#Overview. Does application submission behave differently when submit from windows or from linux.
Any clue in this regard will be helpful.
Thanks
MG