Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Nodemanager.containermanager.localizer.ResourceLocalizationService [exitCode=137]

Nodemanager.containermanager.localizer.ResourceLocalizationService [exitCode=137]

Explorer

Hi all,

 

I’m having some problems with only one server in my Hadoop Infrastructure (worker node), all the jobs that Yarn schedules to this server to be elaborated always fail...

 

Any idea about the root cause of this issue? At the moment I've stopped the Yarn service of this node. Below the Exception that I got:

 

----------------------------------------------------------------------------------------------------------------------------------------------------

org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService              

Localizer failed

java.io.IOException: Application application_1516880199882_19310 initialization failed (exitCode=137) with output:

               at org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.startLocalizer(LinuxContainerExecutor.java:269)

               at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:1133)

Caused by: ExitCodeException exitCode=137:

               at org.apache.hadoop.util.Shell.runCommand(Shell.java:578)

               at org.apache.hadoop.util.Shell.run(Shell.java:481)

               at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:763)

               at org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.startLocalizer(LinuxContainerExecutor.java:260)

               ... 1 more

-----------------------------------------------------------------------------------------------------------------------------------------

WARN   org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor  

Exit code from container container_e111_1516880199882_19310_02_002043 startLocalizer is : 137

ExitCodeException exitCode=137:

               at org.apache.hadoop.util.Shell.runCommand(Shell.java:578)

               at org.apache.hadoop.util.Shell.run(Shell.java:481)

               at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:763)

               at org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.startLocalizer(LinuxContainerExecutor.java:260)

               at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:1133)

 ----------------------------------------------------------------------------------------------------------------------------------------------------

 

If I restart the service works for a few days, but suddenly this "Exception -> ExitCodeException exitCode=137" appear again…

 

 

### Permission of container executor ###

---Sr-s--- 1 root yarn 49520 Jul 13  2016 /opt/cloudera/parcels/CDH-5.8.0-1.cdh5.8.0.p0.42/lib/hadoop-yarn/bin/container-executor

 

Many thanks in advance for the kind cooperation!

 

Regards,

 

Alex