Hi all,
I’m having some problems with only one server in my Hadoop Infrastructure (worker node), all the jobs that Yarn schedules to this server to be elaborated always fail...
Any idea about the root cause of this issue? At the moment I've stopped the Yarn service of this node. Below the Exception that I got:
----------------------------------------------------------------------------------------------------------------------------------------------------
org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService
Localizer failed
java.io.IOException: Application application_1516880199882_19310 initialization failed (exitCode=137) with output:
at org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.startLocalizer(LinuxContainerExecutor.java:269)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:1133)
Caused by: ExitCodeException exitCode=137:
at org.apache.hadoop.util.Shell.runCommand(Shell.java:578)
at org.apache.hadoop.util.Shell.run(Shell.java:481)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:763)
at org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.startLocalizer(LinuxContainerExecutor.java:260)
... 1 more
-----------------------------------------------------------------------------------------------------------------------------------------
WARN org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor
Exit code from container container_e111_1516880199882_19310_02_002043 startLocalizer is : 137
ExitCodeException exitCode=137:
at org.apache.hadoop.util.Shell.runCommand(Shell.java:578)
at org.apache.hadoop.util.Shell.run(Shell.java:481)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:763)
at org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.startLocalizer(LinuxContainerExecutor.java:260)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:1133)
----------------------------------------------------------------------------------------------------------------------------------------------------
If I restart the service works for a few days, but suddenly this "Exception -> ExitCodeException exitCode=137" appear again…
### Permission of container executor ###
---Sr-s--- 1 root yarn 49520 Jul 13 2016 /opt/cloudera/parcels/CDH-5.8.0-1.cdh5.8.0.p0.42/lib/hadoop-yarn/bin/container-executor
Many thanks in advance for the kind cooperation!
Regards,
Alex
