Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Mapreduce - Container exited with a non-zero exit code 1

Highlighted

Mapreduce - Container exited with a non-zero exit code 1

Explorer
Hi All

My MR job is failing with container exit code non zero.

Any suggestions on what could be the root cause here ? Thanks.

Log :
2019-09-19 12:22:26,591 INFO [main] org.apache.hadoop.yarn.client.RMProxy: Connecting to ResourceManager at ip-10-132-29-223.ec2.internal/10.132.29.223:8030 2019-09-19 12:22:26,673 INFO [main] org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator: maxContainerCapability: <memory:23424, vCores:128> 2019-09-19 12:22:26,673 INFO [main] org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator: queue: default 2019-09-19 12:22:26,677 INFO [main] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Upper limit on the thread pool size is 500 2019-09-19 12:22:26,678 INFO [main] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: The thread pool initial size is 10 2019-09-19 12:22:26,680 INFO [main] org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy: yarn.client.max-cached-nodemanagers-proxies : 0 2019-09-19 12:22:26,686 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: job_1568811337583_0009Job Transitioned from INITED to SETUP 2019-09-19 12:22:26,691 INFO [CommitterEvent Processor #0] org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler: Processing the event EventType: JOB_SETUP 2019-09-19 12:22:26,702 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: job_1568811337583_0009Job Transitioned from SETUP to RUNNING 2019-09-19 12:22:26,728 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1568811337583_0009_m_000000 Task Transitioned from NEW to SCHEDULED 2019-09-19 12:22:26,739 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1568811337583_0009_m_000000_0 TaskAttempt Transitioned from NEW to UNASSIGNED 2019-09-19 12:22:26,744 INFO [Thread-85] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: mapResourceRequest:<memory:1024, vCores:1> 2019-09-19 12:22:26,762 INFO [eventHandlingThread] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Event Writer setup for JobId: job_1568811337583_0009, File: hdfs://ip-10-132-29-223.ec2.internal:8020/tmp/hadoop-yarn/staging/infa/.staging/job_1568811337583_0009/job_1568811337583_0009_1.jhist 2019-09-19 12:22:27,676 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Before Scheduling: PendingReds:0 ScheduledMaps:1 ScheduledReds:0 AssignedMaps:0 AssignedReds:0 CompletedMaps:0 CompletedReds:0 ContAlloc:0 ContRel:0 HostLocal:0 RackLocal:0 2019-09-19 12:22:27,724 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: getResources() for application_1568811337583_0009: ask=1 release= 0 newContainers=0 finishedContainers=0 resourcelimit=<memory:16384, vCores:1> knownNMs=1 2019-09-19 12:22:28,735 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Got allocated containers 1 2019-09-19 12:22:28,737 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned container container_1568811337583_0009_01_000002 to attempt_1568811337583_0009_m_000000_0 2019-09-19 12:22:28,738 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: After Scheduling: PendingReds:0 ScheduledMaps:0 ScheduledReds:0 AssignedMaps:1 AssignedReds:0 CompletedMaps:0 CompletedReds:0 ContAlloc:1 ContRel:0 HostLocal:0 RackLocal:0 2019-09-19 12:22:28,811 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: The job-jar file on the remote FS is hdfs://ip-10-132-29-223.ec2.internal:8020/tmp/hadoop-yarn/staging/infa/.staging/job_1568811337583_0009/job.jar 2019-09-19 12:22:28,813 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: The job-conf file on the remote FS is /tmp/hadoop-yarn/staging/infa/.staging/job_1568811337583_0009/job.xml 2019-09-19 12:22:28,814 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Adding #0 tokens and #1 secret keys for NM use for launching container 2019-09-19 12:22:28,814 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Size of containertokens_dob is 1 2019-09-19 12:22:28,815 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Putting shuffle token in serviceData 2019-09-19 12:22:28,845 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1568811337583_0009_m_000000_0 TaskAttempt Transitioned from UNASSIGNED to ASSIGNED 2019-09-19 12:22:28,850 INFO [ContainerLauncher #0] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_REMOTE_LAUNCH for container container_1568811337583_0009_01_000002 taskAttempt attempt_1568811337583_0009_m_000000_0 2019-09-19 12:22:28,852 INFO [ContainerLauncher #0] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Launching attempt_1568811337583_0009_m_000000_0 2019-09-19 12:22:28,920 INFO [ContainerLauncher #0] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Shuffle port returned by ContainerManager for attempt_1568811337583_0009_m_000000_0 : 13562 2019-09-19 12:22:28,923 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: TaskAttempt: [attempt_1568811337583_0009_m_000000_0] using containerId: [container_1568811337583_0009_01_000002 on NM: [USE1BTRGBAEMR02.AWS.ABG.COM.aws.abg.com:8041] 2019-09-19 12:22:28,927 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1568811337583_0009_m_000000_0 TaskAttempt Transitioned from ASSIGNED to RUNNING 2019-09-19 12:22:28,928 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1568811337583_0009_m_000000 Task Transitioned from SCHEDULED to RUNNING 2019-09-19 12:22:29,741 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: getResources() for application_1568811337583_0009: ask=1 release= 0 newContainers=0 finishedContainers=0 resourcelimit=<memory:14336, vCores:1> knownNMs=1 2019-09-19 12:22:30,516 INFO [Socket Reader #1 for port 35161] SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for job_1568811337583_0009 (auth:SIMPLE) 2019-09-19 12:22:30,537 INFO [IPC Server handler 0 on 35161] org.apache.hadoop.mapred.TaskAttemptListenerImpl: JVM with ID : jvm_1568811337583_0009_m_000002 asked for a task 2019-09-19 12:22:30,537 INFO [IPC Server handler 0 on 35161] org.apache.hadoop.mapred.TaskAttemptListenerImpl: JVM with ID: jvm_1568811337583_0009_m_000002 given task: attempt_1568811337583_0009_m_000000_0 2019-09-19 12:22:32,751 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Received completed container container_1568811337583_0009_01_000002 2019-09-19 12:22:32,752 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: After Scheduling: PendingReds:0 ScheduledMaps:0 ScheduledReds:0 AssignedMaps:0 AssignedReds:0 CompletedMaps:0 CompletedReds:0 ContAlloc:1 ContRel:0 HostLocal:0 RackLocal:0 2019-09-19 12:22:32,752 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics report from attempt_1568811337583_0009_m_000000_0: Exception from container-launch. Container id: container_1568811337583_0009_01_000002 Exit code: 1 Stack trace: ExitCodeException exitCode=1: at org.apache.hadoop.util.Shell.runCommand(Shell.java:972) at org.apache.hadoop.util.Shell.run(Shell.java:869) at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:1170) at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:235) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:299) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:83) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Container exited with a non-zero exit code 1