question error Container exited with a non-zero exit code 137 Killed by external signal in Support Questions

question error Container exited with a non-zero exit code 137 Killed by external signal in Support Questions https://community.cloudera.com/t5/Support-Questions/error-Container-exited-with-a-non-zero-exit-code-137-Killed/m-p/67427#M4335 <P>All hive jobs started to fail with this error:</P><P> </P><P> </P><PRE>2018-05-18 08:50:58,682 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Created MRAppMaster for application appattempt_1526609705906_0273_000001 2018-05-18 08:50:59,959 WARN [main] org.apache.hadoop.util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2018-05-18 08:51:00,073 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: OutputCommitter set in config org.apache.hadoop.hive.ql.io.HiveFileFormatUtils$NullOutputCommitter 2018-05-18 08:51:00,075 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: OutputCommitter is org.apache.hadoop.hive.ql.io.HiveFileFormatUtils$NullOutputCommitter 2018-05-18 08:51:00,123 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.jobhistory.EventType for class org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler 2018-05-18 08:51:00,126 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventType for class org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler 2018-05-18 08:51:00,132 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncher$EventType for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$ContainerLauncherRouter 2018-05-18 08:51:00,169 INFO [main] org.apache.hadoop.mapreduce.v2.jobhistory.JobHistoryUtils: Default file system [hdfs://ip-172-31-4-192.ap-south-1.compute.internal:8020] 2018-05-18 08:51:00,194 INFO [main] org.apache.hadoop.mapreduce.v2.jobhistory.JobHistoryUtils: Default file system [hdfs://ip-172-31-4-192.ap-south-1.compute.internal:8020] 2018-05-18 08:51:00,212 INFO [main] org.apache.hadoop.mapreduce.v2.jobhistory.JobHistoryUtils: Default file system [hdfs://ip-172-31-4-192.ap-south-1.compute.internal:8020] 2018-05-18 08:51:00,223 INFO [main] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Emitting job history data to the timeline server is not enabled 2018-05-18 08:51:00,277 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.job.event.JobFinishEvent$Type for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobFinishEventHandler 2018-05-18 08:51:00,537 INFO [main] org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from hadoop-metrics2.properties 2018-05-18 08:51:00,605 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot period at 10 second(s). 2018-05-18 08:51:00,605 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: MRAppMaster metrics system started 2018-05-18 08:51:00,625 INFO [main] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Adding job token for job_1526609705906_0273 to jobTokenSecretManager 2018-05-18 08:51:00,825 INFO [main] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Not uberizing job_1526609705906_0273 because: not enabled; too many reduces; too much input; 2018-05-18 08:51:00,857 INFO [main] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Input size for job job_1526609705906_0273 = 1155935596. Number of splits = 4 2018-05-18 08:51:00,861 INFO [main] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Number of reduces for job job_1526609705906_0273 = 18 2018-05-18 08:51:00,861 INFO [main] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: job_1526609705906_0273Job Transitioned from NEW to INITED 2018-05-18 08:51:00,863 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: MRAppMaster launching normal, non-uberized, multi-container job job_1526609705906_0273. 2018-05-18 08:51:00,901 INFO [main] org.apache.hadoop.ipc.CallQueueManager: Using callQueue: class java.util.concurrent.LinkedBlockingQueue queueCapacity: 100 2018-05-18 08:51:00,912 INFO [Socket Reader #1 for port 42741] org.apache.hadoop.ipc.Server: Starting Socket Reader #1 for port 42741 2018-05-18 08:51:00,936 INFO [main] org.apache.hadoop.yarn.factories.impl.pb.RpcServerFactoryPBImpl: Adding protocol org.apache.hadoop.mapreduce.v2.api.MRClientProtocolPB to the server 2018-05-18 08:51:00,936 INFO [IPC Server Responder] org.apache.hadoop.ipc.Server: IPC Server Responder: starting 2018-05-18 08:51:00,937 INFO [IPC Server listener on 42741] org.apache.hadoop.ipc.Server: IPC Server listener on 42741: starting 2018-05-18 08:51:00,938 INFO [main] org.apache.hadoop.mapreduce.v2.app.client.MRClientService: Instantiated MRClientService at ip-172-31-13-118.ap-south-1.compute.internal/172.31.13.118:42741 2018-05-18 08:51:01,021 INFO [main] org.mortbay.log: Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog 2018-05-18 08:51:01,031 INFO [main] org.apache.hadoop.security.authentication.server.AuthenticationFilter: Unable to initialize FileSignerSecretProvider, falling back to use random secrets. 2018-05-18 08:51:01,036 INFO [main] org.apache.hadoop.http.HttpRequestLog: Http request log for http.requests.mapreduce is not defined 2018-05-18 08:51:01,044 INFO [main] org.apache.hadoop.http.HttpServer2: Added global filter 'safety' (class=org.apache.hadoop.http.HttpServer2$QuotingInputFilter) 2018-05-18 08:51:01,050 INFO [main] org.apache.hadoop.http.HttpServer2: Added filter AM_PROXY_FILTER (class=org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter) to context mapreduce 2018-05-18 08:51:01,050 INFO [main] org.apache.hadoop.http.HttpServer2: Added filter AM_PROXY_FILTER (class=org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter) to context static 2018-05-18 08:51:01,053 INFO [main] org.apache.hadoop.http.HttpServer2: adding path spec: /mapreduce/* 2018-05-18 08:51:01,053 INFO [main] org.apache.hadoop.http.HttpServer2: adding path spec: /ws/* 2018-05-18 08:51:01,061 INFO [main] org.apache.hadoop.http.HttpServer2: Jetty bound to port 36178 2018-05-18 08:51:01,061 INFO [main] org.mortbay.log: jetty-6.1.26.cloudera.4 2018-05-18 08:51:01,088 INFO [main] org.mortbay.log: Extract jar:file:/opt/cloudera/parcels/CDH-5.10.1-1.cdh5.10.1.p0.10/jars/hadoop-yarn-common-2.6.0-cdh5.10.1.jar!/webapps/mapreduce to /tmp/Jetty_0_0_0_0_36178_mapreduce____davbey/webapp 2018-05-18 08:51:01,355 INFO [main] org.mortbay.log: Started HttpServer2$SelectChannelConnectorWithSafeStartup@0.0.0.0:36178 2018-05-18 08:51:01,355 INFO [main] org.apache.hadoop.yarn.webapp.WebApps: Web app /mapreduce started at 36178 2018-05-18 08:51:01,697 INFO [main] org.apache.hadoop.yarn.webapp.WebApps: Registered webapp guice modules 2018-05-18 08:51:01,701 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.speculate.DefaultSpeculator: JOB_CREATE job_1526609705906_0273 2018-05-18 08:51:01,704 INFO [main] org.apache.hadoop.ipc.CallQueueManager: Using callQueue: class java.util.concurrent.LinkedBlockingQueue queueCapacity: 3000 2018-05-18 08:51:01,704 INFO [Socket Reader #1 for port 34196] org.apache.hadoop.ipc.Server: Starting Socket Reader #1 for port 34196 2018-05-18 08:51:01,712 INFO [IPC Server Responder] org.apache.hadoop.ipc.Server: IPC Server Responder: starting 2018-05-18 08:51:01,712 INFO [IPC Server listener on 34196] org.apache.hadoop.ipc.Server: IPC Server listener on 34196: starting 2018-05-18 08:51:01,732 INFO [main] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: nodeBlacklistingEnabled:true 2018-05-18 08:51:01,732 INFO [main] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: maxTaskFailuresPerNode is 3 2018-05-18 08:51:01,732 INFO [main] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: blacklistDisablePercent is 33 2018-05-18 08:51:01,795 INFO [main] org.apache.hadoop.yarn.client.RMProxy: Connecting to ResourceManager at ip-172-31-4-192.ap-south-1.compute.internal/172.31.4.192:8030 2018-05-18 08:51:01,852 INFO [main] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: maxContainerCapability: <memory:8192, vCores:6> 2018-05-18 08:51:01,852 INFO [main] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: queue: root.users.hue 2018-05-18 08:51:01,855 INFO [main] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Upper limit on the thread pool size is 500 2018-05-18 08:51:01,855 INFO [main] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: The thread pool initial size is 10 2018-05-18 08:51:01,864 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: job_1526609705906_0273Job Transitioned from INITED to SETUP 2018-05-18 08:51:01,865 INFO [CommitterEvent Processor #0] org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler: Processing the event EventType: JOB_SETUP 2018-05-18 08:51:01,867 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: job_1526609705906_0273Job Transitioned from SETUP to RUNNING 2018-05-18 08:51:01,936 INFO [eventHandlingThread] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Event Writer setup for JobId: job_1526609705906_0273, File: hdfs://ip-172-31-4-192.ap-south-1.compute.internal:8020/user/hue/.staging/job_1526609705906_0273/job_1526609705906_0273_1.jhist 2018-05-18 08:51:01,947 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1526609705906_0273_m_000000 Task Transitioned from NEW to SCHEDULED 2018-05-18 08:51:01,983 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1526609705906_0273_m_000001 Task Transitioned from NEW to SCHEDULED 2018-05-18 08:51:02,010 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1526609705906_0273_m_000002 Task Transitioned from NEW to SCHEDULED 2018-05-18 08:51:02,037 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1526609705906_0273_m_000003 Task Transitioned from NEW to SCHEDULED 2018-05-18 08:51:02,038 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1526609705906_0273_r_000000 Task Transitioned from NEW to SCHEDULED 2018-05-18 08:51:02,038 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1526609705906_0273_r_000001 Task Transitioned from NEW to SCHEDULED 2018-05-18 08:51:02,039 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1526609705906_0273_r_000002 Task Transitioned from NEW to SCHEDULED 2018-05-18 08:51:02,039 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1526609705906_0273_r_000003 Task Transitioned from NEW to SCHEDULED 2018-05-18 08:51:02,040 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1526609705906_0273_r_000004 Task Transitioned from NEW to SCHEDULED 2018-05-18 08:51:02,048 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1526609705906_0273_r_000015 Task Transitioned from NEW to SCHEDULED 2018-05-18 08:51:02,048 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1526609705906_0273_r_000016 Task Transitioned from NEW to SCHEDULED 2018-05-18 08:51:02,049 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1526609705906_0273_r_000017 Task Transitioned from NEW to SCHEDULED 2018-05-18 08:51:02,051 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1526609705906_0273_m_000000_0 TaskAttempt Transitioned from NEW to UNASSIGNED 2018-05-18 08:51:02,051 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1526609705906_0273_m_000001_0 TaskAttempt Transitioned from NEW to UNASSIGNED 2018-05-18 08:51:02,051 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1526609705906_0273_m_000002_0 TaskAttempt Transitioned from NEW to UNASSIGNED 2018-05-18 08:51:02,051 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1526609705906_0273_m_000003_0 TaskAttempt Transitioned from NEW to UNASSIGNED 2018-05-18 08:51:02,051 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1526609705906_0273_r_000000_0 TaskAttempt Transitioned from NEW to UNASSIGNED 2018-05-18 08:51:02,054 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1526609705906_0273_r_000014_0 TaskAttempt Transitioned from NEW to UNASSIGNED 2018-05-18 08:51:02,054 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1526609705906_0273_r_000016_0 TaskAttempt Transitioned from NEW to UNASSIGNED 2018-05-18 08:51:02,054 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1526609705906_0273_r_000017_0 TaskAttempt Transitioned from NEW to UNASSIGNED 2018-05-18 08:51:02,056 INFO [Thread-52] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: mapResourceRequest:<memory:1024, vCores:1> 2018-05-18 08:51:02,073 INFO [Thread-52] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: reduceResourceRequest:<memory:1024, vCores:1> 2018-05-18 08:51:02,448 INFO [eventHandlingThread] org.apache.hadoop.mapreduce.v2.jobhistory.JobHistoryUtils: Default file system [hdfs://ip-172-31-4-192.ap-south-1.compute.internal:8020] 2018-05-18 08:51:02,854 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Before Scheduling: PendingReds:18 ScheduledMaps:4 ScheduledReds:0 AssignedMaps:0 AssignedReds:0 CompletedMaps:0 CompletedReds:0 ContAlloc:0 ContRel:0 HostLocal:0 RackLocal:0 2018-05-18 08:51:02,892 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: getResources() for application_1526609705906_0273: ask=6 release= 0 newContainers=0 finishedContainers=0 resourcelimit=<memory:101888, vCores:54> knownNMs=6 2018-05-18 08:51:02,892 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Recalculating schedule, headroom=<memory:101888, vCores:54> 2018-05-18 08:51:02,892 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Reduce slow start threshold not met. completedMapsForReduceSlowstart 4 2018-05-18 08:51:02,996 INFO [IPC Server handler 0 on 42741] org.apache.hadoop.mapreduce.v2.app.client.MRClientService: Getting task report for MAP job_1526609705906_0273. Report-size will be 4 2018-05-18 08:51:03,051 INFO [IPC Server handler 0 on 42741] org.apache.hadoop.mapreduce.v2.app.client.MRClientService: Getting task report for REDUCE job_1526609705906_0273. Report-size will be 18 2018-05-18 08:51:03,904 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Got allocated containers 4 2018-05-18 08:51:03,906 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned container container_1526609705906_0273_01_000002 to attempt_1526609705906_0273_m_000001_0 2018-05-18 08:51:03,907 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned container container_1526609705906_0273_01_000003 to attempt_1526609705906_0273_m_000000_0 2018-05-18 08:51:03,908 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned container container_1526609705906_0273_01_000004 to attempt_1526609705906_0273_m_000002_0 2018-05-18 08:51:03,908 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned container container_1526609705906_0273_01_000005 to attempt_1526609705906_0273_m_000003_0 2018-05-18 08:51:03,908 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Recalculating schedule, headroom=<memory:97792, vCores:50> 2018-05-18 08:51:03,908 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Reduce slow start threshold not met. completedMapsForReduceSlowstart 4 2018-05-18 08:51:03,908 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: After Scheduling: PendingReds:18 ScheduledMaps:0 ScheduledReds:0 AssignedMaps:4 AssignedReds:0 CompletedMaps:0 CompletedReds:0 ContAlloc:4 ContRel:0 HostLocal:1 RackLocal:3 2018-05-18 08:51:03,974 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: The job-jar file on the remote FS is hdfs://ip-172-31-4-192.ap-south-1.compute.internal:8020/user/hue/.staging/job_1526609705906_0273/job.jar 2018-05-18 08:51:03,976 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: The job-conf file on the remote FS is /user/hue/.staging/job_1526609705906_0273/job.xml 2018-05-18 08:51:03,985 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Adding #0 tokens and #1 secret keys for NM use for launching container 2018-05-18 08:51:03,985 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Size of containertokens_dob is 1 2018-05-18 08:51:03,985 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Putting shuffle token in serviceData 2018-05-18 08:51:04,022 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapred.JobConf: Task java-opts do not specify heap size. Setting task attempt jvm max heap size to -Xmx820m 2018-05-18 08:51:04,026 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1526609705906_0273_m_000001_0 TaskAttempt Transitioned from UNASSIGNED to ASSIGNED 2018-05-18 08:51:04,029 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapred.JobConf: Task java-opts do not specify heap size. Setting task attempt jvm max heap size to -Xmx820m 2018-05-18 08:51:04,029 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1526609705906_0273_m_000000_0 TaskAttempt Transitioned from UNASSIGNED to ASSIGNED 2018-05-18 08:51:04,030 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapred.JobConf: Task java-opts do not specify heap size. Setting task attempt jvm max heap size to -Xmx820m 2018-05-18 08:51:04,030 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1526609705906_0273_m_000002_0 TaskAttempt Transitioned from UNASSIGNED to ASSIGNED 2018-05-18 08:51:04,031 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapred.JobConf: Task java-opts do not specify heap size. Setting task attempt jvm max heap size to -Xmx820m 2018-05-18 08:51:04,032 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1526609705906_0273_m_000003_0 TaskAttempt Transitioned from UNASSIGNED to ASSIGNED 2018-05-18 08:51:04,034 INFO [ContainerLauncher #1] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_REMOTE_LAUNCH for container container_1526609705906_0273_01_000003 taskAttempt attempt_1526609705906_0273_m_000000_0 2018-05-18 08:51:04,034 INFO [ContainerLauncher #0] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_REMOTE_LAUNCH for container container_1526609705906_0273_01_000002 taskAttempt attempt_1526609705906_0273_m_000001_0 2018-05-18 08:51:04,034 INFO [ContainerLauncher #2] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_REMOTE_LAUNCH for container container_1526609705906_0273_01_000004 taskAttempt attempt_1526609705906_0273_m_000002_0 2018-05-18 08:51:04,039 INFO [ContainerLauncher #3] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_REMOTE_LAUNCH for container container_1526609705906_0273_01_000005 taskAttempt attempt_1526609705906_0273_m_000003_0 2018-05-18 08:51:04,042 INFO [ContainerLauncher #1] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Launching attempt_1526609705906_0273_m_000000_0 2018-05-18 08:51:04,042 INFO [ContainerLauncher #0] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Launching attempt_1526609705906_0273_m_000001_0 2018-05-18 08:51:04,042 INFO [ContainerLauncher #2] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Launching attempt_1526609705906_0273_m_000002_0 2018-05-18 08:51:04,042 INFO [ContainerLauncher #3] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Launching attempt_1526609705906_0273_m_000003_0 2018-05-18 08:51:04,168 INFO [ContainerLauncher #2] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Shuffle port returned by ContainerManager for attempt_1526609705906_0273_m_000002_0 : 13562 2018-05-18 08:51:04,168 INFO [ContainerLauncher #0] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Shuffle port returned by ContainerManager for attempt_1526609705906_0273_m_000001_0 : 13562 2018-05-18 08:51:04,170 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: TaskAttempt: [attempt_1526609705906_0273_m_000002_0] using containerId: [container_1526609705906_0273_01_000004 on NM: [ip-172-31-1-207.ap-south-1.compute.internal:8041] 2018-05-18 08:51:04,173 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1526609705906_0273_m_000002_0 TaskAttempt Transitioned from ASSIGNED to RUNNING 2018-05-18 08:51:04,174 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: TaskAttempt: [attempt_1526609705906_0273_m_000001_0] using containerId: [container_1526609705906_0273_01_000002 on NM: [ip-172-31-1-207.ap-south-1.compute.internal:8041] 2018-05-18 08:51:04,174 INFO [ContainerLauncher #3] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Shuffle port returned by ContainerManager for attempt_1526609705906_0273_m_000003_0 : 13562 2018-05-18 08:51:04,174 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1526609705906_0273_m_000001_0 TaskAttempt Transitioned from ASSIGNED to RUNNING 2018-05-18 08:51:04,175 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1526609705906_0273_m_000002 Task Transitioned from SCHEDULED to RUNNING 2018-05-18 08:51:04,175 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: TaskAttempt: [attempt_1526609705906_0273_m_000003_0] using containerId: [container_1526609705906_0273_01_000005 on NM: [ip-172-31-1-207.ap-south-1.compute.internal:8041] 2018-05-18 08:51:04,175 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1526609705906_0273_m_000003_0 TaskAttempt Transitioned from ASSIGNED to RUNNING 2018-05-18 08:51:04,176 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1526609705906_0273_m_000001 Task Transitioned from SCHEDULED to RUNNING 2018-05-18 08:51:04,176 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1526609705906_0273_m_000003 Task Transitioned from SCHEDULED to RUNNING 2018-05-18 08:51:04,182 INFO [ContainerLauncher #1] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Shuffle port returned by ContainerManager for attempt_1526609705906_0273_m_000000_0 : 13562 2018-05-18 08:51:04,183 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: TaskAttempt: [attempt_1526609705906_0273_m_000000_0] using containerId: [container_1526609705906_0273_01_000003 on NM: [ip-172-31-1-207.ap-south-1.compute.internal:8041] 2018-05-18 08:51:04,184 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1526609705906_0273_m_000000_0 TaskAttempt Transitioned from ASSIGNED to RUNNING 2018-05-18 08:51:04,185 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1526609705906_0273_m_000000 Task Transitioned from SCHEDULED to RUNNING 2018-05-18 08:51:04,911 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: getResources() for application_1526609705906_0273: ask=6 release= 0 newContainers=0 finishedContainers=0 resourcelimit=<memory:97792, vCores:49> knownNMs=6 2018-05-18 08:51:04,911 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Recalculating schedule, headroom=<memory:97792, vCores:49> 2018-05-18 08:51:04,911 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Reduce slow start threshold not met. completedMapsForReduceSlowstart 4 2018-05-18 08:51:07,918 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Recalculating schedule, headroom=<memory:62806, vCores:48> 2018-05-18 08:51:07,918 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Reduce slow start threshold not met. completedMapsForReduceSlowstart 4 2018-05-18 08:51:08,925 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Received completed container container_1526609705906_0273_01_000004 2018-05-18 08:51:08,926 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Received completed container container_1526609705906_0273_01_000003 2018-05-18 08:51:08,926 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Recalculating schedule, headroom=<memory:64854, vCores:49> 2018-05-18 08:51:08,926 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics report from attempt_1526609705906_0273_m_000002_0: Container killed on request. Exit code is 137 Container exited with a non-zero exit code 137 Killed by external signal 2018-05-18 08:51:08,926 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Reduce slow start threshold not met. completedMapsForReduceSlowstart 4 2018-05-18 08:51:08,926 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: After Scheduling: PendingReds:18 ScheduledMaps:0 ScheduledReds:0 AssignedMaps:2 AssignedReds:0 CompletedMaps:0 CompletedReds:0 ContAlloc:4 ContRel:0 HostLocal:1 RackLocal:3 2018-05-18 08:51:08,932 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1526609705906_0273_m_000002_0 TaskAttempt Transitioned from RUNNING to FAILED 2018-05-18 08:51:08,932 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics report from attempt_1526609705906_0273_m_000000_0: Container killed on request. Exit code is 137 Container exited with a non-zero exit code 137 Killed by external signal 2018-05-18 08:51:08,932 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1526609705906_0273_m_000000_0 TaskAttempt Transitioned from RUNNING to FAILED 2018-05-18 08:51:08,933 INFO [ContainerLauncher #4] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_COMPLETED for container container_1526609705906_0273_01_000004 taskAttempt attempt_1526609705906_0273_m_000002_0 2018-05-18 08:51:08,943 INFO [Thread-52] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: 1 failures on node ip-172-31-1-207.ap-south-1.compute.internal 2018-05-18 08:51:08,944 INFO [ContainerLauncher #5] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_COMPLETED for container container_1526609705906_0273_01_000003 taskAttempt attempt_1526609705906_0273_m_000000_0 2018-05-18 08:51:08,945 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1526609705906_0273_m_000002_1 TaskAttempt Transitioned from NEW to UNASSIGNED 2018-05-18 08:51:08,945 INFO [Thread-52] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: 2 failures on node ip-172-31-1-207.ap-south-1.compute.internal 2018-05-18 08:51:08,945 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1526609705906_0273_m_000000_1 TaskAttempt Transitioned from NEW to UNASSIGNED 2018-05-18 08:51:08,946 INFO [Thread-52] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Added attempt_1526609705906_0273_m_000002_1 to list of failed maps 2018-05-18 08:51:08,946 INFO [Thread-52] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Added attempt_1526609705906_0273_m_000000_1 to list of failed maps</PRE><P>There were no changes made to the cluster before this and I started getting this out of nowhere. Event cluster restart did not fix it and it kept throwing the error for about until 2 hours later. </P><P> </P><P>How do I fix it? I have looked at server logs using diagnostics but have not found anything. any pointers on this would be useful</P><P> </P><P>Here's server logs from diagnostics bar in CM:</P><TABLE><TBODY><TR><TD>May 18, 8:43:53.796 AM</TD><TD>INFO</TD><TD>org.apache.hadoop.mapreduce.Job</TD><TD><PRE>[HiveServer2-Background-Pool: Thread-1391]: The url to track the job: http://ip-172-31-4-192.ap-south-1.compute.internal:8088/proxy/application_1526609705906_0246/</PRE></TD></TR><TR><TD>May 18, 8:43:53.797 AM</TD><TD>INFO</TD><TD>org.apache.hadoop.hive.ql.exec.Task</TD><TD><PRE>[HiveServer2-Background-Pool: Thread-1391]: Starting Job = job_1526609705906_0246, Tracking URL = http://ip-172-31-4-192.ap-south-1.compute.internal:8088/proxy/application_1526609705906_0246/</PRE></TD></TR><TR><TD>May 18, 8:43:53.797 AM</TD><TD>INFO</TD><TD>org.apache.hadoop.hive.ql.exec.Task</TD><TD><PRE>[HiveServer2-Background-Pool: Thread-1391]: Kill Command = /opt/cloudera/parcels/CDH-5.10.1-1.cdh5.10.1.p0.10/lib/hadoop/bin/hadoop job -kill job_1526609705906_0246</PRE></TD></TR><TR><TD>May 18, 8:44:00.027 AM</TD><TD>INFO</TD><TD>org.apache.hadoop.hive.ql.exec.Task</TD><TD><PRE>[HiveServer2-Background-Pool: Thread-1391]: Hadoop job information for Stage-2: number of mappers: 4; number of reducers: 1</PRE></TD></TR><TR><TD>May 18, 8:44:00.063 AM</TD><TD>WARN</TD><TD>mapreduce.Counters</TD><TD><PRE>[HiveServer2-Background-Pool: Thread-1391]: Group org.apache.hadoop.mapred.Task$Counter is deprecated. Use org.apache.hadoop.mapreduce.TaskCounter instead</PRE></TD></TR><TR><TD>May 18, 8:44:00.063 AM</TD><TD>INFO</TD><TD>org.apache.hadoop.hive.ql.exec.Task</TD><TD><PRE>[HiveServer2-Background-Pool: Thread-1391]: 2018-05-18 08:44:00,063 Stage-2 map = 0%, reduce = 0%</PRE></TD></TR><TR><TD>May 18, 8:44:07.404 AM</TD><TD>WARN</TD><TD>mapreduce.Counters</TD><TD><PRE>[HiveServer2-Background-Pool: Thread-1391]: Group org.apache.hadoop.mapred.Task$Counter is deprecated. Use org.apache.hadoop.mapreduce.TaskCounter instead</PRE></TD></TR><TR><TD>May 18, 8:44:07.406 AM</TD><TD>ERROR</TD><TD>org.apache.hadoop.hive.ql.exec.Task</TD><TD><PRE>[HiveServer2-Background-Pool: Thread-1391]: Ended Job = job_1526609705906_0246 with errors</PRE></TD></TR><TR><TD>May 18, 8:44:07.407 AM</TD><TD>ERROR</TD><TD>org.apache.hadoop.hive.ql.exec.Task</TD><TD><PRE>[Thread-1631]: Error during job, obtaining debugging information...</PRE></TD></TR><TR><TD>May 18, 8:44:07.416 AM</TD><TD>INFO</TD><TD>org.apache.hadoop.yarn.client.api.impl.YarnClientImpl</TD><TD><PRE>[HiveServer2-Background-Pool: Thread-1391]: Killed application application_1526609705906_0246</PRE></TD></TR><TR><TD>May 18, 8:44:07.418 AM</TD><TD>ERROR</TD><TD>org.apache.hadoop.hive.ql.Driver</TD><TD><PRE>[HiveServer2-Background-Pool: Thread-1391]: FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask</PRE></TD></TR><TR><TD>May 18, 8:44:07.418 AM</TD><TD>INFO</TD><TD>org.apache.hadoop.hive.ql.log.PerfLogger</TD><TD><PRE>[HiveServer2-Background-Pool: Thread-1391]: </PERFLOG method=Driver.execute start=1526613040763 end=1526613247418 duration=206655 from=org.apache.hadoop.hive.ql.Driver></PRE></TD></TR><TR><TD>May 18, 8:44:07.418 AM</TD><TD>INFO</TD><TD>org.apache.hadoop.hive.ql.Driver</TD><TD><PRE>[HiveServer2-Background-Pool: Thread-1391]: MapReduce Jobs Launched: </PRE></TD></TR><TR><TD>May 18, 8:44:07.418 AM</TD><TD>INFO</TD><TD>org.apache.hadoop.hive.ql.Driver</TD><TD><PRE>[HiveServer2-Background-Pool: Thread-1391]: Stage-Stage-1: Map: 1 Reduce: 1 Cumulative CPU: 4.09 sec HDFS Read: 238035 HDFS Write: 1380 SUCCESS</PRE></TD></TR><TR><TD>May 18, 8:44:07.418 AM</TD><TD>INFO</TD><TD>org.apache.hadoop.hive.ql.Driver</TD><TD><PRE>[HiveServer2-Background-Pool: Thread-1391]: Stage-Stage-4: Map: 1 Reduce: 1 Cumulative CPU: 4.07 sec HDFS Read: 238035 HDFS Write: 1291 SUCCESS</PRE></TD></TR><TR><TD>May 18, 8:44:07.419 AM</TD><TD>INFO</TD><TD>org.apache.hadoop.hive.ql.Driver</TD><TD><PRE>[HiveServer2-Background-Pool: Thread-1391]: Stage-Stage-5: Map: 1 Reduce: 1 Cumulative CPU: 3.82 sec HDFS Read: 238003 HDFS Write: 1209 SUCCESS</PRE></TD></TR><TR><TD>May 18, 8:44:07.419 AM</TD><TD>INFO</TD><TD>org.apache.hadoop.hive.ql.Driver</TD><TD><PRE>[HiveServer2-Background-Pool: Thread-1391]: Stage-Stage-6: Map: 1 Reduce: 1 Cumulative CPU: 3.64 sec HDFS Read: 237973 HDFS Write: 1293 SUCCESS</PRE></TD></TR><TR><TD>May 18, 8:44:07.419 AM</TD><TD>WARN</TD><TD>mapreduce.Counters</TD><TD><PRE>[HiveServer2-Background-Pool: Thread-1391]: Group FileSystemCounters is deprecated. Use org.apache.hadoop.mapreduce.FileSystemCounter instead</PRE></TD></TR><TR><TD>May 18, 8:44:07.419 AM</TD><TD>INFO</TD><TD>org.apache.hadoop.hive.ql.Driver</TD><TD><PRE>[HiveServer2-Background-Pool: Thread-1391]: Stage-Stage-2: Map: 4 Reduce: 1 HDFS Read: 0 HDFS Write: 0 FAIL</PRE></TD></TR><TR><TD>May 18, 8:44:07.419 AM</TD><TD>INFO</TD><TD>org.apache.hadoop.hive.ql.Driver</TD><TD><PRE>[HiveServer2-Background-Pool: Thread-1391]: Total MapReduce CPU Time Spent: 15 seconds 620 msec</PRE></TD></TR><TR><TD>May 18, 8:44:07.419 AM</TD><TD>INFO</TD><TD>org.apache.hadoop.hive.ql.Driver</TD><TD><PRE>[HiveServer2-Background-Pool: Thread-1391]: Completed executing command(queryId=hive_20180518084040_4308c812-08c3-41aa-b5ea-04fad833b98f); Time taken: 206.655 seconds</PRE></TD></TR><TR><TD>May 18, 8:44:07.419 AM</TD><TD>INFO</TD><TD>org.apache.hadoop.hive.ql.log.PerfLogger</TD><TD><PRE>[HiveServer2-Background-Pool: Thread-1391]: <PERFLOG method=releaseLocks from=org.apache.hadoop.hive.ql.Driver></PRE></TD></TR><TR><TD>May 18, 8:44:07.425 AM</TD><TD>INFO</TD><TD>org.apache.hadoop.hive.ql.log.PerfLogger</TD><TD><PRE>[HiveServer2-Background-Pool: Thread-1391]: </PERFLOG method=releaseLocks start=1526613247419 end=1526613247425 duration=6 from=org.apache.hadoop.hive.ql.Driver></PRE></TD></TR><TR><TD>May 18, 8:44:07.431 AM</TD><TD>ERROR</TD><TD>org.apache.hive.service.cli.operation.Operation</TD><TD><PRE>[HiveServer2-Background-Pool: Thread-1391]: Error running hive query: org.apache.hive.service.cli.HiveSQLException: Error while processing statement: FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask at org.apache.hive.service.cli.operation.Operation.toSQLException(Operation.java:400) at org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:239) at org.apache.hive.service.cli.operation.SQLOperation.access$300(SQLOperation.java:88) at org.apache.hive.service.cli.operation.SQLOperation$3$1.run(SQLOperation.java:293) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920) at org.apache.hive.service.cli.operation.SQLOperation$3.run(SQLOperation.java:306) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745)</PRE></TD></TR><TR><TD>May 18, 8:44:07.433 AM</TD><TD>INFO</TD><TD>org.apache.hive.service.cli.thrift.ThriftCLIService</TD><TD><PRE>[HiveServer2-Handler-Pool: Thread-575]: Session disconnected without closing properly, close it now</PRE></TD></TR></TBODY></TABLE> Fri, 16 Sep 2022 13:14:18 GMT sim6 2022-09-16T13:14:18Z