My cluster memory is 147GB and I get this error when the server has not used it's entire memory.   I can see there is memory free and yet my jobs get killed with this error.  There is no error in logs and I don't get any error using dmesg command or in /var/log/messages     Also, it happens randomly and on any of the nodes. Please suggest. Been trying to get in touch with Cloudera sales support but no luck and it's urgent. ... View more

All hive jobs started to fail with this error:     2018-05-18 08:50:58,682 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Created MRAppMaster for application appattempt_1526609705906_0273_000001 2018-05-18 08:50:59,959 WARN [main] org.apache.hadoop.util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2018-05-18 08:51:00,073 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: OutputCommitter set in config org.apache.hadoop.hive.ql.io.HiveFileFormatUtils$NullOutputCommitter 2018-05-18 08:51:00,075 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: OutputCommitter is org.apache.hadoop.hive.ql.io.HiveFileFormatUtils$NullOutputCommitter 2018-05-18 08:51:00,123 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.jobhistory.EventType for class org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler 2018-05-18 08:51:00,126 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventType for class org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler 2018-05-18 08:51:00,132 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncher$EventType for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$ContainerLauncherRouter 2018-05-18 08:51:00,169 INFO [main] org.apache.hadoop.mapreduce.v2.jobhistory.JobHistoryUtils: Default file system [hdfs://ip-172-31-4-192.ap-south-1.compute.internal:8020] 2018-05-18 08:51:00,194 INFO [main] org.apache.hadoop.mapreduce.v2.jobhistory.JobHistoryUtils: Default file system [hdfs://ip-172-31-4-192.ap-south-1.compute.internal:8020] 2018-05-18 08:51:00,212 INFO [main] org.apache.hadoop.mapreduce.v2.jobhistory.JobHistoryUtils: Default file system [hdfs://ip-172-31-4-192.ap-south-1.compute.internal:8020] 2018-05-18 08:51:00,223 INFO [main] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Emitting job history data to the timeline server is not enabled 2018-05-18 08:51:00,277 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.job.event.JobFinishEvent$Type for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobFinishEventHandler 2018-05-18 08:51:00,537 INFO [main] org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from hadoop-metrics2.properties 2018-05-18 08:51:00,605 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot period at 10 second(s). 2018-05-18 08:51:00,605 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: MRAppMaster metrics system started 2018-05-18 08:51:00,625 INFO [main] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Adding job token for job_1526609705906_0273 to jobTokenSecretManager 2018-05-18 08:51:00,825 INFO [main] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Not uberizing job_1526609705906_0273 because: not enabled; too many reduces; too much input; 2018-05-18 08:51:00,857 INFO [main] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Input size for job job_1526609705906_0273 = 1155935596. Number of splits = 4 2018-05-18 08:51:00,861 INFO [main] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Number of reduces for job job_1526609705906_0273 = 18 2018-05-18 08:51:00,861 INFO [main] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: job_1526609705906_0273Job Transitioned from NEW to INITED 2018-05-18 08:51:00,863 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: MRAppMaster launching normal, non-uberized, multi-container job job_1526609705906_0273. 2018-05-18 08:51:00,901 INFO [main] org.apache.hadoop.ipc.CallQueueManager: Using callQueue: class java.util.concurrent.LinkedBlockingQueue queueCapacity: 100 2018-05-18 08:51:00,912 INFO [Socket Reader #1 for port 42741] org.apache.hadoop.ipc.Server: Starting Socket Reader #1 for port 42741 2018-05-18 08:51:00,936 INFO [main] org.apache.hadoop.yarn.factories.impl.pb.RpcServerFactoryPBImpl: Adding protocol org.apache.hadoop.mapreduce.v2.api.MRClientProtocolPB to the server 2018-05-18 08:51:00,936 INFO [IPC Server Responder] org.apache.hadoop.ipc.Server: IPC Server Responder: starting 2018-05-18 08:51:00,937 INFO [IPC Server listener on 42741] org.apache.hadoop.ipc.Server: IPC Server listener on 42741: starting 2018-05-18 08:51:00,938 INFO [main] org.apache.hadoop.mapreduce.v2.app.client.MRClientService: Instantiated MRClientService at ip-172-31-13-118.ap-south-1.compute.internal/172.31.13.118:42741 2018-05-18 08:51:01,021 INFO [main] org.mortbay.log: Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog 2018-05-18 08:51:01,031 INFO [main] org.apache.hadoop.security.authentication.server.AuthenticationFilter: Unable to initialize FileSignerSecretProvider, falling back to use random secrets. 2018-05-18 08:51:01,036 INFO [main] org.apache.hadoop.http.HttpRequestLog: Http request log for http.requests.mapreduce is not defined 2018-05-18 08:51:01,044 INFO [main] org.apache.hadoop.http.HttpServer2: Added global filter 'safety' (class=org.apache.hadoop.http.HttpServer2$QuotingInputFilter) 2018-05-18 08:51:01,050 INFO [main] org.apache.hadoop.http.HttpServer2: Added filter AM_PROXY_FILTER (class=org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter) to context mapreduce 2018-05-18 08:51:01,050 INFO [main] org.apache.hadoop.http.HttpServer2: Added filter AM_PROXY_FILTER (class=org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter) to context static 2018-05-18 08:51:01,053 INFO [main] org.apache.hadoop.http.HttpServer2: adding path spec: /mapreduce/* 2018-05-18 08:51:01,053 INFO [main] org.apache.hadoop.http.HttpServer2: adding path spec: /ws/* 2018-05-18 08:51:01,061 INFO [main] org.apache.hadoop.http.HttpServer2: Jetty bound to port 36178 2018-05-18 08:51:01,061 INFO [main] org.mortbay.log: jetty-6.1.26.cloudera.4 2018-05-18 08:51:01,088 INFO [main] org.mortbay.log: Extract jar:file:/opt/cloudera/parcels/CDH-5.10.1-1.cdh5.10.1.p0.10/jars/hadoop-yarn-common-2.6.0-cdh5.10.1.jar!/webapps/mapreduce to /tmp/Jetty_0_0_0_0_36178_mapreduce____davbey/webapp 2018-05-18 08:51:01,355 INFO [main] org.mortbay.log: Started HttpServer2$SelectChannelConnectorWithSafeStartup@0.0.0.0:36178 2018-05-18 08:51:01,355 INFO [main] org.apache.hadoop.yarn.webapp.WebApps: Web app /mapreduce started at 36178 2018-05-18 08:51:01,697 INFO [main] org.apache.hadoop.yarn.webapp.WebApps: Registered webapp guice modules 2018-05-18 08:51:01,701 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.speculate.DefaultSpeculator: JOB_CREATE job_1526609705906_0273 2018-05-18 08:51:01,704 INFO [main] org.apache.hadoop.ipc.CallQueueManager: Using callQueue: class java.util.concurrent.LinkedBlockingQueue queueCapacity: 3000 2018-05-18 08:51:01,704 INFO [Socket Reader #1 for port 34196] org.apache.hadoop.ipc.Server: Starting Socket Reader #1 for port 34196 2018-05-18 08:51:01,712 INFO [IPC Server Responder] org.apache.hadoop.ipc.Server: IPC Server Responder: starting 2018-05-18 08:51:01,712 INFO [IPC Server listener on 34196] org.apache.hadoop.ipc.Server: IPC Server listener on 34196: starting 2018-05-18 08:51:01,732 INFO [main] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: nodeBlacklistingEnabled:true 2018-05-18 08:51:01,732 INFO [main] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: maxTaskFailuresPerNode is 3 2018-05-18 08:51:01,732 INFO [main] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: blacklistDisablePercent is 33 2018-05-18 08:51:01,795 INFO [main] org.apache.hadoop.yarn.client.RMProxy: Connecting to ResourceManager at ip-172-31-4-192.ap-south-1.compute.internal/172.31.4.192:8030 2018-05-18 08:51:01,852 INFO [main] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: maxContainerCapability: <memory:8192, vCores:6> 2018-05-18 08:51:01,852 INFO [main] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: queue: root.users.hue 2018-05-18 08:51:01,855 INFO [main] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Upper limit on the thread pool size is 500 2018-05-18 08:51:01,855 INFO [main] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: The thread pool initial size is 10 2018-05-18 08:51:01,864 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: job_1526609705906_0273Job Transitioned from INITED to SETUP 2018-05-18 08:51:01,865 INFO [CommitterEvent Processor #0] org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler: Processing the event EventType: JOB_SETUP 2018-05-18 08:51:01,867 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: job_1526609705906_0273Job Transitioned from SETUP to RUNNING 2018-05-18 08:51:01,936 INFO [eventHandlingThread] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Event Writer setup for JobId: job_1526609705906_0273, File: hdfs://ip-172-31-4-192.ap-south-1.compute.internal:8020/user/hue/.staging/job_1526609705906_0273/job_1526609705906_0273_1.jhist 2018-05-18 08:51:01,947 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1526609705906_0273_m_000000 Task Transitioned from NEW to SCHEDULED 2018-05-18 08:51:01,983 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1526609705906_0273_m_000001 Task Transitioned from NEW to SCHEDULED 2018-05-18 08:51:02,010 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1526609705906_0273_m_000002 Task Transitioned from NEW to SCHEDULED 2018-05-18 08:51:02,037 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1526609705906_0273_m_000003 Task Transitioned from NEW to SCHEDULED 2018-05-18 08:51:02,038 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1526609705906_0273_r_000000 Task Transitioned from NEW to SCHEDULED 2018-05-18 08:51:02,038 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1526609705906_0273_r_000001 Task Transitioned from NEW to SCHEDULED 2018-05-18 08:51:02,039 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1526609705906_0273_r_000002 Task Transitioned from NEW to SCHEDULED 2018-05-18 08:51:02,039 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1526609705906_0273_r_000003 Task Transitioned from NEW to SCHEDULED 2018-05-18 08:51:02,040 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1526609705906_0273_r_000004 Task Transitioned from NEW to SCHEDULED 2018-05-18 08:51:02,048 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1526609705906_0273_r_000015 Task Transitioned from NEW to SCHEDULED 2018-05-18 08:51:02,048 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1526609705906_0273_r_000016 Task Transitioned from NEW to SCHEDULED 2018-05-18 08:51:02,049 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1526609705906_0273_r_000017 Task Transitioned from NEW to SCHEDULED 2018-05-18 08:51:02,051 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1526609705906_0273_m_000000_0 TaskAttempt Transitioned from NEW to UNASSIGNED 2018-05-18 08:51:02,051 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1526609705906_0273_m_000001_0 TaskAttempt Transitioned from NEW to UNASSIGNED 2018-05-18 08:51:02,051 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1526609705906_0273_m_000002_0 TaskAttempt Transitioned from NEW to UNASSIGNED 2018-05-18 08:51:02,051 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1526609705906_0273_m_000003_0 TaskAttempt Transitioned from NEW to UNASSIGNED 2018-05-18 08:51:02,051 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1526609705906_0273_r_000000_0 TaskAttempt Transitioned from NEW to UNASSIGNED 2018-05-18 08:51:02,054 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1526609705906_0273_r_000014_0 TaskAttempt Transitioned from NEW to UNASSIGNED 2018-05-18 08:51:02,054 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1526609705906_0273_r_000016_0 TaskAttempt Transitioned from NEW to UNASSIGNED 2018-05-18 08:51:02,054 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1526609705906_0273_r_000017_0 TaskAttempt Transitioned from NEW to UNASSIGNED 2018-05-18 08:51:02,056 INFO [Thread-52] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: mapResourceRequest:<memory:1024, vCores:1> 2018-05-18 08:51:02,073 INFO [Thread-52] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: reduceResourceRequest:<memory:1024, vCores:1> 2018-05-18 08:51:02,448 INFO [eventHandlingThread] org.apache.hadoop.mapreduce.v2.jobhistory.JobHistoryUtils: Default file system [hdfs://ip-172-31-4-192.ap-south-1.compute.internal:8020] 2018-05-18 08:51:02,854 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Before Scheduling: PendingReds:18 ScheduledMaps:4 ScheduledReds:0 AssignedMaps:0 AssignedReds:0 CompletedMaps:0 CompletedReds:0 ContAlloc:0 ContRel:0 HostLocal:0 RackLocal:0 2018-05-18 08:51:02,892 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: getResources() for application_1526609705906_0273: ask=6 release= 0 newContainers=0 finishedContainers=0 resourcelimit=<memory:101888, vCores:54> knownNMs=6 2018-05-18 08:51:02,892 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Recalculating schedule, headroom=<memory:101888, vCores:54> 2018-05-18 08:51:02,892 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Reduce slow start threshold not met. completedMapsForReduceSlowstart 4 2018-05-18 08:51:02,996 INFO [IPC Server handler 0 on 42741] org.apache.hadoop.mapreduce.v2.app.client.MRClientService: Getting task report for MAP job_1526609705906_0273. Report-size will be 4 2018-05-18 08:51:03,051 INFO [IPC Server handler 0 on 42741] org.apache.hadoop.mapreduce.v2.app.client.MRClientService: Getting task report for REDUCE job_1526609705906_0273. Report-size will be 18 2018-05-18 08:51:03,904 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Got allocated containers 4 2018-05-18 08:51:03,906 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned container container_1526609705906_0273_01_000002 to attempt_1526609705906_0273_m_000001_0 2018-05-18 08:51:03,907 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned container container_1526609705906_0273_01_000003 to attempt_1526609705906_0273_m_000000_0 2018-05-18 08:51:03,908 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned container container_1526609705906_0273_01_000004 to attempt_1526609705906_0273_m_000002_0 2018-05-18 08:51:03,908 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned container container_1526609705906_0273_01_000005 to attempt_1526609705906_0273_m_000003_0 2018-05-18 08:51:03,908 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Recalculating schedule, headroom=<memory:97792, vCores:50> 2018-05-18 08:51:03,908 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Reduce slow start threshold not met. completedMapsForReduceSlowstart 4 2018-05-18 08:51:03,908 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: After Scheduling: PendingReds:18 ScheduledMaps:0 ScheduledReds:0 AssignedMaps:4 AssignedReds:0 CompletedMaps:0 CompletedReds:0 ContAlloc:4 ContRel:0 HostLocal:1 RackLocal:3 2018-05-18 08:51:03,974 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: The job-jar file on the remote FS is hdfs://ip-172-31-4-192.ap-south-1.compute.internal:8020/user/hue/.staging/job_1526609705906_0273/job.jar 2018-05-18 08:51:03,976 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: The job-conf file on the remote FS is /user/hue/.staging/job_1526609705906_0273/job.xml 2018-05-18 08:51:03,985 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Adding #0 tokens and #1 secret keys for NM use for launching container 2018-05-18 08:51:03,985 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Size of containertokens_dob is 1 2018-05-18 08:51:03,985 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Putting shuffle token in serviceData 2018-05-18 08:51:04,022 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapred.JobConf: Task java-opts do not specify heap size. Setting task attempt jvm max heap size to -Xmx820m 2018-05-18 08:51:04,026 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1526609705906_0273_m_000001_0 TaskAttempt Transitioned from UNASSIGNED to ASSIGNED 2018-05-18 08:51:04,029 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapred.JobConf: Task java-opts do not specify heap size. Setting task attempt jvm max heap size to -Xmx820m 2018-05-18 08:51:04,029 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1526609705906_0273_m_000000_0 TaskAttempt Transitioned from UNASSIGNED to ASSIGNED 2018-05-18 08:51:04,030 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapred.JobConf: Task java-opts do not specify heap size. Setting task attempt jvm max heap size to -Xmx820m 2018-05-18 08:51:04,030 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1526609705906_0273_m_000002_0 TaskAttempt Transitioned from UNASSIGNED to ASSIGNED 2018-05-18 08:51:04,031 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapred.JobConf: Task java-opts do not specify heap size. Setting task attempt jvm max heap size to -Xmx820m 2018-05-18 08:51:04,032 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1526609705906_0273_m_000003_0 TaskAttempt Transitioned from UNASSIGNED to ASSIGNED 2018-05-18 08:51:04,034 INFO [ContainerLauncher #1] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_REMOTE_LAUNCH for container container_1526609705906_0273_01_000003 taskAttempt attempt_1526609705906_0273_m_000000_0 2018-05-18 08:51:04,034 INFO [ContainerLauncher #0] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_REMOTE_LAUNCH for container container_1526609705906_0273_01_000002 taskAttempt attempt_1526609705906_0273_m_000001_0 2018-05-18 08:51:04,034 INFO [ContainerLauncher #2] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_REMOTE_LAUNCH for container container_1526609705906_0273_01_000004 taskAttempt attempt_1526609705906_0273_m_000002_0 2018-05-18 08:51:04,039 INFO [ContainerLauncher #3] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_REMOTE_LAUNCH for container container_1526609705906_0273_01_000005 taskAttempt attempt_1526609705906_0273_m_000003_0 2018-05-18 08:51:04,042 INFO [ContainerLauncher #1] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Launching attempt_1526609705906_0273_m_000000_0 2018-05-18 08:51:04,042 INFO [ContainerLauncher #0] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Launching attempt_1526609705906_0273_m_000001_0 2018-05-18 08:51:04,042 INFO [ContainerLauncher #2] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Launching attempt_1526609705906_0273_m_000002_0 2018-05-18 08:51:04,042 INFO [ContainerLauncher #3] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Launching attempt_1526609705906_0273_m_000003_0 2018-05-18 08:51:04,168 INFO [ContainerLauncher #2] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Shuffle port returned by ContainerManager for attempt_1526609705906_0273_m_000002_0 : 13562 2018-05-18 08:51:04,168 INFO [ContainerLauncher #0] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Shuffle port returned by ContainerManager for attempt_1526609705906_0273_m_000001_0 : 13562 2018-05-18 08:51:04,170 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: TaskAttempt: [attempt_1526609705906_0273_m_000002_0] using containerId: [container_1526609705906_0273_01_000004 on NM: [ip-172-31-1-207.ap-south-1.compute.internal:8041] 2018-05-18 08:51:04,173 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1526609705906_0273_m_000002_0 TaskAttempt Transitioned from ASSIGNED to RUNNING 2018-05-18 08:51:04,174 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: TaskAttempt: [attempt_1526609705906_0273_m_000001_0] using containerId: [container_1526609705906_0273_01_000002 on NM: [ip-172-31-1-207.ap-south-1.compute.internal:8041] 2018-05-18 08:51:04,174 INFO [ContainerLauncher #3] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Shuffle port returned by ContainerManager for attempt_1526609705906_0273_m_000003_0 : 13562 2018-05-18 08:51:04,174 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1526609705906_0273_m_000001_0 TaskAttempt Transitioned from ASSIGNED to RUNNING 2018-05-18 08:51:04,175 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1526609705906_0273_m_000002 Task Transitioned from SCHEDULED to RUNNING 2018-05-18 08:51:04,175 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: TaskAttempt: [attempt_1526609705906_0273_m_000003_0] using containerId: [container_1526609705906_0273_01_000005 on NM: [ip-172-31-1-207.ap-south-1.compute.internal:8041] 2018-05-18 08:51:04,175 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1526609705906_0273_m_000003_0 TaskAttempt Transitioned from ASSIGNED to RUNNING 2018-05-18 08:51:04,176 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1526609705906_0273_m_000001 Task Transitioned from SCHEDULED to RUNNING 2018-05-18 08:51:04,176 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1526609705906_0273_m_000003 Task Transitioned from SCHEDULED to RUNNING 2018-05-18 08:51:04,182 INFO [ContainerLauncher #1] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Shuffle port returned by ContainerManager for attempt_1526609705906_0273_m_000000_0 : 13562 2018-05-18 08:51:04,183 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: TaskAttempt: [attempt_1526609705906_0273_m_000000_0] using containerId: [container_1526609705906_0273_01_000003 on NM: [ip-172-31-1-207.ap-south-1.compute.internal:8041] 2018-05-18 08:51:04,184 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1526609705906_0273_m_000000_0 TaskAttempt Transitioned from ASSIGNED to RUNNING 2018-05-18 08:51:04,185 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1526609705906_0273_m_000000 Task Transitioned from SCHEDULED to RUNNING 2018-05-18 08:51:04,911 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: getResources() for application_1526609705906_0273: ask=6 release= 0 newContainers=0 finishedContainers=0 resourcelimit=<memory:97792, vCores:49> knownNMs=6 2018-05-18 08:51:04,911 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Recalculating schedule, headroom=<memory:97792, vCores:49> 2018-05-18 08:51:04,911 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Reduce slow start threshold not met. completedMapsForReduceSlowstart 4 2018-05-18 08:51:07,918 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Recalculating schedule, headroom=<memory:62806, vCores:48> 2018-05-18 08:51:07,918 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Reduce slow start threshold not met. completedMapsForReduceSlowstart 4 2018-05-18 08:51:08,925 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Received completed container container_1526609705906_0273_01_000004 2018-05-18 08:51:08,926 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Received completed container container_1526609705906_0273_01_000003 2018-05-18 08:51:08,926 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Recalculating schedule, headroom=<memory:64854, vCores:49> 2018-05-18 08:51:08,926 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics report from attempt_1526609705906_0273_m_000002_0: Container killed on request. Exit code is 137 Container exited with a non-zero exit code 137 Killed by external signal 2018-05-18 08:51:08,926 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Reduce slow start threshold not met. completedMapsForReduceSlowstart 4 2018-05-18 08:51:08,926 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: After Scheduling: PendingReds:18 ScheduledMaps:0 ScheduledReds:0 AssignedMaps:2 AssignedReds:0 CompletedMaps:0 CompletedReds:0 ContAlloc:4 ContRel:0 HostLocal:1 RackLocal:3 2018-05-18 08:51:08,932 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1526609705906_0273_m_000002_0 TaskAttempt Transitioned from RUNNING to FAILED 2018-05-18 08:51:08,932 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics report from attempt_1526609705906_0273_m_000000_0: Container killed on request. Exit code is 137 Container exited with a non-zero exit code 137 Killed by external signal 2018-05-18 08:51:08,932 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1526609705906_0273_m_000000_0 TaskAttempt Transitioned from RUNNING to FAILED 2018-05-18 08:51:08,933 INFO [ContainerLauncher #4] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_COMPLETED for container container_1526609705906_0273_01_000004 taskAttempt attempt_1526609705906_0273_m_000002_0 2018-05-18 08:51:08,943 INFO [Thread-52] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: 1 failures on node ip-172-31-1-207.ap-south-1.compute.internal 2018-05-18 08:51:08,944 INFO [ContainerLauncher #5] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_COMPLETED for container container_1526609705906_0273_01_000003 taskAttempt attempt_1526609705906_0273_m_000000_0 2018-05-18 08:51:08,945 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1526609705906_0273_m_000002_1 TaskAttempt Transitioned from NEW to UNASSIGNED 2018-05-18 08:51:08,945 INFO [Thread-52] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: 2 failures on node ip-172-31-1-207.ap-south-1.compute.internal 2018-05-18 08:51:08,945 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1526609705906_0273_m_000000_1 TaskAttempt Transitioned from NEW to UNASSIGNED 2018-05-18 08:51:08,946 INFO [Thread-52] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Added attempt_1526609705906_0273_m_000002_1 to list of failed maps 2018-05-18 08:51:08,946 INFO [Thread-52] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Added attempt_1526609705906_0273_m_000000_1 to list of failed maps There were no changes made to the cluster before this and I started getting this out of nowhere. Event cluster restart did not fix it and it kept throwing the error for about until 2 hours later.    How do I fix it? I have looked at server logs using diagnostics but have not found anything. any pointers on this would be useful   Here's server logs from diagnostics bar in CM: May 18, 8:43:53.796 AM INFO org.apache.hadoop.mapreduce.Job [HiveServer2-Background-Pool: Thread-1391]: The url to track the job: http://ip-172-31-4-192.ap-south-1.compute.internal:8088/proxy/application_1526609705906_0246/ May 18, 8:43:53.797 AM INFO org.apache.hadoop.hive.ql.exec.Task [HiveServer2-Background-Pool: Thread-1391]: Starting Job = job_1526609705906_0246, Tracking URL = http://ip-172-31-4-192.ap-south-1.compute.internal:8088/proxy/application_1526609705906_0246/ May 18, 8:43:53.797 AM INFO org.apache.hadoop.hive.ql.exec.Task [HiveServer2-Background-Pool: Thread-1391]: Kill Command = /opt/cloudera/parcels/CDH-5.10.1-1.cdh5.10.1.p0.10/lib/hadoop/bin/hadoop job -kill job_1526609705906_0246 May 18, 8:44:00.027 AM INFO org.apache.hadoop.hive.ql.exec.Task [HiveServer2-Background-Pool: Thread-1391]: Hadoop job information for Stage-2: number of mappers: 4; number of reducers: 1 May 18, 8:44:00.063 AM WARN mapreduce.Counters [HiveServer2-Background-Pool: Thread-1391]: Group org.apache.hadoop.mapred.Task$Counter is deprecated. Use org.apache.hadoop.mapreduce.TaskCounter instead May 18, 8:44:00.063 AM INFO org.apache.hadoop.hive.ql.exec.Task [HiveServer2-Background-Pool: Thread-1391]: 2018-05-18 08:44:00,063 Stage-2 map = 0%, reduce = 0% May 18, 8:44:07.404 AM WARN mapreduce.Counters [HiveServer2-Background-Pool: Thread-1391]: Group org.apache.hadoop.mapred.Task$Counter is deprecated. Use org.apache.hadoop.mapreduce.TaskCounter instead May 18, 8:44:07.406 AM ERROR org.apache.hadoop.hive.ql.exec.Task [HiveServer2-Background-Pool: Thread-1391]: Ended Job = job_1526609705906_0246 with errors May 18, 8:44:07.407 AM ERROR org.apache.hadoop.hive.ql.exec.Task [Thread-1631]: Error during job, obtaining debugging information... May 18, 8:44:07.416 AM INFO org.apache.hadoop.yarn.client.api.impl.YarnClientImpl [HiveServer2-Background-Pool: Thread-1391]: Killed application application_1526609705906_0246 May 18, 8:44:07.418 AM ERROR org.apache.hadoop.hive.ql.Driver [HiveServer2-Background-Pool: Thread-1391]: FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask May 18, 8:44:07.418 AM INFO org.apache.hadoop.hive.ql.log.PerfLogger [HiveServer2-Background-Pool: Thread-1391]: </PERFLOG method=Driver.execute start=1526613040763 end=1526613247418 duration=206655 from=org.apache.hadoop.hive.ql.Driver> May 18, 8:44:07.418 AM INFO org.apache.hadoop.hive.ql.Driver [HiveServer2-Background-Pool: Thread-1391]: MapReduce Jobs Launched: May 18, 8:44:07.418 AM INFO org.apache.hadoop.hive.ql.Driver [HiveServer2-Background-Pool: Thread-1391]: Stage-Stage-1: Map: 1 Reduce: 1 Cumulative CPU: 4.09 sec HDFS Read: 238035 HDFS Write: 1380 SUCCESS May 18, 8:44:07.418 AM INFO org.apache.hadoop.hive.ql.Driver [HiveServer2-Background-Pool: Thread-1391]: Stage-Stage-4: Map: 1 Reduce: 1 Cumulative CPU: 4.07 sec HDFS Read: 238035 HDFS Write: 1291 SUCCESS May 18, 8:44:07.419 AM INFO org.apache.hadoop.hive.ql.Driver [HiveServer2-Background-Pool: Thread-1391]: Stage-Stage-5: Map: 1 Reduce: 1 Cumulative CPU: 3.82 sec HDFS Read: 238003 HDFS Write: 1209 SUCCESS May 18, 8:44:07.419 AM INFO org.apache.hadoop.hive.ql.Driver [HiveServer2-Background-Pool: Thread-1391]: Stage-Stage-6: Map: 1 Reduce: 1 Cumulative CPU: 3.64 sec HDFS Read: 237973 HDFS Write: 1293 SUCCESS May 18, 8:44:07.419 AM WARN mapreduce.Counters [HiveServer2-Background-Pool: Thread-1391]: Group FileSystemCounters is deprecated. Use org.apache.hadoop.mapreduce.FileSystemCounter instead May 18, 8:44:07.419 AM INFO org.apache.hadoop.hive.ql.Driver [HiveServer2-Background-Pool: Thread-1391]: Stage-Stage-2: Map: 4 Reduce: 1 HDFS Read: 0 HDFS Write: 0 FAIL May 18, 8:44:07.419 AM INFO org.apache.hadoop.hive.ql.Driver [HiveServer2-Background-Pool: Thread-1391]: Total MapReduce CPU Time Spent: 15 seconds 620 msec May 18, 8:44:07.419 AM INFO org.apache.hadoop.hive.ql.Driver [HiveServer2-Background-Pool: Thread-1391]: Completed executing command(queryId=hive_20180518084040_4308c812-08c3-41aa-b5ea-04fad833b98f); Time taken: 206.655 seconds May 18, 8:44:07.419 AM INFO org.apache.hadoop.hive.ql.log.PerfLogger [HiveServer2-Background-Pool: Thread-1391]: <PERFLOG method=releaseLocks from=org.apache.hadoop.hive.ql.Driver> May 18, 8:44:07.425 AM INFO org.apache.hadoop.hive.ql.log.PerfLogger [HiveServer2-Background-Pool: Thread-1391]: </PERFLOG method=releaseLocks start=1526613247419 end=1526613247425 duration=6 from=org.apache.hadoop.hive.ql.Driver> May 18, 8:44:07.431 AM ERROR org.apache.hive.service.cli.operation.Operation [HiveServer2-Background-Pool: Thread-1391]: Error running hive query: org.apache.hive.service.cli.HiveSQLException: Error while processing statement: FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask at org.apache.hive.service.cli.operation.Operation.toSQLException(Operation.java:400) at org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:239) at org.apache.hive.service.cli.operation.SQLOperation.access$300(SQLOperation.java:88) at org.apache.hive.service.cli.operation.SQLOperation$3$1.run(SQLOperation.java:293) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920) at org.apache.hive.service.cli.operation.SQLOperation$3.run(SQLOperation.java:306) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) May 18, 8:44:07.433 AM INFO org.apache.hive.service.cli.thrift.ThriftCLIService [HiveServer2-Handler-Pool: Thread-575]: Session disconnected without closing properly, close it now ... View more

I have my Hive jobs scheduled through oozie and I randomly get: ERROR : Failed to execute spark task, with exception 'org.apache.hadoop.hive.ql.metadata.HiveException(Failed to create spark client.)' org.apache.hadoop.hive.ql.metadata.HiveException: Failed to create spark client. at org.apache.hadoop.hive.ql.exec.spark.session.SparkSessionImpl.open(SparkSessionImpl.java:64) at org.apache.hadoop.hive.ql.exec.spark.session.SparkSessionManagerImpl.getSession(SparkSessionManagerImpl.java:114) at org.apache.hadoop.hive.ql.exec.spark.SparkUtilities.getSparkSession(SparkUtilities.java:111) at org.apache.hadoop.hive.ql.exec.spark.SparkTask.execute(SparkTask.java:99) at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:214) at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:100) at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1977) at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1690) at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1422) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1206) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1201) at org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:237) at org.apache.hive.service.cli.operation.SQLOperation.access$300(SQLOperation.java:88) at org.apache.hive.service.cli.operation.SQLOperation$3$1.run(SQLOperation.java:293) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920) at org.apache.hive.service.cli.operation.SQLOperation$3.run(SQLOperation.java:306) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) Caused by: java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.util.concurrent.TimeoutException: Timed out waiting for client connection. at com.google.common.base.Throwables.propagate(Throwables.java:156) at org.apache.hive.spark.client.SparkClientImpl.<init>(SparkClientImpl.java:124) at org.apache.hive.spark.client.SparkClientFactory.createClient(SparkClientFactory.java:80) at org.apache.hadoop.hive.ql.exec.spark.RemoteHiveSparkClient.createRemoteClient(RemoteHiveSparkClient.java:100) at org.apache.hadoop.hive.ql.exec.spark.RemoteHiveSparkClient.<init>(RemoteHiveSparkClient.java:96) at org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactory.createHiveSparkClient(HiveSparkClientFactory.java:68) at org.apache.hadoop.hive.ql.exec.spark.session.SparkSessionImpl.open(SparkSessionImpl.java:62) ... 22 more Caused by: java.util.concurrent.ExecutionException: java.util.concurrent.TimeoutException: Timed out waiting for client connection. at io.netty.util.concurrent.AbstractFuture.get(AbstractFuture.java:37) at org.apache.hive.spark.client.SparkClientImpl.<init>(SparkClientImpl.java:108) ... 27 more Caused by: java.util.concurrent.TimeoutException: Timed out waiting for client connection. at org.apache.hive.spark.client.rpc.RpcServer$2.run(RpcServer.java:141) at io.netty.util.concurrent.PromiseTask$RunnableAdapter.call(PromiseTask.java:38) at io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:120) at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:357) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:357) at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111) ... 1 more ERROR : Failed to execute spark task, with exception 'org.apache.hadoop.hive.ql.metadata.HiveException(Failed to create spark client.)' org.apache.hadoop.hive.ql.metadata.HiveException: Failed to create spark client. at org.apache.hadoop.hive.ql.exec.spark.session.SparkSessionImpl.open(SparkSessionImpl.java:64) at org.apache.hadoop.hive.ql.exec.spark.session.SparkSessionManagerImpl.getSession(SparkSessionManagerImpl.java:114) at org.apache.hadoop.hive.ql.exec.spark.SparkUtilities.getSparkSession(SparkUtilities.java:111) at org.apache.hadoop.hive.ql.exec.spark.SparkTask.execute(SparkTask.java:99) at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:214) at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:100) at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1977) at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1690) at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1422) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1206) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1201) at org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:237) at org.apache.hive.service.cli.operation.SQLOperation.access$300(SQLOperation.java:88) at org.apache.hive.service.cli.operation.SQLOperation$3$1.run(SQLOperation.java:293) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920) at org.apache.hive.service.cli.operation.SQLOperation$3.run(SQLOperation.java:306) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) Caused by: java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.util.concurrent.TimeoutException: Timed out waiting for client connection. at com.google.common.base.Throwables.propagate(Throwables.java:156) at org.apache.hive.spark.client.SparkClientImpl.<init>(SparkClientImpl.java:124) at org.apache.hive.spark.client.SparkClientFactory.createClient(SparkClientFactory.java:80) at org.apache.hadoop.hive.ql.exec.spark.RemoteHiveSparkClient.createRemoteClient(RemoteHiveSparkClient.java:100) at org.apache.hadoop.hive.ql.exec.spark.RemoteHiveSparkClient.<init>(RemoteHiveSparkClient.java:96) at org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactory.createHiveSparkClient(HiveSparkClientFactory.java:68) at org.apache.hadoop.hive.ql.exec.spark.session.SparkSessionImpl.open(SparkSessionImpl.java:62) ... 22 more Caused by: java.util.concurrent.ExecutionException: java.util.concurrent.TimeoutException: Timed out waiting for client connection. at io.netty.util.concurrent.AbstractFuture.get(AbstractFuture.java:37) at org.apache.hive.spark.client.SparkClientImpl.<init>(SparkClientImpl.java:108) ... 27 more Caused by: java.util.concurrent.TimeoutException: Timed out waiting for client connection. at org.apache.hive.spark.client.rpc.RpcServer$2.run(RpcServer.java:141) at io.netty.util.concurrent.PromiseTask$RunnableAdapter.call(PromiseTask.java:38) at io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:120) at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:357) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:357) at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111) ... 1 more ERROR : FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.spark.SparkTask INFO : Completed executing command(queryId=hive_20180506233232_2a9bc92f-d830-435b-91df-eafd2236a9fc); Time taken: 90.051 seconds Error: Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.spark.SparkTask (state=08S01,code=1) Closing: 0: jdbc:hive2://ip-172-31-4-192.ap-south-1.compute.internal:10000/default Intercepting System.exit(2) Failing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.Hive2Main], exit code [2] What can I do to fix this? It happens randomly even at times when workload is not much Cluster Confi: Total vCores: 60 Total memory: 205 GB 3 worker nodes in my cluster are m4.4x Large ec2 instances, 2 m4.2x large ec2 instance with Yarn (Node manager) Container memory is set to 8GB in Yarn config spark.executor.cores=1(since I will have multiple users running queries and also multiple queries running at same time) spark java heap size = 5GB Memory overhead = 2GB Is there something I can do to improve the situation? ... View more