Member since
05-04-2018
30
Posts
0
Kudos Received
0
Solutions
05-18-2018
03:35 AM
Logged in as: dr.who
Application
AboutJobsTools
Log Type: syslog
Log Upload Time: Fri May 18 08:51:41 +0530 2018
Log Length: 35487
2018-05-18 08:50:58,682 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Created MRAppMaster for application appattempt_1526609705906_0273_000001
2018-05-18 08:50:59,228 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Executing with tokens:
2018-05-18 08:50:59,228 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Kind: YARN_AM_RM_TOKEN, Service: , Ident: (org.apache.hadoop.yarn.security.AMRMTokenIdentifier@b992aa2)
2018-05-18 08:50:59,959 WARN [main] org.apache.hadoop.util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2018-05-18 08:51:00,073 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: OutputCommitter set in config org.apache.hadoop.hive.ql.io.HiveFileFormatUtils$NullOutputCommitter
2018-05-18 08:51:00,075 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: OutputCommitter is org.apache.hadoop.hive.ql.io.HiveFileFormatUtils$NullOutputCommitter
2018-05-18 08:51:00,123 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.jobhistory.EventType for class org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler
2018-05-18 08:51:00,124 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.job.event.JobEventType for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobEventDispatcher
2018-05-18 08:51:00,125 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.job.event.TaskEventType for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$TaskEventDispatcher
2018-05-18 08:51:00,125 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.job.event.TaskAttemptEventType for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$TaskAttemptEventDispatcher
2018-05-18 08:51:00,126 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventType for class org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler
2018-05-18 08:51:00,131 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.speculate.Speculator$EventType for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$SpeculatorEventDispatcher
2018-05-18 08:51:00,131 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.rm.ContainerAllocator$EventType for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$ContainerAllocatorRouter
2018-05-18 08:51:00,132 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncher$EventType for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$ContainerLauncherRouter
2018-05-18 08:51:00,169 INFO [main] org.apache.hadoop.mapreduce.v2.jobhistory.JobHistoryUtils: Default file system [hdfs://ip-172-31-4-192.ap-south-1.compute.internal:8020]
2018-05-18 08:51:00,194 INFO [main] org.apache.hadoop.mapreduce.v2.jobhistory.JobHistoryUtils: Default file system [hdfs://ip-172-31-4-192.ap-south-1.compute.internal:8020]
2018-05-18 08:51:00,212 INFO [main] org.apache.hadoop.mapreduce.v2.jobhistory.JobHistoryUtils: Default file system [hdfs://ip-172-31-4-192.ap-south-1.compute.internal:8020]
2018-05-18 08:51:00,223 INFO [main] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Emitting job history data to the timeline server is not enabled
2018-05-18 08:51:00,277 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.job.event.JobFinishEvent$Type for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobFinishEventHandler
2018-05-18 08:51:00,537 INFO [main] org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from hadoop-metrics2.properties
2018-05-18 08:51:00,605 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot period at 10 second(s).
2018-05-18 08:51:00,605 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: MRAppMaster metrics system started
2018-05-18 08:51:00,625 INFO [main] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Adding job token for job_1526609705906_0273 to jobTokenSecretManager
2018-05-18 08:51:00,825 INFO [main] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Not uberizing job_1526609705906_0273 because: not enabled; too many reduces; too much input;
2018-05-18 08:51:00,857 INFO [main] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Input size for job job_1526609705906_0273 = 1155935596. Number of splits = 4
2018-05-18 08:51:00,861 INFO [main] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Number of reduces for job job_1526609705906_0273 = 18
2018-05-18 08:51:00,861 INFO [main] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: job_1526609705906_0273Job Transitioned from NEW to INITED
2018-05-18 08:51:00,863 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: MRAppMaster launching normal, non-uberized, multi-container job job_1526609705906_0273.
2018-05-18 08:51:00,901 INFO [main] org.apache.hadoop.ipc.CallQueueManager: Using callQueue: class java.util.concurrent.LinkedBlockingQueue queueCapacity: 100
2018-05-18 08:51:00,912 INFO [Socket Reader #1 for port 42741] org.apache.hadoop.ipc.Server: Starting Socket Reader #1 for port 42741
2018-05-18 08:51:00,936 INFO [main] org.apache.hadoop.yarn.factories.impl.pb.RpcServerFactoryPBImpl: Adding protocol org.apache.hadoop.mapreduce.v2.api.MRClientProtocolPB to the server
2018-05-18 08:51:00,936 INFO [IPC Server Responder] org.apache.hadoop.ipc.Server: IPC Server Responder: starting
2018-05-18 08:51:00,937 INFO [IPC Server listener on 42741] org.apache.hadoop.ipc.Server: IPC Server listener on 42741: starting
2018-05-18 08:51:00,938 INFO [main] org.apache.hadoop.mapreduce.v2.app.client.MRClientService: Instantiated MRClientService at ip-172-31-13-118.ap-south-1.compute.internal/172.31.13.118:42741
2018-05-18 08:51:01,021 INFO [main] org.mortbay.log: Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog
2018-05-18 08:51:01,031 INFO [main] org.apache.hadoop.security.authentication.server.AuthenticationFilter: Unable to initialize FileSignerSecretProvider, falling back to use random secrets.
2018-05-18 08:51:01,036 INFO [main] org.apache.hadoop.http.HttpRequestLog: Http request log for http.requests.mapreduce is not defined
2018-05-18 08:51:01,044 INFO [main] org.apache.hadoop.http.HttpServer2: Added global filter 'safety' (class=org.apache.hadoop.http.HttpServer2$QuotingInputFilter)
2018-05-18 08:51:01,050 INFO [main] org.apache.hadoop.http.HttpServer2: Added filter AM_PROXY_FILTER (class=org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter) to context mapreduce
2018-05-18 08:51:01,050 INFO [main] org.apache.hadoop.http.HttpServer2: Added filter AM_PROXY_FILTER (class=org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter) to context static
2018-05-18 08:51:01,053 INFO [main] org.apache.hadoop.http.HttpServer2: adding path spec: /mapreduce/*
2018-05-18 08:51:01,053 INFO [main] org.apache.hadoop.http.HttpServer2: adding path spec: /ws/*
2018-05-18 08:51:01,061 INFO [main] org.apache.hadoop.http.HttpServer2: Jetty bound to port 36178
2018-05-18 08:51:01,061 INFO [main] org.mortbay.log: jetty-6.1.26.cloudera.4
2018-05-18 08:51:01,088 INFO [main] org.mortbay.log: Extract jar:file:/opt/cloudera/parcels/CDH-5.10.1-1.cdh5.10.1.p0.10/jars/hadoop-yarn-common-2.6.0-cdh5.10.1.jar!/webapps/mapreduce to /tmp/Jetty_0_0_0_0_36178_mapreduce____davbey/webapp
2018-05-18 08:51:01,355 INFO [main] org.mortbay.log: Started HttpServer2$SelectChannelConnectorWithSafeStartup@0.0.0.0:36178
2018-05-18 08:51:01,355 INFO [main] org.apache.hadoop.yarn.webapp.WebApps: Web app /mapreduce started at 36178
2018-05-18 08:51:01,697 INFO [main] org.apache.hadoop.yarn.webapp.WebApps: Registered webapp guice modules
2018-05-18 08:51:01,701 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.speculate.DefaultSpeculator: JOB_CREATE job_1526609705906_0273
2018-05-18 08:51:01,704 INFO [main] org.apache.hadoop.ipc.CallQueueManager: Using callQueue: class java.util.concurrent.LinkedBlockingQueue queueCapacity: 3000
2018-05-18 08:51:01,704 INFO [Socket Reader #1 for port 34196] org.apache.hadoop.ipc.Server: Starting Socket Reader #1 for port 34196
2018-05-18 08:51:01,712 INFO [IPC Server Responder] org.apache.hadoop.ipc.Server: IPC Server Responder: starting
2018-05-18 08:51:01,712 INFO [IPC Server listener on 34196] org.apache.hadoop.ipc.Server: IPC Server listener on 34196: starting
2018-05-18 08:51:01,732 INFO [main] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: nodeBlacklistingEnabled:true
2018-05-18 08:51:01,732 INFO [main] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: maxTaskFailuresPerNode is 3
2018-05-18 08:51:01,732 INFO [main] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: blacklistDisablePercent is 33
2018-05-18 08:51:01,795 INFO [main] org.apache.hadoop.yarn.client.RMProxy: Connecting to ResourceManager at ip-172-31-4-192.ap-south-1.compute.internal/172.31.4.192:8030
2018-05-18 08:51:01,852 INFO [main] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: maxContainerCapability: <memory:8192, vCores:6>
2018-05-18 08:51:01,852 INFO [main] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: queue: root.users.hue
2018-05-18 08:51:01,855 INFO [main] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Upper limit on the thread pool size is 500
2018-05-18 08:51:01,855 INFO [main] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: The thread pool initial size is 10
2018-05-18 08:51:01,864 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: job_1526609705906_0273Job Transitioned from INITED to SETUP
2018-05-18 08:51:01,865 INFO [CommitterEvent Processor #0] org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler: Processing the event EventType: JOB_SETUP
2018-05-18 08:51:01,867 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: job_1526609705906_0273Job Transitioned from SETUP to RUNNING
2018-05-18 08:51:01,936 INFO [eventHandlingThread] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Event Writer setup for JobId: job_1526609705906_0273, File: hdfs://ip-172-31-4-192.ap-south-1.compute.internal:8020/user/hue/.staging/job_1526609705906_0273/job_1526609705906_0273_1.jhist
2018-05-18 08:51:01,947 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1526609705906_0273_m_000000 Task Transitioned from NEW to SCHEDULED
2018-05-18 08:51:01,983 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1526609705906_0273_m_000001 Task Transitioned from NEW to SCHEDULED
2018-05-18 08:51:02,010 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1526609705906_0273_m_000002 Task Transitioned from NEW to SCHEDULED
2018-05-18 08:51:02,037 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1526609705906_0273_m_000003 Task Transitioned from NEW to SCHEDULED
2018-05-18 08:51:02,038 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1526609705906_0273_r_000000 Task Transitioned from NEW to SCHEDULED
2018-05-18 08:51:02,038 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1526609705906_0273_r_000001 Task Transitioned from NEW to SCHEDULED
2018-05-18 08:51:02,039 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1526609705906_0273_r_000002 Task Transitioned from NEW to SCHEDULED
2018-05-18 08:51:02,039 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1526609705906_0273_r_000003 Task Transitioned from NEW to SCHEDULED
2018-05-18 08:51:02,040 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1526609705906_0273_r_000004 Task Transitioned from NEW to SCHEDULED
2018-05-18 08:51:02,041 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1526609705906_0273_r_000005 Task Transitioned from NEW to SCHEDULED
2018-05-18 08:51:02,041 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1526609705906_0273_r_000006 Task Transitioned from NEW to SCHEDULED
2018-05-18 08:51:02,042 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1526609705906_0273_r_000007 Task Transitioned from NEW to SCHEDULED
2018-05-18 08:51:02,042 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1526609705906_0273_r_000008 Task Transitioned from NEW to SCHEDULED
2018-05-18 08:51:02,043 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1526609705906_0273_r_000009 Task Transitioned from NEW to SCHEDULED
2018-05-18 08:51:02,044 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1526609705906_0273_r_000010 Task Transitioned from NEW to SCHEDULED
2018-05-18 08:51:02,045 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1526609705906_0273_r_000011 Task Transitioned from NEW to SCHEDULED
2018-05-18 08:51:02,046 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1526609705906_0273_r_000012 Task Transitioned from NEW to SCHEDULED
2018-05-18 08:51:02,047 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1526609705906_0273_r_000013 Task Transitioned from NEW to SCHEDULED
2018-05-18 08:51:02,047 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1526609705906_0273_r_000014 Task Transitioned from NEW to SCHEDULED
2018-05-18 08:51:02,048 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1526609705906_0273_r_000015 Task Transitioned from NEW to SCHEDULED
2018-05-18 08:51:02,048 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1526609705906_0273_r_000016 Task Transitioned from NEW to SCHEDULED
2018-05-18 08:51:02,049 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1526609705906_0273_r_000017 Task Transitioned from NEW to SCHEDULED
2018-05-18 08:51:02,051 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1526609705906_0273_m_000000_0 TaskAttempt Transitioned from NEW to UNASSIGNED
2018-05-18 08:51:02,051 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1526609705906_0273_m_000001_0 TaskAttempt Transitioned from NEW to UNASSIGNED
2018-05-18 08:51:02,051 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1526609705906_0273_m_000002_0 TaskAttempt Transitioned from NEW to UNASSIGNED
2018-05-18 08:51:02,051 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1526609705906_0273_m_000003_0 TaskAttempt Transitioned from NEW to UNASSIGNED
2018-05-18 08:51:02,051 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1526609705906_0273_r_000000_0 TaskAttempt Transitioned from NEW to UNASSIGNED
2018-05-18 08:51:02,051 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1526609705906_0273_r_000001_0 TaskAttempt Transitioned from NEW to UNASSIGNED
2018-05-18 08:51:02,052 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1526609705906_0273_r_000002_0 TaskAttempt Transitioned from NEW to UNASSIGNED
2018-05-18 08:51:02,052 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1526609705906_0273_r_000003_0 TaskAttempt Transitioned from NEW to UNASSIGNED
2018-05-18 08:51:02,052 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1526609705906_0273_r_000004_0 TaskAttempt Transitioned from NEW to UNASSIGNED
2018-05-18 08:51:02,052 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1526609705906_0273_r_000005_0 TaskAttempt Transitioned from NEW to UNASSIGNED
2018-05-18 08:51:02,052 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1526609705906_0273_r_000006_0 TaskAttempt Transitioned from NEW to UNASSIGNED
2018-05-18 08:51:02,053 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1526609705906_0273_r_000007_0 TaskAttempt Transitioned from NEW to UNASSIGNED
2018-05-18 08:51:02,053 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1526609705906_0273_r_000008_0 TaskAttempt Transitioned from NEW to UNASSIGNED
2018-05-18 08:51:02,053 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1526609705906_0273_r_000009_0 TaskAttempt Transitioned from NEW to UNASSIGNED
2018-05-18 08:51:02,053 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1526609705906_0273_r_000010_0 TaskAttempt Transitioned from NEW to UNASSIGNED
2018-05-18 08:51:02,053 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1526609705906_0273_r_000011_0 TaskAttempt Transitioned from NEW to UNASSIGNED
2018-05-18 08:51:02,054 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1526609705906_0273_r_000012_0 TaskAttempt Transitioned from NEW to UNASSIGNED
2018-05-18 08:51:02,054 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1526609705906_0273_r_000013_0 TaskAttempt Transitioned from NEW to UNASSIGNED
2018-05-18 08:51:02,054 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1526609705906_0273_r_000014_0 TaskAttempt Transitioned from NEW to UNASSIGNED
2018-05-18 08:51:02,054 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1526609705906_0273_r_000015_0 TaskAttempt Transitioned from NEW to UNASSIGNED
2018-05-18 08:51:02,054 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1526609705906_0273_r_000016_0 TaskAttempt Transitioned from NEW to UNASSIGNED
2018-05-18 08:51:02,054 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1526609705906_0273_r_000017_0 TaskAttempt Transitioned from NEW to UNASSIGNED
2018-05-18 08:51:02,056 INFO [Thread-52] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: mapResourceRequest:<memory:1024, vCores:1>
2018-05-18 08:51:02,073 INFO [Thread-52] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: reduceResourceRequest:<memory:1024, vCores:1>
2018-05-18 08:51:02,448 INFO [eventHandlingThread] org.apache.hadoop.mapreduce.v2.jobhistory.JobHistoryUtils: Default file system [hdfs://ip-172-31-4-192.ap-south-1.compute.internal:8020]
2018-05-18 08:51:02,854 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Before Scheduling: PendingReds:18 ScheduledMaps:4 ScheduledReds:0 AssignedMaps:0 AssignedReds:0 CompletedMaps:0 CompletedReds:0 ContAlloc:0 ContRel:0 HostLocal:0 RackLocal:0
2018-05-18 08:51:02,892 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: getResources() for application_1526609705906_0273: ask=6 release= 0 newContainers=0 finishedContainers=0 resourcelimit=<memory:101888, vCores:54> knownNMs=6
2018-05-18 08:51:02,892 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Recalculating schedule, headroom=<memory:101888, vCores:54>
2018-05-18 08:51:02,892 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Reduce slow start threshold not met. completedMapsForReduceSlowstart 4
2018-05-18 08:51:02,996 INFO [IPC Server handler 0 on 42741] org.apache.hadoop.mapreduce.v2.app.client.MRClientService: Getting task report for MAP job_1526609705906_0273. Report-size will be 4
2018-05-18 08:51:03,051 INFO [IPC Server handler 0 on 42741] org.apache.hadoop.mapreduce.v2.app.client.MRClientService: Getting task report for REDUCE job_1526609705906_0273. Report-size will be 18
2018-05-18 08:51:03,904 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Got allocated containers 4
2018-05-18 08:51:03,906 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned container container_1526609705906_0273_01_000002 to attempt_1526609705906_0273_m_000001_0
2018-05-18 08:51:03,907 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned container container_1526609705906_0273_01_000003 to attempt_1526609705906_0273_m_000000_0
2018-05-18 08:51:03,908 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned container container_1526609705906_0273_01_000004 to attempt_1526609705906_0273_m_000002_0
2018-05-18 08:51:03,908 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned container container_1526609705906_0273_01_000005 to attempt_1526609705906_0273_m_000003_0
2018-05-18 08:51:03,908 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Recalculating schedule, headroom=<memory:97792, vCores:50>
2018-05-18 08:51:03,908 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Reduce slow start threshold not met. completedMapsForReduceSlowstart 4
2018-05-18 08:51:03,908 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: After Scheduling: PendingReds:18 ScheduledMaps:0 ScheduledReds:0 AssignedMaps:4 AssignedReds:0 CompletedMaps:0 CompletedReds:0 ContAlloc:4 ContRel:0 HostLocal:1 RackLocal:3
2018-05-18 08:51:03,974 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: The job-jar file on the remote FS is hdfs://ip-172-31-4-192.ap-south-1.compute.internal:8020/user/hue/.staging/job_1526609705906_0273/job.jar
2018-05-18 08:51:03,976 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: The job-conf file on the remote FS is /user/hue/.staging/job_1526609705906_0273/job.xml
2018-05-18 08:51:03,985 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Adding #0 tokens and #1 secret keys for NM use for launching container
2018-05-18 08:51:03,985 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Size of containertokens_dob is 1
2018-05-18 08:51:03,985 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Putting shuffle token in serviceData
2018-05-18 08:51:04,022 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapred.JobConf: Task java-opts do not specify heap size. Setting task attempt jvm max heap size to -Xmx820m
2018-05-18 08:51:04,026 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1526609705906_0273_m_000001_0 TaskAttempt Transitioned from UNASSIGNED to ASSIGNED
2018-05-18 08:51:04,029 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapred.JobConf: Task java-opts do not specify heap size. Setting task attempt jvm max heap size to -Xmx820m
2018-05-18 08:51:04,029 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1526609705906_0273_m_000000_0 TaskAttempt Transitioned from UNASSIGNED to ASSIGNED
2018-05-18 08:51:04,030 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapred.JobConf: Task java-opts do not specify heap size. Setting task attempt jvm max heap size to -Xmx820m
2018-05-18 08:51:04,030 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1526609705906_0273_m_000002_0 TaskAttempt Transitioned from UNASSIGNED to ASSIGNED
2018-05-18 08:51:04,031 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapred.JobConf: Task java-opts do not specify heap size. Setting task attempt jvm max heap size to -Xmx820m
2018-05-18 08:51:04,032 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1526609705906_0273_m_000003_0 TaskAttempt Transitioned from UNASSIGNED to ASSIGNED
2018-05-18 08:51:04,034 INFO [ContainerLauncher #1] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_REMOTE_LAUNCH for container container_1526609705906_0273_01_000003 taskAttempt attempt_1526609705906_0273_m_000000_0
2018-05-18 08:51:04,034 INFO [ContainerLauncher #0] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_REMOTE_LAUNCH for container container_1526609705906_0273_01_000002 taskAttempt attempt_1526609705906_0273_m_000001_0
2018-05-18 08:51:04,034 INFO [ContainerLauncher #2] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_REMOTE_LAUNCH for container container_1526609705906_0273_01_000004 taskAttempt attempt_1526609705906_0273_m_000002_0
2018-05-18 08:51:04,039 INFO [ContainerLauncher #3] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_REMOTE_LAUNCH for container container_1526609705906_0273_01_000005 taskAttempt attempt_1526609705906_0273_m_000003_0
2018-05-18 08:51:04,042 INFO [ContainerLauncher #1] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Launching attempt_1526609705906_0273_m_000000_0
2018-05-18 08:51:04,042 INFO [ContainerLauncher #0] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Launching attempt_1526609705906_0273_m_000001_0
2018-05-18 08:51:04,042 INFO [ContainerLauncher #2] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Launching attempt_1526609705906_0273_m_000002_0
2018-05-18 08:51:04,042 INFO [ContainerLauncher #3] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Launching attempt_1526609705906_0273_m_000003_0
2018-05-18 08:51:04,168 INFO [ContainerLauncher #2] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Shuffle port returned by ContainerManager for attempt_1526609705906_0273_m_000002_0 : 13562
2018-05-18 08:51:04,168 INFO [ContainerLauncher #0] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Shuffle port returned by ContainerManager for attempt_1526609705906_0273_m_000001_0 : 13562
2018-05-18 08:51:04,170 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: TaskAttempt: [attempt_1526609705906_0273_m_000002_0] using containerId: [container_1526609705906_0273_01_000004 on NM: [ip-172-31-1-207.ap-south-1.compute.internal:8041]
2018-05-18 08:51:04,173 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1526609705906_0273_m_000002_0 TaskAttempt Transitioned from ASSIGNED to RUNNING
2018-05-18 08:51:04,174 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: TaskAttempt: [attempt_1526609705906_0273_m_000001_0] using containerId: [container_1526609705906_0273_01_000002 on NM: [ip-172-31-1-207.ap-south-1.compute.internal:8041]
2018-05-18 08:51:04,174 INFO [ContainerLauncher #3] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Shuffle port returned by ContainerManager for attempt_1526609705906_0273_m_000003_0 : 13562
2018-05-18 08:51:04,174 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1526609705906_0273_m_000001_0 TaskAttempt Transitioned from ASSIGNED to RUNNING
2018-05-18 08:51:04,175 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1526609705906_0273_m_000002 Task Transitioned from SCHEDULED to RUNNING
2018-05-18 08:51:04,175 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: TaskAttempt: [attempt_1526609705906_0273_m_000003_0] using containerId: [container_1526609705906_0273_01_000005 on NM: [ip-172-31-1-207.ap-south-1.compute.internal:8041]
2018-05-18 08:51:04,175 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1526609705906_0273_m_000003_0 TaskAttempt Transitioned from ASSIGNED to RUNNING
2018-05-18 08:51:04,176 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1526609705906_0273_m_000001 Task Transitioned from SCHEDULED to RUNNING
2018-05-18 08:51:04,176 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1526609705906_0273_m_000003 Task Transitioned from SCHEDULED to RUNNING
2018-05-18 08:51:04,182 INFO [ContainerLauncher #1] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Shuffle port returned by ContainerManager for attempt_1526609705906_0273_m_000000_0 : 13562
2018-05-18 08:51:04,183 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: TaskAttempt: [attempt_1526609705906_0273_m_000000_0] using containerId: [container_1526609705906_0273_01_000003 on NM: [ip-172-31-1-207.ap-south-1.compute.internal:8041]
2018-05-18 08:51:04,184 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1526609705906_0273_m_000000_0 TaskAttempt Transitioned from ASSIGNED to RUNNING
2018-05-18 08:51:04,185 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1526609705906_0273_m_000000 Task Transitioned from SCHEDULED to RUNNING
2018-05-18 08:51:04,911 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: getResources() for application_1526609705906_0273: ask=6 release= 0 newContainers=0 finishedContainers=0 resourcelimit=<memory:97792, vCores:49> knownNMs=6
2018-05-18 08:51:04,911 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Recalculating schedule, headroom=<memory:97792, vCores:49>
2018-05-18 08:51:04,911 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Reduce slow start threshold not met. completedMapsForReduceSlowstart 4
2018-05-18 08:51:07,918 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Recalculating schedule, headroom=<memory:62806, vCores:48>
2018-05-18 08:51:07,918 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Reduce slow start threshold not met. completedMapsForReduceSlowstart 4
2018-05-18 08:51:08,925 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Received completed container container_1526609705906_0273_01_000004
2018-05-18 08:51:08,926 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Received completed container container_1526609705906_0273_01_000003
2018-05-18 08:51:08,926 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Recalculating schedule, headroom=<memory:64854, vCores:49>
2018-05-18 08:51:08,926 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics report from attempt_1526609705906_0273_m_000002_0: Container killed on request. Exit code is 137
Container exited with a non-zero exit code 137
Killed by external signal
2018-05-18 08:51:08,926 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Reduce slow start threshold not met. completedMapsForReduceSlowstart 4
2018-05-18 08:51:08,926 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: After Scheduling: PendingReds:18 ScheduledMaps:0 ScheduledReds:0 AssignedMaps:2 AssignedReds:0 CompletedMaps:0 CompletedReds:0 ContAlloc:4 ContRel:0 HostLocal:1 RackLocal:3
2018-05-18 08:51:08,932 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1526609705906_0273_m_000002_0 TaskAttempt Transitioned from RUNNING to FAILED
2018-05-18 08:51:08,932 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics report from attempt_1526609705906_0273_m_000000_0: Container killed on request. Exit code is 137
Container exited with a non-zero exit code 137
Killed by external signal
2018-05-18 08:51:08,932 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1526609705906_0273_m_000000_0 TaskAttempt Transitioned from RUNNING to FAILED
2018-05-18 08:51:08,933 INFO [ContainerLauncher #4] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_COMPLETED for container container_1526609705906_0273_01_000004 taskAttempt attempt_1526609705906_0273_m_000002_0
2018-05-18 08:51:08,943 INFO [Thread-52] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: 1 failures on node ip-172-31-1-207.ap-south-1.compute.internal
2018-05-18 08:51:08,944 INFO [ContainerLauncher #5] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_COMPLETED for container container_1526609705906_0273_01_000003 taskAttempt attempt_1526609705906_0273_m_000000_0
2018-05-18 08:51:08,945 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1526609705906_0273_m_000002_1 TaskAttempt Transitioned from NEW to UNASSIGNED
2018-05-18 08:51:08,945 INFO [Thread-52] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: 2 failures on node ip-172-31-1-207.ap-south-1.compute.internal
2018-05-18 08:51:08,945 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1526609705906_0273_m_000000_1 TaskAttempt Transitioned from NEW to UNASSIGNED
2018-05-18 08:51:08,946 INFO [Thread-52] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Added attempt_1526609705906_0273_m_000002_1 to list of failed maps
2018-05-18 08:51:08,946 INFO [Thread-52] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Added attempt_1526609705906_0273_m_000000_1 to list of failed maps
This morning I started getting this error on all my jobs. What is the issue here? There were no changes made in cluster when this started happening.
... View more
Labels:
- Labels:
-
Apache Hive
05-08-2018
01:19 AM
hive.spark.client.connect.timeout=90000; I need to set this and I would like to set it in a configuration file and not in hql files. I do not see this property in hive-site.xml or spark xml. Which files do I put this in? For now, I have put it in: Service Monitor Client Config Overrides Is this the right place for it? Also, I don't have it in files anywhere so not sure if I am expected to add it in overrides.
... View more
Labels:
- Labels:
-
Apache Hive
-
Apache Spark
05-07-2018
05:00 AM
I believe I am already using Beeline. Yes, I tried switching back to MapReduce execution engine but still get the same error. @Geoffrey Shelton Okot I do have hive server 2 up and running on the cluster. Also, I don't find it ideal having to switch to Hive on Spark because of this unidentified issue. Do you mind pointing out what could be the other reasons for interruption on Map Reduce job or if it is possible to escalate it?
... View more
05-06-2018
11:43 AM
@Geoffrey Shelton Okot: Thanks. Is there something else that I can do to resolve this issue in Map-red? Spark isn't working all that well for me.
... View more
05-05-2018
08:39 PM
@Geoffrey Shelton Okot I am still getting the error Also, from the metadata, I see that it is executing on the node that has both Nodemanager and HDFS. Is there somewhere I can trace down reason for this error? Note: /var/log/hive/hadoop-cmf-CD-HIVE-XCVXskZf-HIVESERVER2-ip-172-31-4-192.ap-south-1.compute.internal.log.out This path does not exist on any of my nodes. Once I start getting this error, all my hive jobs fail until after a while and are usually fixed if I restart my cluster. ERROR : Execution failed with exit status: 1
ERROR : Obtaining error information
ERROR :
Task failed!
Task ID:
Stage-20
Logs:
ERROR : <a href="http://PUBLICDOMAINNAME:8888/filebrowser/view=/var/log/hive/hadoop-cmf-CD-HIVE-XCVXskZf-HIVESERVER2-ip-172-31-4-192.ap-south-1.compute.internal.log.out">/var/log/hive/hadoop-cmf-CD-HIVE-XCVXskZf-HIVESERVER2-ip-172-31-4-192.ap-south-1.compute.internal.log.out</a>
ERROR : FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask
WARN : Shutting down task : Stage-1:MAPRED
WARN : Shutting down task : Stage-7:MAPRED
WARN : Shutting down task : Stage-11:MAPRED
INFO : Completed executing command(queryId=hive_20180506020606_5277a480-1de4-4a48-9469-b6a2e7d0b200); Time taken: 0.163 seconds
Error: Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask (state=08S01,code=1)
Closing: 0: jdbc:hive2://ip-172-31-4-192.ap-south-1.compute.internal:10000/default
Intercepting System.exit(2)
Failing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.Hive2Main], exit code [2]
... View more
05-05-2018
08:03 PM
@Geoffrey Shelton Okot : I have made the changes i.e
dfs.client.use.datanode.hostname=true 2. Removed yarn from nodes that did not have HDFS 3. Although still have a node that only has HDFS (Is there a way that Hadoop can prioritize execution of jobs on worker nodes that have both of these services started on them?) The error happens randomly throughout the day and in morning hours when most of our hive reports run. I am in IST time zone. Please give me until tomorrow morning IST to confirm if this has completely resolved the issue 🙂 Also, I would like to know what does exactly happen if zookeeper data is lost? I tried installing zookeeper server instance on other nodes but the new instances did not start and kept returning supervisord failed to start.
... View more
05-05-2018
11:59 AM
Thanks. Let me check with
dfs.client.use.datanode.hostname=true and I will confirm. Also, adding 3 zookeeper servers as well. Thanks for clarifying that is required to take SNN on a different node when only Namenode is working in HA mode. But what I am not able to understand is if the error: Job Submission failed with exception 'java.lang.RuntimeException(Error caching map.xml: java.io.InterruptedIOException: Interruped while waiting for IO on channel java.nio.channels.SocketChannel[connected local=/172.31.4.192:42632 remote=/172.31.4.192:50010].
and ERROR : FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask
INFO : Completed executing command(queryId=hive_20180504194545_4cbaf9cb-9f83-410f-9804-1618f1ee10d1); Time taken: 0.156 seconds
Error: Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask (state=08S01,code=1)
Closing: 0: jdbc:hive2://ip-172-31-4-192.ap-south-1.compute.internal:10000/default
Intercepting System.exit(2) is related to having under-replicated data. My data has been replicated across 3 nodes now and I am still running into this error and so was I before decommissioning my data nodes that have run out of space and commissioning the new ones. @Geoffrey Shelton Okot 3. Gateway server has following roles: 7 Role(s) HDFS Gateway
HDFS HttpFS
Hive Gateway
YARN (MR2 Included) NodeManager
Sentry Gateway Spark Gateway 9.28s ago 4. This server only has
YARN (MR2 Included) NodeManager Added it since our vCPU's are upto full usage sometimes. I would also like to clarify if it is okay to have HDFS Data Node on a server without Node Manager on it and if it is okay to have node manager without having HDFS on same node.
... View more
05-05-2018
10:39 AM
Also, it used to work fine until sometime back. I still don't understand why did it start happening out of nowhere. My hive queries running through oozie still work alot of times and some other times it breaks with this error
... View more
05-05-2018
10:37 AM
Thanks. Could you please explain what a multi-home cluster is. For sake of clarification, I am sharing how my services are distributed across servers: It's a 6 node cluster 3 of my instances are m4.4x Large 3 of instances are m4.2x Large 1 Master Server has following services on it: -> hue ->hIVE -> Zoopkeer ->oozie ->sentry ->spark ->sqoop 2 -> nn -> snn 2 workers nodes with ->Node Manager -> HDFS 3. Gateway server with -> gateways ->node manager 4.1 node with -> node manager on it Is my architecture a problem apart from NN and SNN being on same server? @Geoffrey Shelton Okot Recently, I have added node manager only node and commissioned two data nodes with more space (since older one was running out of space). I had to rebalance the cluster and post this all my hive queries have been running so slow. would be able to point to possible reasons? My master node has HDFS on it but not node manager. Could this be the reason?
... View more
05-04-2018
04:54 PM
Also,In cluster metrics screen, I see: application_1525427506968_1194 dr.who hadoop YARNr oot.users.dr_dot_who Fri May 4 22:20:24 +0550 2018Fri May 4 22:20:25 +0550 2018FAILEDFAILEDN/AN/AN/A
Historyapplication_1525427506968_1193dr.whohadoopYARNroot.users.dr_dot_whoFri May 4 22:20:20 +0550 2018Fri May 4 22:20:23 +0550 2018FAILEDFAILEDN/AN/AN/A
Historyapplication_1525427506968_1192dr.whohadoopYARNroot.users.dr_dot_whoFri May 4 22:20:17 +0550 2018Fri May 4 22:20:19 +0550 2018FAILEDFAILEDN/AN/AN/A
Historyapplication_1525427506968_1191dr.whohadoopYARNroot.users.dr_dot_whoFri May 4 22:20:13 +0550 2018Fri May 4 22:20:15 +0550 2018FAILEDFAILEDN/AN/AN/A
Historyapplication_1525427506968_1190dr.whohadoopYARNroot.users.dr_dot_whoFri May 4 22:20:09 +0550 2018Fri May 4 22:20:11 +0550 2018FAILEDFAILEDN/AN/AN/A
Historyapplication_1525427506968_1189dr.whohadoopYARNroot.users.dr_dot_whoFri May 4 22:20:06 +0550 2018Fri May 4 22:20:08 +0550 2018FAILEDFAILEDN/AN/AN/A
Historyapplication_1525427506968_1188dr.whohadoopYARNroot.users.dr_dot_whoFri May 4 22:20:02 +0550 2018Fri May 4 22:20:05 +0550 2018FAILEDFAILEDN/AN/AN/A
Historyapplication_1525427506968_1187dr.whohadoopYARNroot.users.dr_dot_whoFri May 4 22:19:59 +0550 2018Fri May 4 22:20:01 +0550 2018FAILEDFAILEDN/AN/AN/A
Historyapplication_1525427506968_1186hueCreate TEMPORARY TABL...year(current_date())(Stage-1)MAPREDUCEroot.users.hueFri May 4 22:19:49 +0550 2018Fri May 4 22:20:31 +0550 2018FINISHEDSUCCEEDEDN/A All of the applications that have dr.who as user have failed otherwise those being executed as hue users have run fine. @Harald Berghoff
... View more