Reply
Jay
New Contributor
Posts: 3
Registered: ‎06-09-2014

simple pi calculation Job or any job is failing in newly CDH 5.2 configured cluster

Simple pi calculation Job or any job is failing in newly CDH 5.2 configured cluster.
All services are running perfectly.

 

[root@node6 hadoop-mapreduce]# hadoop jar hadoop-mapreduce-examples.jar pi 1 1
Number of Maps = 1
Samples per Map = 1
Wrote input for Map #0
Starting Job
14/11/13 05:30:09 INFO client.RMProxy: Connecting to ResourceManager at mynode6.cdh.com/10.1.80.6:8032
14/11/13 05:30:10 INFO input.FileInputFormat: Total input paths to process : 1
14/11/13 05:30:10 INFO mapreduce.JobSubmitter: number of splits:1
14/11/13 05:30:10 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1415873852760_0001
14/11/13 05:30:10 INFO impl.YarnClientImpl: Submitted application application_1415873852760_0001
14/11/13 05:30:10 INFO mapreduce.Job: The url to track the job: http://mynode6.cdh.com:8088/proxy/application_1415873852760_0001/
14/11/13 05:30:10 INFO mapreduce.Job: Running job: job_1415873852760_0001

 

 

It hangs after transmitting above message to console.

Resource manger error log shows below message:

 

INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler: Added node mynode6.cdh.com:8041 cluster capacity: <memory:1450, vCores:4>
2014-11-13 05:04:06,865 ERROR org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: RECEIVED SIGNAL 15: SIGTERM
2014-11-13 05:04:06,873 INFO org.mortbay.log: Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@mynode6.cdh.com:8088
2014-11-13 05:04:06,973 INFO org.apache.hadoop.ipc.Server: Stopping server on 8032
2014-11-13 05:04:06,976 INFO org.apache.hadoop.ipc.Server: Stopping IPC Server listener on 8032
2014-11-13 05:04:06,978 INFO org.apache.hadoop.ipc.Server: Stopping IPC Server Responder
2014-11-13 05:04:06,978 INFO org.apache.hadoop.ipc.Server: Stopping server on 8033
2014-11-13 05:04:06,979 INFO org.apache.hadoop.ipc.Server: Stopping IPC Server listener on 8033
2014-11-13 05:04:06,979 INFO org.apache.hadoop.ipc.Server: Stopping IPC Server Responder
2014-11-13 05:04:06,979 INFO org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Transitioning to standby state
2014-11-13 05:04:06,979 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping ResourceManager metrics system...
2014-11-13 05:04:06,980 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: ResourceManager metrics system stopped.
2014-11-13 05:04:06,981 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: ResourceManager metrics system shutdown complete.
2014-11-13 05:04:06,981 INFO org.apache.hadoop.yarn.event.AsyncDispatcher: AsyncDispatcher is draining to stop, igonring any new events.
2014-11-13 05:04:06,981 WARN org.apache.hadoop.yarn.server.resourcemanager.amlauncher.ApplicationMasterLauncher: org.apache.hadoop.yarn.server.resourcemanager.amlauncher.ApplicationMasterLauncher$LauncherThread interrupted. Returning.
2014-11-13 05:04:06,981 INFO org.apache.hadoop.ipc.Server: Stopping server on 8030
2014-11-13 05:04:06,985 INFO org.apache.hadoop.ipc.Server: Stopping IPC Server listener on 8030
2014-11-13 05:04:06,986 INFO org.apache.hadoop.ipc.Server: Stopping server on 8031
2014-11-13 05:04:06,986 INFO org.apache.hadoop.ipc.Server: Stopping IPC Server Responder
2014-11-13 05:04:06,990 INFO org.apache.hadoop.ipc.Server: Stopping IPC Server listener on 8031
2014-11-13 05:04:06,991 INFO org.apache.hadoop.ipc.Server: Stopping IPC Server Responder
2014-11-13 05:04:06,992 INFO org.apache.hadoop.yarn.util.AbstractLivelinessMonitor: NMLivelinessMonitor thread interrupted
2014-11-13 05:04:06,992 ERROR org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Returning, interrupted : java.lang.InterruptedException
2014-11-13 05:04:06,992 WARN org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler: Update thread interrupted. Exiting.
2014-11-13 05:04:06,993 WARN org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler: Continuous scheduling thread interrupted. Exiting.
java.lang.InterruptedException: sleep interrupted
at java.lang.Thread.sleep(Native Method)
at org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler$ContinuousSchedulingThread.run(FairScheduler.java:281)
2014-11-13 05:04:07,009 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.AllocationFileLoaderService: Interrupted while waiting to reload alloc configuration
2014-11-13 05:04:07,009 INFO org.apache.hadoop.yarn.event.AsyncDispatcher: AsyncDispatcher is draining to stop, igonring any new events.
2014-11-13 05:04:07,009 INFO org.apache.hadoop.yarn.event.AsyncDispatcher: AsyncDispatcher is draining to stop, igonring any new events.
2014-11-13 05:04:07,009 INFO org.apache.hadoop.yarn.event.AsyncDispatcher: AsyncDispatcher is draining to stop, igonring any new events.
2014-11-13 05:04:07,010 INFO org.apache.hadoop.yarn.event.AsyncDispatcher: AsyncDispatcher is draining to stop, igonring any new events.
2014-11-13 05:04:07,010 INFO org.apache.hadoop.yarn.event.AsyncDispatcher: AsyncDispatcher is draining to stop, igonring any new events.
2014-11-13 05:04:07,010 INFO org.apache.hadoop.yarn.event.AsyncDispatcher: AsyncDispatcher is draining to stop, igonring any new events.
2014-11-13 05:04:07,013 INFO org.apache.hadoop.yarn.event.AsyncDispatcher: AsyncDispatcher is draining to stop, igonring any new events.
2014-11-13 05:04:07,013 INFO org.apache.hadoop.yarn.event.AsyncDispatcher: AsyncDispatcher is draining to stop, igonring any new events.
2014-11-13 05:04:07,013 INFO org.apache.hadoop.yarn.event.AsyncDispatcher: AsyncDispatcher is draining to stop, igonring any new events.
2014-11-13 05:04:07,014 INFO org.apache.hadoop.yarn.event.AsyncDispatcher: AsyncDispatcher is draining to stop, igonring any new events.
2014-11-13 05:04:07,014 INFO org.apache.hadoop.yarn.util.AbstractLivelinessMonitor: AMLivelinessMonitor thread interrupted
2014-11-13 05:04:07,014 INFO org.apache.hadoop.yarn.util.AbstractLivelinessMonitor: org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.ContainerAllocationExpirer thread interrupted
2014-11-13 05:04:07,014 ERROR org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager: ExpiredTokenRemover received java.lang.InterruptedException: sleep interrupted
2014-11-13 05:04:07,014 INFO org.apache.hadoop.yarn.util.AbstractLivelinessMonitor: AMLivelinessMonitor thread interrupted
2014-11-13 05:04:07,015 INFO org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Transitioned to standby state
2014-11-13 05:04:07,015 INFO org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: SHUTDOWN_MSG:

 

 

Any Idea on this ?

Highlighted
Jay
New Contributor
Posts: 3
Registered: ‎06-09-2014

Re: simple pi calculation Job or any job is failing in newly CDH 5.2 configured cluster

Adding log :

 

 


2014-11-13 04:54:53,098 INFO org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.yarn.server.resourcemanager.RMFatalEventType for class org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMFatalEventDispatcher
2014-11-13 04:54:53,128 WARN com.cloudera.cmf.event.publish.EventStorePublisherWithRetry: Failed to publish event: SimpleEvent{attributes={ROLE_TYPE=[RESOURCEMANAGER], CATEGORY=[LOG_MESSAGE], ROLE=[yarn-RESOURCEMANAGER-ef02b4962ec1793c147acc1f976532d3], SEVERITY=[IMPORTANT], SERVICE=[yarn], HOST_IDS=[9db07d08-be36-4563-9740-9ee7dfb90605], SERVICE_TYPE=[YARN], LOG_LEVEL=[WARN], HOSTS=[node6.cdh.com], EVENTCODE=[EV_LOG_EVENT]}, content=java.io.BufferedInputStream@6ca79f01:an attempt to override final parameter: hadoop.ssl.require.client.cert; Ignoring., timestamp=1415872493017}
2014-11-13 04:54:53,329 INFO org.apache.hadoop.yarn.server.resourcemanager.security.NMTokenSecretManagerInRM: NMTokenKeyRollingInterval: 86400000ms and NMTokenKeyActivationDelay: 900000ms
2014-11-13 04:54:53,332 INFO org.apache.hadoop.yarn.server.resourcemanager.security.RMContainerTokenSecretManager: ContainerTokenKeyRollingInterval: 86400000ms and ContainerTokenKeyActivationDelay: 900000ms
2014-11-13 04:54:53,337 INFO org.apache.hadoop.yarn.server.resourcemanager.security.AMRMTokenSecretManager: AMRMTokenKeyRollingInterval: 86400000ms and AMRMTokenKeyActivationDelay: 900000 ms
2014-11-13 04:54:53,366 INFO org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.yarn.server.resourcemanager.recovery.RMStateStoreEventType for class org.apache.hadoop.yarn.server.resourcemanager.recovery.RMStateStore$ForwardingEventHandler
2014-11-13 04:54:53,534 INFO org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.yarn.server.resourcemanager.NodesListManagerEventType for class org.apache.hadoop.yarn.server.resourcemanager.NodesListManager
2014-11-13 04:54:53,534 INFO org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Using Scheduler: org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler
2014-11-13 04:54:53,583 INFO org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.yarn.server.resourcemanager.scheduler.event.SchedulerEventType for class org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$SchedulerEventDispatcher
2014-11-13 04:54:53,584 INFO org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppEventType for class org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$ApplicationEventDispatcher
2014-11-13 04:54:53,585 INFO org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptEventType for class org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$ApplicationAttemptEventDispatcher
2014-11-13 04:54:53,586 INFO org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class

Posts: 1,567
Kudos: 289
Solutions: 240
Registered: ‎07-31-2013

Re: simple pi calculation Job or any job is failing in newly CDH 5.2 configured cluster

A hanging pre-progress job indicates that your cluster likely isn't configured right in terms of memory and CPU resources under YARN. Please take a look at this guide to ensure you have configured your YARN service properly in CM: http://www.cloudera.com/content/cloudera/en/documentation/core/latest/topics/cdh_ig_mapreduce_to_yar... especially the NodeManager Container Memory and CPUs settings.
Backline Customer Operations Engineer
Announcements