Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Yarn Wordcount Demo : Stuck at MAP 0 and Reduce 0

Highlighted

Yarn Wordcount Demo : Stuck at MAP 0 and Reduce 0

New Contributor

Hello ALL,

 

A common issue , but still i am not able to resolve it .

 

wc mapred job is stucking at map and reduce 0% .

Below is node manager log :

 

015-11-23 10:15:18,789 INFO org.apache.spark.network.yarn.YarnShuffleService: Started YARN shuffle service for Spark on port 7337. Authentication is not enabled.  Registered executor file is /yarn/nm/registeredExecutors.ldb
2015-11-23 10:15:18,804 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:  Using ResourceCalculatorPlugin : org.apache.hadoop.yarn.util.LinuxResourceCalculatorPlugin@6b2d2828
2015-11-23 10:15:18,804 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:  Using ResourceCalculatorProcessTree : null
2015-11-23 10:15:18,804 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Physical memory check enabled: true
2015-11-23 10:15:18,804 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Virtual memory check enabled: false
2015-11-23 10:15:18,815 INFO org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Initialized nodemanager for null: physical-memory=1024 virtual-memory=2151 virtual-cores=2
2015-11-23 10:15:18,816 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.LogAggregationService: org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.LogAggregationService waiting for pending aggregation during exit
2015-11-23 10:15:18,819 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping NodeManager metrics system...
2015-11-23 10:15:18,820 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NodeManager metrics system stopped.
2015-11-23 10:15:18,821 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NodeManager metrics system shutdown complete.
2015-11-23 10:15:18,821 INFO org.apache.hadoop.yarn.server.nodemanager.NodeManager: SHUTDOWN_MSG: 

 

I am not sure whether below is the main reason :

ContainersMonitorImpl:  Using ResourceCalculatorProcessTree : null

 

 

I updated yarn site and mapred site xml resource mb details ; but getting the same error.

Any suggestions is most appreciated .Also can anyone let m eknow how to do a pseudo cluster refresh ?


Thanks

Don't have an account?
Coming from Hortonworks? Activate your account here