Created 11-24-2015 11:05 AM
Hello ALL,
A common issue , but still i am not able to resolve it .
wc mapred job is stucking at map and reduce 0% .
Below is node manager log :
015-11-23 10:15:18,789 INFO org.apache.spark.network.yarn.YarnShuffleService: Started YARN shuffle service for Spark on port 7337. Authentication is not enabled. Registered executor file is /yarn/nm/registeredExecutors.ldb 2015-11-23 10:15:18,804 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Using ResourceCalculatorPlugin : org.apache.hadoop.yarn.util.LinuxResourceCalculatorPlugin@6b2d2828 2015-11-23 10:15:18,804 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Using ResourceCalculatorProcessTree : null 2015-11-23 10:15:18,804 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Physical memory check enabled: true 2015-11-23 10:15:18,804 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Virtual memory check enabled: false 2015-11-23 10:15:18,815 INFO org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Initialized nodemanager for null: physical-memory=1024 virtual-memory=2151 virtual-cores=2 2015-11-23 10:15:18,816 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.LogAggregationService: org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.LogAggregationService waiting for pending aggregation during exit 2015-11-23 10:15:18,819 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping NodeManager metrics system... 2015-11-23 10:15:18,820 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NodeManager metrics system stopped. 2015-11-23 10:15:18,821 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NodeManager metrics system shutdown complete. 2015-11-23 10:15:18,821 INFO org.apache.hadoop.yarn.server.nodemanager.NodeManager: SHUTDOWN_MSG:
I am not sure whether below is the main reason :
ContainersMonitorImpl: Using ResourceCalculatorProcessTree : null
I updated yarn site and mapred site xml resource mb details ; but getting the same error.
Any suggestions is most appreciated .Also can anyone let m eknow how to do a pseudo cluster refresh ?
Thanks