Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Mapreduce not running in all datanones

Highlighted

Mapreduce not running in all datanones

Explorer

Hi,

 

I have setuped cloudera 5.3 in 3 node cluster (1 Namenode, 2 Datanodes). All are running with same OS, JAVA and other softwares with same configuration (4GB RAM). Loaded some data files, everything is replicated across two datanodes (Replication factor 3) but still have some unreplicated blocks (thats not a concern). When I issue query through HIVE, the job creates around 37mappers to achieve the results, but all are getting executed in single datanode instead two datanodes. I dont know what is happening, but below are the details I captured related to job info.

 

Application OverviewUser:Name:Application Type:Application Tags:State:FinalStatus:Started:Elapsed:Tracking URL:Diagnostics:
hdfs
select sair_str_num,s...ir_upc_num,salesdate(Stage-1)
MAPREDUCE
 
FINISHED
SUCCEEDED
Thu Mar 05 17:15:51 -0500 2015
8mins, 50sec
History
 
 
Application MetricsTotal Resource Preempted:Total Number of Non-AM Containers Preempted:Total Number of AM Containers Preempted:Resource Preempted from Current Attempt:Number of Non-AM Containers Preempted from Current Attempt:Aggregate Resource Allocation:
<memory:0, vCores:0>
0
0
<memory:0, vCores:0>
0
1070822 MB-seconds, 1028 vcore-seconds
ApplicationMasterAttempt Number Start Time Node Logs
1Thu Mar 05 17:15:51 -0500 2015server003.enterprisenet.org:8042logs

 

 

 Logs:

 

Log Type: stderr
Log Length: 2257
Mar 05, 2015 5:16:08 PM com.google.inject.servlet.InternalServletModule$BackwardsCompatibleServletContextProvider get
WARNING: You are attempting to use a deprecated API (specifically, attempting to @Inject ServletContext inside an eagerly created singleton. While we allow this for backwards compatibility, be warned that this MAY have unexpected behavior if you have more than one injector (with ServletModule) running in the same JVM. Please consult the Guice documentation at http://code.google.com/p/google-guice/wiki/Servlets for more information.
Mar 05, 2015 5:16:09 PM com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory register
INFO: Registering org.apache.hadoop.mapreduce.v2.app.webapp.JAXBContextResolver as a provider class
Mar 05, 2015 5:16:09 PM com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory register
INFO: Registering org.apache.hadoop.yarn.webapp.GenericExceptionHandler as a provider class
Mar 05, 2015 5:16:09 PM com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory register
INFO: Registering org.apache.hadoop.mapreduce.v2.app.webapp.AMWebServices as a root resource class
Mar 05, 2015 5:16:09 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.9 09/02/2011 11:17 AM'
Mar 05, 2015 5:16:09 PM com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory getComponentProvider
INFO: Binding org.apache.hadoop.mapreduce.v2.app.webapp.JAXBContextResolver to GuiceManagedComponentProvider with the scope "Singleton"
Mar 05, 2015 5:16:09 PM com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory getComponentProvider
INFO: Binding org.apache.hadoop.yarn.webapp.GenericExceptionHandler to GuiceManagedComponentProvider with the scope "Singleton"
Mar 05, 2015 5:16:10 PM com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory getComponentProvider
INFO: Binding org.apache.hadoop.mapreduce.v2.app.webapp.AMWebServices to GuiceManagedComponentProvider with the scope "PerRequest"
log4j:WARN No appenders could be found for logger (org.apache.hadoop.ipc.Server).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.

 

Log Type: stdout
Log Length: 0

 

Log Type: syslog
Log Length: 528383
Showing 4096 bytes of 528383 total. Click here for the full log.
1425593751461-hdfs-select+sair_str_num%2Cs...ir_upc_num%2Csalesdate%28Stage-1425594282045-37-10-SUCCEEDED-root.hdfs-1425593764268.jhist_tmp
2015-03-05 17:24:42,157 INFO [eventHandlingThread] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Copied to done location: hdfs://server001.enterprisenet.org:8020/user/history/done_intermediate/hdfs/job_1425593189300_0002-1425593751461-hdfs-select+sair_str_num%2Cs...ir_upc_num%2Csalesdate%28Stage-1425594282045-37-10-SUCCEEDED-root.hdfs-1425593764268.jhist_tmp
2015-03-05 17:24:42,162 INFO [eventHandlingThread] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Copying hdfs://server001.enterprisenet.org:8020/user/hdfs/.staging/job_1425593189300_0002/job_1425593189300_0002_1_conf.xml to hdfs://server001.enterprisenet.org:8020/user/history/done_intermediate/hdfs/job_1425593189300_0002_conf.xml_tmp
2015-03-05 17:24:42,197 INFO [eventHandlingThread] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Copied to done location: hdfs://server001.enterprisenet.org:8020/user/history/done_intermediate/hdfs/job_1425593189300_0002_conf.xml_tmp
2015-03-05 17:24:42,210 INFO [eventHandlingThread] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Moved tmp to done: hdfs://server001.enterprisenet.org:8020/user/history/done_intermediate/hdfs/job_1425593189300_0002.summary_tmp to hdfs://server001.enterprisenet.org:8020/user/history/done_intermediate/hdfs/job_1425593189300_0002.summary
2015-03-05 17:24:42,214 INFO [eventHandlingThread] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Moved tmp to done: hdfs://server001.enterprisenet.org:8020/user/history/done_intermediate/hdfs/job_1425593189300_0002_conf.xml_tmp to hdfs://server001.enterprisenet.org:8020/user/history/done_intermediate/hdfs/job_1425593189300_0002_conf.xml
2015-03-05 17:24:42,218 INFO [eventHandlingThread] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Moved tmp to done: hdfs://server001.enterprisenet.org:8020/user/history/done_intermediate/hdfs/job_1425593189300_0002-1425593751461-hdfs-select+sair_str_num%2Cs...ir_upc_num%2Csalesdate%28Stage-1425594282045-37-10-SUCCEEDED-root.hdfs-1425593764268.jhist_tmp to hdfs://server001.enterprisenet.org:8020/user/history/done_intermediate/hdfs/job_1425593189300_0002-1425593751461-hdfs-select+sair_str_num%2Cs...ir_upc_num%2Csalesdate%28Stage-1425594282045-37-10-SUCCEEDED-root.hdfs-1425593764268.jhist
2015-03-05 17:24:42,219 INFO [Thread-184] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Stopped JobHistoryEventHandler. super.stop()
2015-03-05 17:24:42,224 INFO [Thread-184] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Setting job diagnostics to
2015-03-05 17:24:42,225 INFO [Thread-184] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: History url is http://server001.enterprisenet.org:19888/jobhistory/job/job_1425593189300_0002
2015-03-05 17:24:42,234 INFO [Thread-184] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Waiting for application to be successfully unregistered.
2015-03-05 17:24:43,236 INFO [Thread-184] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Final Stats: PendingReds:0 ScheduledMaps:0 ScheduledReds:1 AssignedMaps:0 AssignedReds:1 CompletedMaps:37 CompletedReds:9 ContAlloc:47 ContRel:0 HostLocal:19 RackLocal:18
2015-03-05 17:24:43,239 INFO [Thread-184] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Deleting staging directory hdfs://server001.enterprisenet.org:8020 /user/hdfs/.staging/job_1425593189300_0002
2015-03-05 17:24:43,252 INFO [Thread-184] org.apache.hadoop.ipc.Server: Stopping server on 48930
2015-03-05 17:24:43,262 INFO [IPC Server listener on 48930] org.apache.hadoop.ipc.Server: Stopping IPC Server listener on 48930
2015-03-05 17:24:43,262 INFO [IPC Server Responder] org.apache.hadoop.ipc.Server: Stopping IPC Server Responder
2015-03-05 17:24:43,262 INFO [TaskHeartbeatHandler PingChecker] org.apache.hadoop.mapreduce.v2.app.TaskHeartbeatHandler: TaskHeartbeatHandler thread interrupted

 

Please help me to run mapreduce in both datanodes.