Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

CDH 5.3 / 5.5 recharging blocked data to 0%

CDH 5.3 / 5.5 recharging blocked data to 0%

New Contributor

I just installed Ubuntu 5.3 CDH, I respected all the configurations recommended by Cloudera, I installed hadoop + HBase, the problem when I load the data and try to dump the job I still stagnate, and I always reload 0%

OS: Ubuntu 14.04 64

Parcel CDH 5.3 (or 5.5.1)

log: 2016-02-12 04: 06: 33.869 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - HadoopJobId: job_1455246282704_0001 2016-02-12 04: 06: 33.869 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Processing has aliases 2016-02-12 04: 06: 33.869 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - detailed locations: M: a [1,4] C: R: 2016-02-12 04: 06: 34.121 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 0% Complete

3 REPLIES 3

Re: CDH 5.3 / 5.5 recharging blocked data to 0%

Contributor

 Hi Grenloble,

 

Can you provide some additional details regarding the behavior on your cluster, in particular:

 

- How are all the services distributed on your cluster, and how much memory is allocated for each? 
 
- What are the values set for the following properties in YARN?
ApplicationMaster Java Maximum Heap Size
mapreduce.map.java.opts.max.heap
mapreduce.map.memory.mb
mapreduce.reduce.java.opts.max.heap
mapreduce.reduce.memory.mb
yarn.app.mapreduce.am.resource.mb
yarn.nodemanager.resource.memory-mb
yarn.scheduler.minimum-allocation-mb
yarn.scheduler.increment-allocation-mb
yarn.scheduler.maximum-allocation-mb
 
- Are you able to run a simple Pi job successfully?  
For Parcel installs:
$ hadoop jar /opt/cloudera/parcels/CDH/lib/hadoop-0.20-mapreduce/hadoop-examples.jar pi 5 5
 
For package-based installs:
$ hadoop jar /usr/lib/hadoop-0.20-mapreduce/hadoop-examples.jar pi 5 5
 
- Do all Pig jobs fail?
 
- Can you provide the Resource Manager service log snippet along with the Application Master log containing the Pig job that is reportedly hanging?
 

Re: CDH 5.3 / 5.5 recharging blocked data to 0%

New Contributor

Hi Anthony ,

Thank you first for your return, for my installation of cdh 5.5, I used Cloudera-manager-installer.bin and I leave everything by default, here are my yarn parameters;

ApplicationMaster Java Maximum Heap Size =  787.69 MiB
mapreduce.map.java.opts.max.heap = 0 Gib
mapreduce.map.memory.mb = 0 Gib
mapreduce.reduce.java.opts.max.heap = 0 Gib
mapreduce.reduce.memory.mb = 0 Gib
yarn.app.mapreduce.am.resource.mb = 1 Gib
yarn.nodemanager.resource.memory-mb = 2200 Mib
yarn.scheduler.minimum-allocation-mb = 1 Gib
yarn.scheduler.increment-allocation-mb = 512 Mib
yarn.scheduler.maximum-allocation-mb = 2000 Mib

----

 

For Parcel installs:
$ hadoop jar /opt/cloudera/parcels/CDH/lib/hadoop-0.20-mapreduce/hadoop-examples.jar pi 5 5


nadir@localhost:~$ hadoop jar /opt/cloudera/parcels/CDH/lib/hadoop-0.20-mapreduce/hadoop-examples.jar pi 5 5
Number of Maps  = 5
Samples per Map = 5
Wrote input for Map #0
Wrote input for Map #1
Wrote input for Map #2
Wrote input for Map #3
Wrote input for Map #4
Starting Job
16/02/12 19:58:34 INFO client.RMProxy: Connecting to ResourceManager at localhost/127.0.0.1:8032
16/02/12 19:58:39 INFO input.FileInputFormat: Total input paths to process : 5
16/02/12 19:58:41 INFO mapreduce.JobSubmitter: number of splits:5
16/02/12 19:58:45 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1455299522014_0007
16/02/12 19:58:49 INFO impl.YarnClientImpl: Submitted application application_1455299522014_0007
16/02/12 19:58:50 INFO mapreduce.Job: The url to track the job: http://localhost:8088/proxy/application_1455299522014_0007/
16/02/12 19:58:50 INFO mapreduce.Job: Running job: job_1455299522014_0007
16/02/12 20:01:00 INFO mapreduce.Job: Job job_1455299522014_0007 running in uber mode : false
16/02/12 20:01:00 INFO mapreduce.Job:  map 0% reduce 0%



tail -f 4.install-cloudera-manager-server.log
Paramétrage de cloudera-manager-server (5.5.1-1.cm551.p0.8~trusty-cm5) ...
 Adding system startup for /etc/init.d/cloudera-scm-server ...
   /etc/rc0.d/K10cloudera-scm-server -> ../init.d/cloudera-scm-server
   /etc/rc1.d/K10cloudera-scm-server -> ../init.d/cloudera-scm-server
   /etc/rc6.d/K10cloudera-scm-server -> ../init.d/cloudera-scm-server
   /etc/rc2.d/S90cloudera-scm-server -> ../init.d/cloudera-scm-server
   /etc/rc3.d/S90cloudera-scm-server -> ../init.d/cloudera-scm-server
   /etc/rc4.d/S90cloudera-scm-server -> ../init.d/cloudera-scm-server
   /etc/rc5.d/S90cloudera-scm-server -> ../init.d/cloudera-scm-server
Traitement déclenché pour  ureadahead (0.100.0-16) ..

Highlighted

Re: CDH 5.3 / 5.5 recharging blocked data to 0%

Contributor

Hi Grenoble,

 

Thanks for your reply!  It looks like even the simple Pi job is also stalling, so let's take Pig out of the equation and focus on YARN configuration settings to get at least a successful Pi job to run first.   Judging by the information provided so far, it seems that no MR jobs are able to properly run with the current settings and will need to be reviewed for establishing a basic starting point.

 

There's a few pieces still missing to help establish a starting point-- Can you kindly provide the following info?

- How many NodeManagers are in the cluster?

- How much physical memory is available on each NodeManager?

- How many CPU cores are available on each NodeManager?

- What are all the services configured on each NodeManager (i.e. HBase Regionserver, DataNode, etc).

 

 

Don't have an account?
Coming from Hortonworks? Activate your account here