Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Pig on mapreduce mode is stuck on dumping hdfs data in Hortonworks HDP 2.6

Highlighted

Pig on mapreduce mode is stuck on dumping hdfs data in Hortonworks HDP 2.6

New Contributor

Pig on mapreduce mode is stuck on dumping hdfs data in Hortonworks HDP

Hi

@Saurabh Kumar,

@Neeraj Sabharwal

I have some data files in my Hortonworks HDFS location. My requirement is to dump HDFS data in pig shell using pig-mapreduce mode. After loading the file data from HDFS, when trying to dump the data in pig shell using DUMP command, the map reduce job is getting stuck at 0% and not completing the job as well for a long time.

I have 3 ec2 instances and each of them having 4core cpu and 16GB of total Main Memory and below are the configurations for mapred-site and yarn-site.xml.

yarn-site.xml:

yarn.nodemanager.resource.memory-mb=9216

yarn.scheduler.maximum-allocation-mb=9216
yarn.scheduler.minimum-allocation-mb=1024

yarn.nodemanager.vmem-pmem-ratio=2.1

mapred-site.xml:

mapreduce.map.memory.mb=3072
mapreduce.reduce.memory.mb=6144
mapreduce.map.java.opts= -Xmx2457m

mapreduce.reduce.java.opts= -Xmx4915m

Followed the given below steps:
1) Start pig on mapreduce mode:
pig -x mapreduce

2) Load data into pig from a HDFS directory:
mapdata = load 'hdfs://ip-xxx-xx-xx-xx.us-east-2.compute.internal:8020/user/abc/datadir1' as (a:map[chararray]);

3) Print data:
dump mapdata;

After executing the 3rd step getting given below messages on the shell:

2018-10-09 07:25:51,099 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 0% complete
2018-10-09 07:25:51,099 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Running jobs are [job_1539066382468_0147]

Don't have an account?
Coming from Hortonworks? Activate your account here