Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

oozie pig action - mapreduce job counters limitexception in tez

Highlighted

oozie pig action - mapreduce job counters limitexception in tez

Expert Contributor

Hi, My pig action called form oozie workflow fails due to below error. Though pig is executing in tez it is taking mapreduce max counters. This is only an issue when run through oozie.

org.apache.hadoop.mapreduce.counters.LimitExceededException: Too many counters: 131 max=130

I think this is a known issue https://issues.apache.org/jira/browse/PIG-4760. My version is 0.15.0

I am trying to manually set the mapreduce.job.counters.max to a higher value. But I am not sure how to do this correctly without modifying mapred-site.xml

I tried following things and nothing worked

1. I set in oozie workflow under <configuration> </configuration> tag in <property></property>

2. I tried adding a modified mapred-site.xml and provided in <job-xml></job-xml> tag in oozie workflow.

3. I also tried setting inside pig using set and also tried to set this in workflow.xml using <configuration> </configuration> and <property> in pig action.

None of this works, so is the way to fix this modify the mapred-site.xml in hadoop home directory? Any suggestions would be helpful.

Thanks!

1 REPLY 1

Re: oozie pig action - mapreduce job counters limitexception in tez

Super Guru

@R M can you go into ambari. click on mapreduce service. change mapreduce.job.counters.max to value above 1000.