Hi, My pig action called form oozie workflow fails due to below error. Though pig is executing in tez it is taking mapreduce max counters. This is only an issue when run through oozie.
org.apache.hadoop.mapreduce.counters.LimitExceededException: Too many counters: 131 max=130
I think this is a known issue https://issues.apache.org/jira/browse/PIG-4760. My version is 0.15.0
I am trying to manually set the mapreduce.job.counters.max to a higher value. But I am not sure how to do this correctly without modifying mapred-site.xml
I tried following things and nothing worked
1. I set in oozie workflow under <configuration> </configuration> tag in <property></property>
2. I tried adding a modified mapred-site.xml and provided in <job-xml></job-xml> tag in oozie workflow.
3. I also tried setting inside pig using set and also tried to set this in workflow.xml using <configuration> </configuration> and <property> in pig action.
None of this works, so is the way to fix this modify the mapred-site.xml in hadoop home directory? Any suggestions would be helpful.
@R M can you go into ambari. click on mapreduce service. change mapreduce.job.counters.max to value above 1000.