Pig -x tez_local: counters.LimitExceedsException

[#1]

Hi,

How to increase the counter limit for a pig session. -Dmapreduce.job.counters.limit is what I saw for a session. Should I use this before involing pig session in cmd? Could you provide complete command please. Many Thanks!!!

[#2 · Accepted solution]

What's your max counter now, what does the error message say? You can try to increase tez.counters.max, Tez default is 2000, but in the latest version of Ambari it's set to 10000. Also, make sure you are using Pig-0.15 packaged in one of the latest versions of HDP. In Pig-0.14 tez_local mode was unstable. So, you can change tez.counters.max in Ambari or set it per Pig run:

pig -D tez.counters.max=10000 -x tez_local

By the way, what happens if you run your command in Tez mode, on a cluster?