Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Debugging Pig UDFs on Tez

Highlighted

Debugging Pig UDFs on Tez

I have a Pig UDF which runs great on MR but fails on Tez. I added logs using EvalFunc.getLogger() and also tried System.out.println(). When I run Pig on MR I can find all my log printouts in files obtained using "yarn logs", however when I run the script on Tez, it is producing no log output. I also tried to add "-d DEBUG" but in vain. Any ideas how to produce logs when Pig runs on Tez? [This is HDP-2.3.4 and Pig-0.15]

6 REPLIES 6

Re: Debugging Pig UDFs on Tez

Explorer

First make sure you have Tez job application id, and then just yarn logs tool.

To customize Tez log level, you need to set 'tez.am.log.level' and 'tez.task.log.level'.

Also, you may want to wait for Tez job finished(check yarn web console), because pig may keep a Tez session for a while and log of a running application may be incomplete.

Highlighted

Re: Debugging Pig UDFs on Tez

Thanks for your reply, I'll try it next week. Re tez.*.log.level, I see in the docs that the default is INFO, and I logged my messages using log.info, but will try to set it explicitly to INFO or DEBUG. I'll also try to get logs with some delay, after a cup of coffee :-)

Highlighted

Re: Debugging Pig UDFs on Tez

That didn't work. I set both tez.*.log.level to DEBUG, and used log.info but could find only output from the calls on the frontend, and nothing from the backend. Also waited for a while but to no avail.

Highlighted

Re: Debugging Pig UDFs on Tez

Super Guru

Use the Ambari Pig View, it's much better for running and debugging Pig scripts

Highlighted

Re: Debugging Pig UDFs on Tez

Not applicable, since the views are not installed, and the team doesn't have admin permissions.

Highlighted

Re: Debugging Pig UDFs on Tez

Mentor

Heres a pretty extensive list of options to debug, including Tez and options to run with PigUnit unit testing utility https://pig.apache.org/docs/r0.16.0/test.html

Don't have an account?
Coming from Hortonworks? Activate your account here