I have a Pig UDF which runs great on MR but fails on Tez. I added logs using EvalFunc.getLogger() and also tried System.out.println(). When I run Pig on MR I can find all my log printouts in files obtained using "yarn logs", however when I run the script on Tez, it is producing no log output. I also tried to add "-d DEBUG" but in vain. Any ideas how to produce logs when Pig runs on Tez? [This is HDP-2.3.4 and Pig-0.15]
First make sure you have Tez job application id, and then just yarn logs tool.
To customize Tez log level, you need to set 'tez.am.log.level' and 'tez.task.log.level'.
Also, you may want to wait for Tez job finished(check yarn web console), because pig may keep a Tez session for a while and log of a running application may be incomplete.
Thanks for your reply, I'll try it next week. Re tez.*.log.level, I see in the docs that the default is INFO, and I logged my messages using log.info, but will try to set it explicitly to INFO or DEBUG. I'll also try to get logs with some delay, after a cup of coffee 🙂
That didn't work. I set both tez.*.log.level to DEBUG, and used log.info but could find only output from the calls on the frontend, and nothing from the backend. Also waited for a while but to no avail.