question Re: Pig script execution on Tez gets hung in Support Questions

Pig script execution on Tez gets hung

subrata_bandyop — Sun, 19 Mar 2017 01:26:19 GMT

I'm trying to execute a pig script from Hortontonworks tutorial "Lab 3 - Pig Risk Factor Analysis Introduction". I'm trying with 'Execute on Tez' option checked. But the script shows running for long time and finally I had to kill the job (after more than 30Mins of execution).

I did syntax check before executing and no errors found. I didn't executed the script from shell yet.

Does anybody give me an idea that what might be causing this issue ?

Attached is the log and the script as as below :

a = LOAD 'geolocation' USING org.apache.hive.hcatalog.pig.HCatLoader(); b = FILTER a BY event != 'normal'; c = FOREACH b GENERATE driverid, event, (int) '1' as occurance; d = GROUP c BY driverid; e = FOREACH d GENERATE group as driverid, SUM(c.occurance) as t_occ; g = LOAD 'driver_mileage' USING org.apache.hive.hcatalog.pig.HCatLoader(); h = JOIN e BY driverid, g BY driverid; final_data = foreach h generate $0 as driverid, $1 as events, $3 as totmiles, (float) $3/$1 as riskfactor; STORE final_data INTO 'riskfactor' using org.apache.hive.hcatalog.pig.HCatStorer();

Re: Pig script execution on Tez gets hung

namaheshwari — Sun, 19 Mar 2017 02:37:24 GMT

Can you please post the yarn application logs for application_1489855879213_0002

Use the below command to get the application logs:

 yarn logs -applicationId application_1489855879213_0002

Re: Pig script execution on Tez gets hung

zyang — Sun, 19 Mar 2017 13:31:20 GMT

It looks like Tez AM container cannot be scheduled. Typically this is caused by not enough resource. Please check your cluster capacity to ensure you have enough resource for Tez container.

Re: Pig script execution on Tez gets hung

subrata_bandyop — Sun, 19 Mar 2017 21:59:37 GMT

Thanks for replying. Attaching the yarn log retrieved from shell.

Re: Pig script execution on Tez gets hung

subrata_bandyop — Sun, 19 Mar 2017 22:33:42 GMT

I guess there is some problem with Tez config. As when I execute by checking ''Execute on Tez" it hungs and when unchecked (running as MapReduce) it works fine (attaching the log while executed as MapReduce).pig-script-success-when-exec-as-mapreduce.txt

Re: Pig script execution on Tez gets hung

namaheshwari — Mon, 20 Mar 2017 05:12:41 GMT

There are no explicit errors in the application logs. Can you reproduce the issue?

Re: Pig script execution on Tez gets hung

rbalamohan — Mon, 20 Mar 2017 10:52:33 GMT

>>>

2017-03-18 22:43:27,002 [PigTezLauncher-0] INFO  org.apache.hadoop.yarn.client.api.impl.YarnClientImpl - Submitted application application_1489855879213_0002
2017-03-18 22:43:27,005 [PigTezLauncher-0] INFO  org.apache.tez.client.TezClient - The url to track the Tez Session: http://pc-1.thenet.edu:8088/proxy/application_1489855879213_0002/

>>>

I agree with zyang. Looking at the logs, it appears that there isn't enough capacity to launch. Can you verify the queue configs and cluster capacity?

Re: Pig script execution on Tez gets hung

namaheshwari — Mon, 20 Mar 2017 12:04:38 GMT

@Rajesh Balamohan - The below apache jira talks about the issue when Pig on Tez AM use too much memory on a small cluster:

https://issues.apache.org/jira/browse/PIG-4948

To resolve this, as mentioned in the jira try setting below properties:

Set tez.am.resource.memory.mb to be the same as yarn.scheduler.minimum-allocation-mb the YARN minimum container size.
pig.tez.configure.am.memory to true

Re: Pig script execution on Tez gets hung

subrata_bandyop — Mon, 20 Mar 2017 23:27:38 GMT

@Namit MaheshwariThank you very much for the guidance and appropriate answer. With the above settings the issue got resolved.