Reply
Expert Contributor
Posts: 62
Registered: ‎06-03-2014
Accepted Solution

Pig script fails to write output on first attempt

My pig script (running through Hue) fails to store the results into HDFS on the first attempt. Immediately after attempting to store the data the entire pig script restarts. The script will then complete successfully on the second attempt. Here is my pig script:

 

offers = LOAD '/tmp/file.txt' USING PigStorage AS (tabid:CHARARRAY, offerNum:CHARARRAY);

describe offers;
offers5= LIMIT offers 5;
dump offers5;

STORE offers INTO '/tmp/folder' USING PigStorage();

 

I think my pig script is written poorly, can you identify why the entire script would restart? I can't find anything useful in the logs! Where can I look to try to resolve this issue?

 

Expert Contributor
Posts: 62
Registered: ‎06-03-2014

Re: Pig script fails to write output on first attempt

I stumbled upon articles written about how to use STORE and DUMP appropriately in a pig script. It seems that I have been using a DUMP and a STORE command in our scripts to output some debugging information. Instead I should only be using the STORE command in our scripts. DUMP is used only for debugging. If you combine the two commands the script will run TWICE!

 

From Apache (http://pig.apache.org/docs/r0.12.0/perf.html#store-dump):

 

Store vs. Dump

With multi-query exection, you want to use STORE to save (persist) your results. You do not want to use DUMP as it will disable multi-query execution and is likely to slow down execution. (If you have included DUMP statements in your scripts for debugging purposes, you should remove them.)

DUMP Example: In this script, because the DUMP command is interactive, the multi-query execution will be disabled and two separate jobs will be created to execute this script. The first job will execute A > B > DUMP while the second job will execute A > B > C > STORE.

A = LOAD 'input' AS (x, y, z);
B = FILTER A BY x > 5;
DUMP B;
C = FOREACH B GENERATE y, z;
STORE C INTO 'output';

STORE Example: In this script, multi-query optimization will kick in allowing the entire script to be executed as a single job. Two outputs are produced: output1 and output2.

A = LOAD 'input' AS (x, y, z);
B = FILTER A BY x > 5;
STORE B INTO 'output1';
C = FOREACH B GENERATE y, z;
STORE C INTO 'output2';	
Announcements