Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Pig script fails to write output on first attempt

Solved Go to solution

Pig script fails to write output on first attempt

Expert Contributor

My pig script (running through Hue) fails to store the results into HDFS on the first attempt. Immediately after attempting to store the data the entire pig script restarts. The script will then complete successfully on the second attempt. Here is my pig script:

 

offers = LOAD '/tmp/file.txt' USING PigStorage AS (tabid:CHARARRAY, offerNum:CHARARRAY);

describe offers;
offers5= LIMIT offers 5;
dump offers5;

STORE offers INTO '/tmp/folder' USING PigStorage();

 

I think my pig script is written poorly, can you identify why the entire script would restart? I can't find anything useful in the logs! Where can I look to try to resolve this issue?

 

1 ACCEPTED SOLUTION

Accepted Solutions
Highlighted

Re: Pig script fails to write output on first attempt

Expert Contributor

I stumbled upon articles written about how to use STORE and DUMP appropriately in a pig script. It seems that I have been using a DUMP and a STORE command in our scripts to output some debugging information. Instead I should only be using the STORE command in our scripts. DUMP is used only for debugging. If you combine the two commands the script will run TWICE!

 

From Apache (http://pig.apache.org/docs/r0.12.0/perf.html#store-dump):

 

Store vs. Dump

With multi-query exection, you want to use STORE to save (persist) your results. You do not want to use DUMP as it will disable multi-query execution and is likely to slow down execution. (If you have included DUMP statements in your scripts for debugging purposes, you should remove them.)

DUMP Example: In this script, because the DUMP command is interactive, the multi-query execution will be disabled and two separate jobs will be created to execute this script. The first job will execute A > B > DUMP while the second job will execute A > B > C > STORE.

A = LOAD 'input' AS (x, y, z);
B = FILTER A BY x > 5;
DUMP B;
C = FOREACH B GENERATE y, z;
STORE C INTO 'output';

STORE Example: In this script, multi-query optimization will kick in allowing the entire script to be executed as a single job. Two outputs are produced: output1 and output2.

A = LOAD 'input' AS (x, y, z);
B = FILTER A BY x > 5;
STORE B INTO 'output1';
C = FOREACH B GENERATE y, z;
STORE C INTO 'output2';	
2 REPLIES 2
Highlighted

Re: Pig script fails to write output on first attempt

Expert Contributor

I stumbled upon articles written about how to use STORE and DUMP appropriately in a pig script. It seems that I have been using a DUMP and a STORE command in our scripts to output some debugging information. Instead I should only be using the STORE command in our scripts. DUMP is used only for debugging. If you combine the two commands the script will run TWICE!

 

From Apache (http://pig.apache.org/docs/r0.12.0/perf.html#store-dump):

 

Store vs. Dump

With multi-query exection, you want to use STORE to save (persist) your results. You do not want to use DUMP as it will disable multi-query execution and is likely to slow down execution. (If you have included DUMP statements in your scripts for debugging purposes, you should remove them.)

DUMP Example: In this script, because the DUMP command is interactive, the multi-query execution will be disabled and two separate jobs will be created to execute this script. The first job will execute A > B > DUMP while the second job will execute A > B > C > STORE.

A = LOAD 'input' AS (x, y, z);
B = FILTER A BY x > 5;
DUMP B;
C = FOREACH B GENERATE y, z;
STORE C INTO 'output';

STORE Example: In this script, multi-query optimization will kick in allowing the entire script to be executed as a single job. Two outputs are produced: output1 and output2.

A = LOAD 'input' AS (x, y, z);
B = FILTER A BY x > 5;
STORE B INTO 'output1';
C = FOREACH B GENERATE y, z;
STORE C INTO 'output2';	

how to split tuple into to different columns in pig

Explorer

Dear All .

 

Below is my data format in pig output file but $1 column contains name :-

 

    name                 dob      gender                 address

(LACTE^MIRAM,19659802,F,4395 NOREY AVE^^ATTIC CITY^N9J^09901)
(JSON^MAW,19820309,M,COUNTRY CLUB DR^^NORELD^NJ^99225)

 

but want like below:-

firstname  lastname  dob     gender         address

(LACTE,MIRAM,19659802,F,4395 NOREY AVE^^ATTIC CITY^N9J^09901)
(JSON,MAW,19820309,M,COUNTRY CLUB DR^^NORELD^NJ^99225)

 

 

 

 

Thanks

HadoopHelp

Don't have an account?
Coming from Hortonworks? Activate your account here