Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Specify the file path in STORE command of PIG

Highlighted

Specify the file path in STORE command of PIG

Hi,

I am running a very simple pig program with the following commands as given below.

Program : samplemaxtemp.pig

fulldata = LOAD '/root/kbk/sampletemp.txt' USING PigStorage(',') AS (year:int, tmpr:int, tqlty:int); DESCRIBE fulldata; STORE fulldata INTO '$outputdir'; STORE fulldata INTO 'mymaxtemp';

The above is called from a shell script : simplepig.sh whose contents are as given below

pig -x local -param outputdir=/root/kbk/maxtemp1 samplemaxtemp.pig

Question:

There are two STORE commands . Only in the command that has the directory specified explicitly (mymaxtemp), I am getting the output file. In the other STORE command (wherein the output path is passed as parameter - outputdir), there is no output file.

outputdir(maxtemp1) - has only one file :_SUCCESS

mymaxtemp - has two files (1) _SUCCESS (2). part-m-00000

Please explain

1. Why there is no output file in case of the first STORE command.

2. Can we specify the full file path in the STORE command or not. Or will the output file be always written to the current user directory?s

2 REPLIES 2
Highlighted

Re: Specify the file path in STORE command of PIG

Super Collaborator

I could recreate the same in my sandbox. What I find is that the -x local takes only 'relative_path' as input.

The issue does not exist in the cluster mode. If you run the same script in your cluster (without the -x local) flag, you will get the part files for both store commands.

Highlighted

Re: Specify the file path in STORE command of PIG

Explorer

@Bharath Kumar K all you need to do is add the correct path prefix. In your case, your '-param' should be outputdir=file:///root/kbk/maxtemp1.

You should get to see the expected results.

Don't have an account?
Coming from Hortonworks? Activate your account here