I am running a very simple pig program with the following commands as given below.
Program : samplemaxtemp.pig
fulldata = LOAD '/root/kbk/sampletemp.txt' USING PigStorage(',') AS (year:int, tmpr:int, tqlty:int); DESCRIBE fulldata; STORE fulldata INTO '$outputdir'; STORE fulldata INTO 'mymaxtemp';
The above is called from a shell script : simplepig.sh whose contents are as given below
pig -x local -param outputdir=/root/kbk/maxtemp1 samplemaxtemp.pig
There are two STORE commands . Only in the command that has the directory specified explicitly (mymaxtemp), I am getting the output file. In the other STORE command (wherein the output path is passed as parameter - outputdir), there is no output file.
outputdir(maxtemp1) - has only one file :_SUCCESS
mymaxtemp - has two files (1) _SUCCESS (2). part-m-00000
1. Why there is no output file in case of the first STORE command.
2. Can we specify the full file path in the STORE command or not. Or will the output file be always written to the current user directory?s
I could recreate the same in my sandbox. What I find is that the -x local takes only 'relative_path' as input.
The issue does not exist in the cluster mode. If you run the same script in your cluster (without the -x local) flag, you will get the part files for both store commands.