Similar question has been asked before https://community.hortonworks.com/questions/20487/store-output-file-as-3-files-using-pig.html I am going to repeat my findings over @Rich Raposa answer. It's only relevant if it's not for the purposes of the exam. This question was bothering me and I needed to try it out. here's a full script, piggybank is both in pig-client/lib and in pig-client directory REGISTER /usr/hdp/current/pig-client/piggybank.jar;
A = LOAD 'data2' USING PigStorage() as (url, count);
fs -rm -R output;
STORE A INTO 'output' USING org.apache.pig.piggybank.storage.MultiStorage('output', '0'); dataset is 1
2
3
4
5 output is -rw-r--r-- 3 root hdfs 3 2016-03-18 01:51 /user/root/output/1/1-0,000
Found 1 items
-rw-r--r-- 3 root hdfs 3 2016-03-18 01:51 /user/root/output/2/2-0,000
Found 1 items
-rw-r--r-- 3 root hdfs 3 2016-03-18 01:51 /user/root/output/3/3-0,000
Found 1 items
-rw-r--r-- 3 root hdfs 3 2016-03-18 01:51 /user/root/output/4/4-0,000
Found 1 items
-rw-r--r-- 3 root hdfs 3 2016-03-18 01:51 /user/root/output/5/5-0,000
-rw-r--r-- 3 root hdfs 0 2016-03-18 01:51 /user/root/output/_SUCCESS
each file has one line [root@sandbox ~]# hdfs dfs -cat /user/root/output/5/5-0,000
5 in case of @Rich Raposa example the output directory would look like so: [root@sandbox ~]# hdfs dfs -ls output3
Found 6 items
-rw-r--r-- 3 root hdfs 0 2016-03-18 01:59 output3/_SUCCESS
-rw-r--r-- 3 root hdfs 3 2016-03-18 01:59 output3/part-v003-o000-r-00000
-rw-r--r-- 3 root hdfs 3 2016-03-18 01:59 output3/part-v003-o000-r-00001
-rw-r--r-- 3 root hdfs 3 2016-03-18 01:59 output3/part-v003-o000-r-00002
-rw-r--r-- 3 root hdfs 3 2016-03-18 01:59 output3/part-v003-o000-r-00003
-rw-r--r-- 3 root hdfs 3 2016-03-18 01:59 output3/part-v003-o000-r-00004 which means with PARALLEL it creates multiple files within the same directory. In terms of MultiStorage, it created a separate directory and separate file. Additionally with MultiStorage you can pass compression, granted it's bz2, gz, no snappy and delimiter. It's clunky and documentation is not the best but if you need that type of control, it's an option.
... View more
