Store output file as 3 files using pig

[#1]

Hello Friends,

Could any one please let me know how I can store the final output from the pig script as 3 files irrespective of source file/block size?

Thanks,

Satish.

[#2]

@Satish S great question, I just learned something new, you can use MultiStorage() as store function. Refer to for http://stackoverflow.com/questions/9314449/how-to-store-grouped-records-into-multiple-files-with-pig for example and javadoc for explanation of all parameters passed to the function https://pig.apache.org/docs/r0.15.0/api/index.html?org/apache/pig/piggybank/storage/MultiStorage.htm... and of course someone wrote a blog about it http://margus.roo.ee/2014/12/18/apache-pig-how-to-save-output-into-different-places/

[#3]

Hi Artem, thanks for the info.

I am trying use this way, but I am getting some java error.

STORE AFO INTO '/user/hortontest/final_3' USING org.apache.pig.piggybank.storage.MultiStorage('/user/horton/test/final_3','0','none',',');

Error:

ERROR org.apache.pig.PigServer - exception during parsing: Error during parsing. Could not resolve org.apache.pig.piggybank.storage.MultiStorage using imports: [, java.lang., org.apache.pig.builtin., org.apache.pig.impl.builtin.]

[#4]

Did you register the jar? Please confirm and I'll test it.

[#5 · Accepted solution]

@Satish S you need to register piggybank jar. Please read the following https://community.hortonworks.com/questions/8519/register-udf-in-pig.html

[#6]

first statement in your script should be

register  /usr/hdp/current/pig-client/lib/piggybank.jar;

[#7]

here's a full script, piggybank is both in pig-client/lib and in pig-client directory

REGISTER /usr/hdp/current/pig-client/piggybank.jar;
A = LOAD 'data2' USING PigStorage() as (url, count);
fs -rm -R output;
STORE A INTO 'output' USING org.apache.pig.piggybank.storage.MultiStorage('output', '0');

my dataset is

1
2
3
4
5

output would be

-rw-r--r--   3 root hdfs          3 2016-03-18 01:51 /user/root/output/1/1-0,000
Found 1 items
-rw-r--r--   3 root hdfs          3 2016-03-18 01:51 /user/root/output/2/2-0,000
Found 1 items
-rw-r--r--   3 root hdfs          3 2016-03-18 01:51 /user/root/output/3/3-0,000
Found 1 items
-rw-r--r--   3 root hdfs          3 2016-03-18 01:51 /user/root/output/4/4-0,000
Found 1 items
-rw-r--r--   3 root hdfs          3 2016-03-18 01:51 /user/root/output/5/5-0,000
-rw-r--r--   3 root hdfs          0 2016-03-18 01:51 /user/root/output/_SUCCESS

and each file would contain one line

[root@sandbox ~]# hdfs dfs -cat /user/root/output/5/5-0,000
5

in case of @Rich Raposa example

the output directory would look like so:

[root@sandbox ~]# hdfs dfs -ls output3
Found 6 items
-rw-r--r--   3 root hdfs          0 2016-03-18 01:59 output3/_SUCCESS
-rw-r--r--   3 root hdfs          3 2016-03-18 01:59 output3/part-v003-o000-r-00000
-rw-r--r--   3 root hdfs          3 2016-03-18 01:59 output3/part-v003-o000-r-00001
-rw-r--r--   3 root hdfs          3 2016-03-18 01:59 output3/part-v003-o000-r-00002
-rw-r--r--   3 root hdfs          3 2016-03-18 01:59 output3/part-v003-o000-r-00003
-rw-r--r--   3 root hdfs          3 2016-03-18 01:59 output3/part-v003-o000-r-00004

which means with PARALLEL it creates multiple files within the same directory. In terms of MultiStorage, it created a separate directory and separate file. Additionally with MultiStorage you can pass compression, granted it's bz2, gz, no snappy and delimiter. It's clunky and documentation is not the best but if you need that type of control, it's an option.

[#8]

In case someone is searching for this in regards to the Hortonworks Certified Developer exam, the question was asked here also:

https://community.hortonworks.com/questions/22439/where-do-i-get-references-for-piggybank.html

Many of the Pig operators have a PARALLEL option for specifying the number of reducers, which also determines the number of output files. For the intent of the certification exam, using PARALLEL is all you need to accomplish this task, plus it is much simpler than trying to register the piggybank and use a special output class.

[#9]

There is some good information in this thread, but I worry that the discussion about the MultiStorage class in the piggybank is going to seem like it's needed on the HDPCD exam. The MultiStorage class is not a part of the exam objectives. For the exam, you need to know how to use the PARALLEL operator, which if used at the right time in a Pig script can determine the number of output files.

So to summarize: the HDPCD exam does not require the use of MutliStorage, but may require the use of PARALLEL.