Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Store output file as 3 files using pig

Solved Go to solution
Highlighted

Store output file as 3 files using pig

Expert Contributor

Hello Friends,

Could any one please let me know how I can store the final output from the pig script as 3 files irrespective of source file/block size?

Thanks,

Satish.

1 ACCEPTED SOLUTION

Accepted Solutions

Re: Store output file as 3 files using pig

Mentor

@Satish S you need to register piggybank jar. Please read the following https://community.hortonworks.com/questions/8519/register-udf-in-pig.html

8 REPLIES 8

Re: Store output file as 3 files using pig

Mentor

@Satish S great question, I just learned something new, you can use MultiStorage() as store function. Refer to for http://stackoverflow.com/questions/9314449/how-to-store-grouped-records-into-multiple-files-with-pig for example and javadoc for explanation of all parameters passed to the function https://pig.apache.org/docs/r0.15.0/api/index.html?org/apache/pig/piggybank/storage/MultiStorage.htm... and of course someone wrote a blog about it http://margus.roo.ee/2014/12/18/apache-pig-how-to-save-output-into-different-places/

Re: Store output file as 3 files using pig

Expert Contributor

Hi Artem, thanks for the info.

I am trying use this way, but I am getting some java error.

STORE AFO INTO '/user/hortontest/final_3' USING org.apache.pig.piggybank.storage.MultiStorage('/user/horton/test/final_3','0','none',',');

Error:

ERROR org.apache.pig.PigServer - exception during parsing: Error during parsing. Could not resolve org.apache.pig.piggybank.storage.MultiStorage using imports: [, java.lang., org.apache.pig.builtin., org.apache.pig.impl.builtin.]

Re: Store output file as 3 files using pig

Mentor

Did you register the jar? Please confirm and I'll test it.

Re: Store output file as 3 files using pig

Mentor

@Satish S you need to register piggybank jar. Please read the following https://community.hortonworks.com/questions/8519/register-udf-in-pig.html

Re: Store output file as 3 files using pig

Mentor

first statement in your script should be

register  /usr/hdp/current/pig-client/lib/piggybank.jar;

Re: Store output file as 3 files using pig

Mentor

here's a full script, piggybank is both in pig-client/lib and in pig-client directory

REGISTER /usr/hdp/current/pig-client/piggybank.jar;
A = LOAD 'data2' USING PigStorage() as (url, count);
fs -rm -R output;
STORE A INTO 'output' USING org.apache.pig.piggybank.storage.MultiStorage('output', '0');

my dataset is

1
2
3
4
5

output would be

-rw-r--r--   3 root hdfs          3 2016-03-18 01:51 /user/root/output/1/1-0,000
Found 1 items
-rw-r--r--   3 root hdfs          3 2016-03-18 01:51 /user/root/output/2/2-0,000
Found 1 items
-rw-r--r--   3 root hdfs          3 2016-03-18 01:51 /user/root/output/3/3-0,000
Found 1 items
-rw-r--r--   3 root hdfs          3 2016-03-18 01:51 /user/root/output/4/4-0,000
Found 1 items
-rw-r--r--   3 root hdfs          3 2016-03-18 01:51 /user/root/output/5/5-0,000
-rw-r--r--   3 root hdfs          0 2016-03-18 01:51 /user/root/output/_SUCCESS

and each file would contain one line

[root@sandbox ~]# hdfs dfs -cat /user/root/output/5/5-0,000
5

in case of @Rich Raposa example

the output directory would look like so:

[root@sandbox ~]# hdfs dfs -ls output3
Found 6 items
-rw-r--r--   3 root hdfs          0 2016-03-18 01:59 output3/_SUCCESS
-rw-r--r--   3 root hdfs          3 2016-03-18 01:59 output3/part-v003-o000-r-00000
-rw-r--r--   3 root hdfs          3 2016-03-18 01:59 output3/part-v003-o000-r-00001
-rw-r--r--   3 root hdfs          3 2016-03-18 01:59 output3/part-v003-o000-r-00002
-rw-r--r--   3 root hdfs          3 2016-03-18 01:59 output3/part-v003-o000-r-00003
-rw-r--r--   3 root hdfs          3 2016-03-18 01:59 output3/part-v003-o000-r-00004

which means with PARALLEL it creates multiple files within the same directory. In terms of MultiStorage, it created a separate directory and separate file. Additionally with MultiStorage you can pass compression, granted it's bz2, gz, no snappy and delimiter. It's clunky and documentation is not the best but if you need that type of control, it's an option.

Re: Store output file as 3 files using pig

Guru

In case someone is searching for this in regards to the Hortonworks Certified Developer exam, the question was asked here also:

https://community.hortonworks.com/questions/22439/where-do-i-get-references-for-piggybank.html

Many of the Pig operators have a PARALLEL option for specifying the number of reducers, which also determines the number of output files. For the intent of the certification exam, using PARALLEL is all you need to accomplish this task, plus it is much simpler than trying to register the piggybank and use a special output class.

Re: Store output file as 3 files using pig

Guru

There is some good information in this thread, but I worry that the discussion about the MultiStorage class in the piggybank is going to seem like it's needed on the HDPCD exam. The MultiStorage class is not a part of the exam objectives. For the exam, you need to know how to use the PARALLEL operator, which if used at the right time in a Pig script can determine the number of output files.

So to summarize: the HDPCD exam does not require the use of MutliStorage, but may require the use of PARALLEL.