question Re: Store output file as 3 files using pig in Archives of Support Questions (Read Only)

Store output file as 3 files using pig

SatishS — Tue, 01 Mar 2016 23:25:31 GMT

Hello Friends,

Could any one please let me know how I can store the final output from the pig script as 3 files irrespective of source file/block size?

Thanks,

Satish.

Re: Store output file as 3 files using pig

aervits — Tue, 01 Mar 2016 23:40:28 GMT

@Satish S great question, I just learned something new, you can use MultiStorage() as store function. Refer to for http://stackoverflow.com/questions/9314449/how-to-store-grouped-records-into-multiple-files-with-pig for example and javadoc for explanation of all parameters passed to the function https://pig.apache.org/docs/r0.15.0/api/index.html?org/apache/pig/piggybank/storage/MultiStorage.html and of course someone wrote a blog about it http://margus.roo.ee/2014/12/18/apache-pig-how-to-save-output-into-different-places/

Re: Store output file as 3 files using pig

SatishS — Wed, 02 Mar 2016 00:28:10 GMT

Hi Artem, thanks for the info.

I am trying use this way, but I am getting some java error.

STORE AFO INTO '/user/hortontest/final_3' USING org.apache.pig.piggybank.storage.MultiStorage('/user/horton/test/final_3','0','none',',');

Error:

ERROR org.apache.pig.PigServer - exception during parsing: Error during parsing. Could not resolve org.apache.pig.piggybank.storage.MultiStorage using imports: [, java.lang., org.apache.pig.builtin., org.apache.pig.impl.builtin.]

Re: Store output file as 3 files using pig

aervits — Wed, 02 Mar 2016 00:42:31 GMT

@Satish S you need to register piggybank jar. Please read the following https://community.hortonworks.com/questions/8519/register-udf-in-pig.html

Re: Store output file as 3 files using pig

aervits — Wed, 02 Mar 2016 00:44:15 GMT

first statement in your script should be

register  /usr/hdp/current/pig-client/lib/piggybank.jar;

Re: Store output file as 3 files using pig

aervits — Fri, 11 Mar 2016 19:54:20 GMT

Did you register the jar? Please confirm and I'll test it.

Re: Store output file as 3 files using pig

rich1 — Fri, 11 Mar 2016 20:13:09 GMT

In case someone is searching for this in regards to the Hortonworks Certified Developer exam, the question was asked here also:

https://community.hortonworks.com/questions/22439/where-do-i-get-references-for-piggybank.html

Many of the Pig operators have a PARALLEL option for specifying the number of reducers, which also determines the number of output files. For the intent of the certification exam, using PARALLEL is all you need to accomplish this task, plus it is much simpler than trying to register the piggybank and use a special output class.

Re: Store output file as 3 files using pig

aervits — Fri, 18 Mar 2016 09:06:46 GMT

here's a full script, piggybank is both in pig-client/lib and in pig-client directory

REGISTER /usr/hdp/current/pig-client/piggybank.jar;
A = LOAD 'data2' USING PigStorage() as (url, count);
fs -rm -R output;
STORE A INTO 'output' USING org.apache.pig.piggybank.storage.MultiStorage('output', '0');

my dataset is

output would be

-rw-r--r--   3 root hdfs          3 2016-03-18 01:51 /user/root/output/1/1-0,000
Found 1 items
-rw-r--r--   3 root hdfs          3 2016-03-18 01:51 /user/root/output/2/2-0,000
Found 1 items
-rw-r--r--   3 root hdfs          3 2016-03-18 01:51 /user/root/output/3/3-0,000
Found 1 items
-rw-r--r--   3 root hdfs          3 2016-03-18 01:51 /user/root/output/4/4-0,000
Found 1 items
-rw-r--r--   3 root hdfs          3 2016-03-18 01:51 /user/root/output/5/5-0,000
-rw-r--r--   3 root hdfs          0 2016-03-18 01:51 /user/root/output/_SUCCESS

and each file would contain one line

[root@sandbox ~]# hdfs dfs -cat /user/root/output/5/5-0,000
5

in case of @Rich Raposa example

the output directory would look like so:

[root@sandbox ~]# hdfs dfs -ls output3
Found 6 items
-rw-r--r--   3 root hdfs          0 2016-03-18 01:59 output3/_SUCCESS
-rw-r--r--   3 root hdfs          3 2016-03-18 01:59 output3/part-v003-o000-r-00000
-rw-r--r--   3 root hdfs          3 2016-03-18 01:59 output3/part-v003-o000-r-00001
-rw-r--r--   3 root hdfs          3 2016-03-18 01:59 output3/part-v003-o000-r-00002
-rw-r--r--   3 root hdfs          3 2016-03-18 01:59 output3/part-v003-o000-r-00003
-rw-r--r--   3 root hdfs          3 2016-03-18 01:59 output3/part-v003-o000-r-00004

which means with PARALLEL it creates multiple files within the same directory. In terms of MultiStorage, it created a separate directory and separate file. Additionally with MultiStorage you can pass compression, granted it's bz2, gz, no snappy and delimiter. It's clunky and documentation is not the best but if you need that type of control, it's an option.

Re: Store output file as 3 files using pig

rich1 — Fri, 18 Mar 2016 11:05:04 GMT

There is some good information in this thread, but I worry that the discussion about the MultiStorage class in the piggybank is going to seem like it's needed on the HDPCD exam. The MultiStorage class is not a part of the exam objectives. For the exam, you need to know how to use the PARALLEL operator, which if used at the right time in a Pig script can determine the number of output files.

So to summarize: the HDPCD exam does not require the use of MutliStorage, but may require the use of PARALLEL.