Yep, know this "classic" script pretty well. You can find it on @Alan Gates's github account at https://github.com/alanfgates/programmingpig/blob/master/examples/ch6/distinct_symbols.pig. The absolute best way to understand things (and very often answer questions) is to simply run something and observe the behavior. To help with this, I loaded up some simple data that equates to just three distinct trading symbols; SYM1, SYM2 and SYM3. [maria_dev@sandbox 104217]$ hdfs dfs -cat /user/maria_dev/hcc/104217/NYSE_daily.csv
NYSE,SYM1,100,100
NYSE,SYM1,200,200
NYSE,SYM2,100,100
NYSE,SYM2,200,200
NYSE,SYM3,100,100
NYSE,SYM3,200,200
[maria_dev@sandbox 104217]$
I got the script ready to run. [maria_dev@sandbox 104217]$ cat distinct_symbols.pig
daily = load '/user/maria_dev/hcc/104217/NYSE_daily.csv'
USING PigStorage(',')
as (exchange, symbol); -- skip other fields
grpd = group daily by exchange;
describe grpd; -- to show where "daily" below comes from
dump grpd;
uniqcnt = foreach grpd {
sym = daily.symbol;
uniq_sym = distinct sym;
generate group, COUNT(uniq_sym);
};
describe uniqcnt;
dump uniqcnt;
[maria_dev@sandbox 104217]$
The first thing you seem to have trouble with is where does "daily" come from. As this output from the describe and dump of grpd shows, it is made up of two attributes; group and daily (where daily is the contents of of all records from the NYSE which is all records since that's all we have in this file). grpd: { group: bytearray, daily: {(exchange: bytearray,symbol: bytearray)} }
( NYSE, {(NYSE,SYM3),(NYSE,SYM3),(NYSE,SYM2),(NYSE,SYM2),(NYSE,SYM1),(NYSE,SYM1)} ) So, there is only one tuple in the grpd alias (there is only one distinct exchange) that we get to loop through. While inside the loop (for the single row it has) sym ends up being all six rows from (grpd current row).daily.symbol and then uniq_sym ends up being the three distinct rows of which we use to generate the second (unnamed) attribute in uniqcnt. From there, we can describe and dump grpd. uniqcnt: {group: bytearray,long}
(NYSE,3) To help illustrate it more, add the following file to the same input directory in HDFS. [maria_dev@sandbox 104217]$ hdfs dfs -cat /user/maria_dev/hcc/104217/NASDAQ_daily.csv
NASDAQ,HDP,10,1000
NASDAQ,ABC,1,1
NASDAQ,XYZ,1,1
NASDAQ,HDP,10,2000
[maria_dev@sandbox 104217]$
Then change the pig script to just read the directory that has two files now and you'll get this updated output. grpd: {group: bytearray,daily: {(exchange: bytearray,symbol: bytearray)}}
(NYSE,{(NYSE,SYM3),(NYSE,SYM3),(NYSE,SYM2),(NYSE,SYM2),(NYSE,SYM1),(NYSE,SYM1)})
(NASDAQ,{(NASDAQ,HDP),(NASDAQ,XYZ),(NASDAQ,ABC),(NASDAQ,HDP)})
uniqcnt: {group: bytearray,long}
(NYSE,3)
(NASDAQ,3) Hope this helps. Good luck and happy Hadooping!
... View more
