@Sree Kupp I have tested this extensively and believe there is a bug in SPLIT OTHERWISE against pig EVAL functions (like avg). I will submit a JIRA and update with the number.
But there is a workaround shown below
-------------------------
Here is what I did to replicate the issue and identify a workaround:
Replicate issue
dataset
777576939330699265,0,3
777576939330699261,1,3
777576939330699262,2,2
777576939330699263,3,1
777576939330699264,4,1
script 1 (replicates your issue)
data_input_2 = LOAD 'data_tweets' USING PigStorage(',');
i1 = FOREACH data_input_2 GENERATE $0 as user_id, (int)$1 as user_followers_count, (int)$2 as avl_user_total_retweets;
i2 = GROUP i1 ALL;
i3 = FOREACH i2 GENERATE AVG(i1.user_followers_count) AS avg_user_followers_count, AVG(i1.avl_user_total_retweets) AS avg_avl_user_total_retweets;
SPLIT i1 INTO
top IF(user_followers_count > i3.avg_user_followers_count),
bot IF(user_followers_count < i3.avg_user_followers_count),
med OTHERWISE;
STORE i1 INTO 'tmp/inf_1' USING JsonStorage();
STORE i2 INTO 'tmp/inf_2' USING JsonStorage();
STORE i3 INTO 'tmp/inf_3' USING JsonStorage();
STORE top INTO 'tmp/split_top' USING JsonStorage();
STORE bot INTO 'tmp/split_bot' USING JsonStorage();
STORE med INTO 'tmp/split_med' USING JsonStorage();Results
tmp/inf_1 and tmp/inf_2: results as expected (5 records, correct grouping)
tmp/inf_3: {"avg_user_followers_count":2.0,"avg_avl_user_total_retweets":2.0}tmp/split_top
{"user_id":"777576939330699263","user_followers_count":3,"avl_user_total_retweets":1}
{"user_id":"777576939330699264","user_followers_count":4,"avl_user_total_retweets":1}
tmp/split_med: no records
tmp/split_bot
{"user_id":"777576939330699265","user_followers_count":0,"avl_user_total_retweets":3}
{"user_id":"777576939330699261","user_followers_count":1,"avl_user_total_retweets":3}Workaround
Here I took the average and hard-coded it into the split. This works (but is an inconvenient hack).
script 2 (hard-coding of average, which works)
Everything the same except I used the following for split, based on dump of i3
SPLIT i1 INTO
top IF(user_followers_count > 2.0),
bot IF(user_followers_count < 2.0),
med OTHERWISE;
Results
Results were identical to first script, except:
tmp/split_med:
{"user_id":"777576939330699262","user_followers_count":2,"avl_user_total_retweets":2}Workaround for your script
Find the values in i3 and then hardcode them in the SPLIT statement.
-------------------------
If this answers your question (I hope so 🙂 ), please let me know by accepting the answer; else let me know if there are remaining gaps.
