you can write a new script using regex to test this column and throw away bad fields or do it all in one step where you pass the date field to UDF and check for formatting ... View more

@jayaprakash gadi here's my solution, considering that result of join is sum of all fields, then if you have 10 columns in A and 4 columns in B, your result row will 14, you can cherry pick columns 1, 2,3 from A and 11, 12, 13 from B. grunt> fs -cat email_list.csv; 1,Christine,Romero,cromero0@eventbrite.com 2,Sara,Hansen,shansen1@tinypic.com 3,Albert,Rogers,arogers2@marriott.com 4,Kimberly,Morrison,kmorrison3@irs.gov 5,Eugene,Baker,ebaker4@cbslocal.com 6,Ann,Alexander,aalexander5@hhs.gov 7,Kathleen,Reed,kreed6@youtu.be 8,Todd,Scott,tscott7@deliciousdays.com 9,Sharon,Mccoy,smccoy8@nature.com 10,Evelyn,Rice,erice9@narod.ru grunt> fs -cat gender_list.csv; 1,Christine,Romero,Female 2,Sara,Hansen,Female 3,Albert,Rogers,Male 4,Kimberly,Morrison,Female 5,Eugene,Baker,Male 6,Ann,Alexander,Female 7,Kathleen,Reed,Female 8,Todd,Scott,Male 9,Sharon,Mccoy,Female 10,Evelyn,Rice,Female grunt> A = load 'email_list.csv' using PigStorage(','); grunt> B = load 'gender_list.csv' using PigStorage(','); grunt> C = join A by ($0, $1, $2), B by ($0, $1, $2); grunt> dump C; (1,Christine,Romero,cromero0@eventbrite.com,1,Christine,Romero,Female) (10,Evelyn,Rice,erice9@narod.ru,10,Evelyn,Rice,Female) (2,Sara,Hansen,shansen1@tinypic.com,2,Sara,Hansen,Female) (3,Albert,Rogers,arogers2@marriott.com,3,Albert,Rogers,Male) (4,Kimberly,Morrison,kmorrison3@irs.gov,4,Kimberly,Morrison,Female) (5,Eugene,Baker,ebaker4@cbslocal.com,5,Eugene,Baker,Male) (6,Ann,Alexander,aalexander5@hhs.gov,6,Ann,Alexander,Female) (7,Kathleen,Reed,kreed6@youtu.be,7,Kathleen,Reed,Female) (8,Todd,Scott,tscott7@deliciousdays.com,8,Todd,Scott,Male) (9,Sharon,Mccoy,smccoy8@nature.com,9,Sharon,Mccoy,Female) grunt> D = foreach C generate $0, $1, $2, $3, $7; grunt> dump D; (1,Christine,Romero,cromero0@eventbrite.com,Female) (10,Evelyn,Rice,erice9@narod.ru,Female) (2,Sara,Hansen,shansen1@tinypic.com,Female) (3,Albert,Rogers,arogers2@marriott.com,Male) (4,Kimberly,Morrison,kmorrison3@irs.gov,Female) (5,Eugene,Baker,ebaker4@cbslocal.com,Male) (6,Ann,Alexander,aalexander5@hhs.gov,Female) (7,Kathleen,Reed,kreed6@youtu.be,Female) (8,Todd,Scott,tscott7@deliciousdays.com,Male) (9,Sharon,Mccoy,smccoy8@nature.com,Female) ... View more

@Koti P I don't see a problem with your code, I'm able to execute your code using HDP 2.4 Sandbox temp = LOAD 'abc.txt' using PigStorage(';','-tagFile'); test = RANK temp; DUMP test; my abc.txt looks like so David,1,N Tete,2,N Ranjit,3,M Ranjit,3,P David,4,Q David,4,Q Jillian,8,Q JaePak,7,Q Michael,8,T Jillian,8,Q Jose,10,V and my output looks like so: (1,abc.txt,David,1,N) (2,abc.txt,Tete,2,N) (3,abc.txt,Ranjit,3,M) (4,abc.txt,Ranjit,3,P) (5,abc.txt,David,4,Q) (6,abc.txt,David,4,Q) (7,abc.txt,Jillian,8,Q) (8,abc.txt,JaePak,7,Q) (9,abc.txt,Michael,8,T) (10,abc.txt,Jillian,8,Q) (11,abc.txt,Jose,10,V) I used tez as executing engine pig -x tez ... View more