There are a couple of issues that I can see with your script. Your first statement that reads the data from the file. my_data = LOAD 'customers.txt'usingPigStorage()as(name:chararray, age:int, eye_color:chararray, height:int); You used PigStorage() method without any parameter. If you don't pass any parameter to this method, it will consider TAB as the delimiter. And looking at your data file, you have a comma as the delimiter. So your LOAD statement should look like follows. my_data = LOAD 'customers.txt'usingPigStorage(',')as(name:chararray, age:int, eye_color:chararray, height:int); This actually is not the problem that you are facing though. In your last statement, where you are creating the final_data relation, you referred to your columns as SUM(brown_eyes) as num_brown_eyes,SUM(blue_eyes) as num_blue_eyes SUM(green_eyes) as num_green_eyes This is incorrect. A describe statement should explain the schema to you. A describe statement should explain the schema for you. grunt> describe by_age;
by_age: {group: int,my_data: {(name: chararray,age: int,eye_color: chararray,height: int)}} You can see that all the columns are clubbed inside my_data column. So the reference to these columns should be made as mentioned below. SUM(my_data.brown_eyes) as num_brown_eyes,SUM(my_data.blue_eyes) as num_blue_eyesSUM(my_data.green_eyes) as num_green_eyes The same way you have used my_data.height in your code. So you final generate statement should look like as follows. final_data = FOREACH by_age GENERATE groupas age, COUNT(my_data)as num_people, AVG(my_data.height)as avg_height, SUM(my_data.brown_eyes)as num_brown_eyes, SUM(my_data.blue_eyes)as num_blue_eyes, SUM(my_data.green_eyes)as num_green_eyes; All in all, your complete script should look like as shown below. my_data = LOAD 'customers.txt'usingPigStorage(',')as(name:chararray, age:int, eye_color:chararray, height:int);
my_data = FOREACH my_data GENERATE name, age, height,(eye_color =='brown'?1:0) AS brown_eyes,(eye_color =='blue'?1:0) AS blue_eyes,(eye_color =='green'?1:0) AS green_eyes;
by_age =group my_data by age;
final_data = FOREACH by_age GENERATE groupas age, COUNT(my_data)as num_people, AVG(my_data.height)as avg_height, SUM(my_data.brown_eyes)as num_brown_eyes, SUM(my_data.blue_eyes)as num_blue_eyes, SUM(my_data.green_eyes)as num_green_eyes; Now you know what were the issues, you will be able to run your script and also prevent those "typos" in future! Happy coding!
... View more
