@John Smith ok it was a problem with me, I copied the same file twice. nevermind that issue, I'm still looking. ... View more

@John Smith birthdate also doesn't exist <line 5, column 79> Invalid field projection. Projected field [sensitiveSet::Birthdate] does not exist in schema: sensitiveSet::Row_ID:long,sensitiveSet::name:chararray,sensitiveSet::customerId:chararray,sensitiveSet::Mileage:chararray,sensitiveSet::Fuel_Consumption:chararray,nonSensSet::Row_ID:long,nonSensSet::name:chararray,nonSensSet::customerId:chararray,nonSensSet::Mileage:chararray,nonSensSet::Fuel_Consumption:chararray. Details at logfile: /root/pig-upload/pig_1454078371113.log -- this works grunt> outputSet = foreach outputSet generate nonSensSet::name,nonSensSet::customerId,nonSensSet::Mileage,nonSensSet::Fuel_Consumption; ... View more

@John Smith I highly recommend you develop your scripts in Pig Grunt shell. This is what happened with your script as I was trying to execute it one by one. grunt> sensitiveSet = load '/user/root/Test-20160129-1401822-ttp.avro' using AvroStorage(); grunt> outputSet = join sensitiveSet by Row_ID, nonSensSet by Row_ID; grunt> outputSet = distinct outputSet; grunt> outputSet = foreach outputSet generate nonSensSet::name,nonSensSet::customerId,sensitiveSet::VIN,sensitiveSet::Birthdate,nonSensSet::Mileage,nonSensSet::Fuel_Consumption; 2016-01-29 14:41:59,228 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 1025: <line 5, column 79> Invalid field projection. Projected field [sensitiveSet::VIN] does not exist in schema: sensitiveSet::Row_ID:long,sensitiveSet::name:chararray,sensitiveSet::customerId:chararray,sensitiveSet::Mileage:chararray,sensitiveSet::Fuel_Consumption:chararray,nonSensSet::Row_ID:long,nonSensSet::name:chararray,nonSensSet::customerId:chararray,nonSensSet::Mileage:chararray,nonSensSet::Fuel_Consumption:chararray. Details at logfile: /root/pig-upload/pig_1454078371113.log grunt> describe sensitiveSet; sensitiveSet: {Row_ID: long,name: chararray,customerId: chararray,Mileage: chararray,Fuel_Consumption: chararray} ... View more

@subhash parise do you have a group called hadoop? Can you make sure hbase user belongs to the hadoop group? Confirm it belongs to hdfs group as well. ... View more

@Nayan Paul are you doing this in Sandbox? Can you confirm you're not running into issues described in this thread ... View more

can you paste sample dataset and pig script, I'll try to reproduce sometime today on my machine. It's hard to see the issue from the logs. @John Smith ... View more

@John Smith I'll review and let you know. ... View more

@John Smith AvroStorage may have different package now though I confirmed with javadoc and I was the same as yours, it may be packaged differently in HDP, classpath may differ, don't know for sure. Please accept this answer. ... View more

@John Smith firstly set is a reserved word, change set to another alias, you can also refer to avro simply by AvroStorage no need to write out full package name. If all else fails, add register piggybank.jar command. ... View more