could you please add that line before STORE outputSet = foreach outputSet generate $0 as (name:chararray) , $1 as (customerId:chararray), $2 as (VIN:chararray) , $3 as (Birthdate:chararray), $4 as (Mileage:chararray) ,$5 as (Fuel_Consumption:chararray); and execute my pig script in your environment? ... View more

i did this: outputSet = foreach outputSet generate $0 as (name:chararray) , $1 as (customerId:chararray), $2 as (VIN:chararray) , $3 as (Birthdate:chararray), $4 as (Mileage:chararray) ,$5 as (Fuel_Consumption:chararray); and command below worked store outputSet into 'avrostorage' using AvroStorage(); Output(s): Successfully stored 100 records in: "file:///root/deploy-3/avrostorage" thats strange, Apparently there is an issue when the relation was describtion as : grunt> describe outputSet; outputSet: {nonSensSet::name: chararray,nonSensSet::customerId: chararray,sensitiveSet::VIN: chararray,sensitiveSet::Birthdate: chararray,nonSensSet::Mileage: chararray,nonSensSet::Fuel_Consumption: chararray} but /AvroStorageSchemaConversionUtilities.java contains code : if (doubleColonsToDoubleUnderscores) { name = name.replace("::", "__"); } There is still the same problem when i try to store using AvroStorage from the script provided: Output(s): Failed to produce result in "/avro-dest/Test-20160129-1401822" ... View more

this is the schema: outputSet: {nonSensSet::name: chararray,nonSensSet::customerId: chararray,sensitiveSet::VIN: chararray,sensitiveSet::Birthdate: chararray,nonSensSet::Mileage: chararray,nonSensSet::Fuel_Consumption: chararray} ... View more

/** * Translates a name in a pig schema to an acceptable Avro name, or * throws an error if the name can't be translated. * @param name The variable name to translate. * @param doubleColonsToDoubleUnderscores Indicates whether to translate * double colons to underscores or throw an error if they are encountered. * @return A name usable by Avro. * @throws IOException If the name is not compatible with Avro. */ private static String toAvroName(String name, final Boolean doubleColonsToDoubleUnderscores) throws IOException { if (name == null) { return null; } if (doubleColonsToDoubleUnderscores) { name = name.replace("::", "__"); } if (name.matches("[A-Za-z_][A-Za-z0-9_]*")) { return name; } else { throw new IOException( "Pig Schema contains a name that is not allowed in Avro"); } } This is the check, and i dont have any characters <> A-Za-z_][A-Za-z0-9_ defined as part of the schema in pig. Btw i dont know why but everything i paste here some CODE/ and click to formate it into code its completely messed up, all newlines are removed... . ... View more

is there anything important in Details at logfile: /root/pig-upload/pig_1454081182813.log ... View more

@Artem Ervits I dont know, i dont do any union, and there no special records (well maybe by union you meant that 2 files needs to be unioned/merged?) That so bad because there essentially doesnt exist any AvroStorage support. What other formats do you mean? ... View more

thats ok 😉 ... View more

i do develop everything in grunt,... you are missing one line in that script nonSensSet = load '/d-spool-dir/Test-20160129-1401822-lake.avro' USING AvroStorage(); ... View more

thats strange... it works for me. grunt> sensitiveSet = load '/t-spool-dir/Test-20160129-1401822-ttp.avro' USING AvroStorage(); grunt> nonSensSet = load '/d-spool-dir/Test-20160129-1401822-lake.avro' USING AvroStorage(); grunt> outputSet = join sensitiveSet by Row_ID, nonSensSet by Row_ID;grunt> outputSet = distinct outputSet; grunt> outputSet = foreach outputSet generate nonSensSet::name,nonSensSet::customerId,sensitiveSet::VIN,sensitiveSet::Birthdate,nonSensSet::Mileage,nonSensSet::Fuel_Consumption;grunt> dump outputSet; ("Kina Buttars",12452346,"WBA32649710927373","1968-08-14",68,10.551) ("Caren Rodman",18853438,"WBA56064572124841","1987-01-24",96,6.779) ("Tierra Bork",89673290,"WBA69315467645466","1958-11-22",52,10.109) ("Thelma Steve",97170856,"WBA73739033913927","1985-12-03",98,5.081) ..... ... View more

@Artem Ervits source files and pig script are included Data , thanks ... View more