ah i already did ... my question was why its there ... when i use local mode its not there .. anyway there is no reply from anyone behind avrostorage... thats pretty odd. ... View more

sure but input data contains all the field, so my question is why it generates [null] as part of the datatype. Also still no luck with https://issues.apache.org/jira/browse/PIG-4793 ... View more

one more important observation, when i dump data into avro using store outputSet into 'avrostorage' using AvroStorage(); the schema inside avro file looks like: {"type":"record","name":"pig_output","fields":[{"name":"name","type":["null","string"]},{"name":"customerId","type":["null","string"]},{"name":"VIN","type":["null","string"]},{"name":"Birthdate","type":["null","string"]},{"name":"Mileage","type":["null","string"]},{"name":"Fuel_Consumption","type":["null","string"]}]} Why each field contains null? ... View more

is there any update on this? ... View more

well i cant live with that workaround, thats the problem. what i HCC? ... View more

yes, works for me also, but when i use STORE outputSet INTO '/avro-dest/Test-20160129-1401822' USING org.apache.pig.piggybank.storage.avro.AvroStorage and i define schema as part of the AvroStorage( schema ) ... it doesnt work ;-((( ... View more

Ok, i added the line outputSet = foreach outputSet generate $0 as (name:chararray) , $1 as (customerId:chararray), $2 as (VIN:chararray) , $3 as (Birthdate:chararray), $4 as (Mileage:chararray) ,$5 as (Fuel_Consumption:chararray); and successfully created output avro file using: store outputSet into 'avrostorage' using AvroStorage(); When i try to store output file using code below it is failing /10.0.1.47:8050 2016-01-29 17:24:39,600 [main] ERROR org.apache.pig.tools.pigstats.mapreduce.MRPigStatsUtil - 1 map reduce job(s) failed! at this point i clearly have no idea what else i can do. STORE outputSet INTO '/avro-dest/Test-20160129-1401822' USING org.apache.pig.piggybank.storage.avro.AvroStorage('no_schema_check', 'schema', '{"type":"record","name":"test","fields":[{"name":"name","type":"string","title":"Customer name","description":"non Surrogate Key for joining files on the BDP","DataOwner":"Bank","ValidityDate":"2015.12.22","ValidityOption":"Delete","DataSensitivityLevel":"0","FieldPosition":"1"},{"name":"customerId","type":"string","title":"customer Id","description":"non sensitive field of customer Id","DataOwner":"Bank","ValidityDate":"2015.12.22","ValidityOption":"Retain","DataSensitivityLevel":"0","FieldPosition":"2"},{"name":"VIN","type":"string","title":"Customer VIN","description":"Customer VIN","DataOwner":"Bank","ValidityDate":"2015.12.22","ValidityOption":"Delete","DataSensitivityLevel":"1","FieldPosition":"3"},{"name":"Birthdate","type":"string","title":"Customer birthdate","description":"Customer birthdate","DataOwner":"Bank","ValidityDate":"2015.12.22","ValidityOption":"Delete","DataSensitivityLevel":"1","FieldPosition":"4"},{"name":"Mileage","type":"string","title":"Customer mileage","description":"Customer mileage","DataOwner":"Bank","ValidityDate":"2015.12.22","ValidityOption":"Delete","DataSensitivityLevel":"0","FieldPosition":"5"},{"name":"Fuel_Consumption","type":"string","title":"Customer fule consumption","description":"Customer fuel consumption","DataOwner":"Bank","ValidityDate":"2015.12.22","ValidityOption":"Delete","DataSensitivityLevel":"0","FieldPosition":"6"}]}'); ... View more

ops sorry my fault ... i dont have that source stored in HDFS ... time to stop debugging for today -) ... View more

I dont know what happened but i cant load any avro file in mapreduce mode ... grunt> sensitiveSet = load '/t-spool-dir/Test-20160129-1401822-ttp.avro' USING AvroStorage(); 2016-01-29 17:06:00,668 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 1200: null Details at logfile: /tmp/hsperfdata_hdfs/pig_1454087102249.log Pig Stack Trace --------------- ERROR 1200: null Failed to parse: null at org.apache.pig.parser.QueryParserDriver.parse(QueryParserDriver.java:201) at org.apache.pig.PigServer$Graph.validateQuery(PigServer.java:1707) at org.apache.pig.PigServer$Graph.registerQuery(PigServer.java:1680) at org.apache.pig.PigServer.registerQuery(PigServer.java:623) at org.apache.pig.tools.grunt.GruntParser.processPig(GruntParser.java:1082) at org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:505) at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:230) at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:205) at org.apache.pig.tools.grunt.Grunt.run(Grunt.java:66) at org.apache.pig.Main.run(Main.java:565) at org.apache.pig.Main.main(Main.java:177) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.util.RunJar.run(RunJar.java:221) at org.apache.hadoop.util.RunJar.main(RunJar.java:136) Caused by: java.lang.NullPointerException at org.apache.pig.builtin.AvroStorage.getAvroSchema(AvroStorage.java:298) at org.apache.pig.builtin.AvroStorage.getAvroSchema(AvroStorage.java:282) at org.apache.pig.builtin.AvroStorage.getSchema(AvroStorage.java:256) at org.apache.pig.newplan.logical.relational.LOLoad.getSchemaFromMetaData(LOLoad.java:175) at org.apache.pig.newplan.logical.relational.LOLoad.<init>(LOLoad.java:89) at org.apache.pig.parser.LogicalPlanBuilder.buildLoadOp(LogicalPlanBuilder.java:901) at org.apache.pig.parser.LogicalPlanGenerator.load_clause(LogicalPlanGenerator.java:3568) at org.apache.pig.parser.LogicalPlanGenerator.op_clause(LogicalPlanGenerator.java:1625) at org.apache.pig.parser.LogicalPlanGenerator.general_statement(LogicalPlanGenerator.java:1102) at org.apache.pig.parser.LogicalPlanGenerator.statement(LogicalPlanGenerator.java:560) at org.apache.pig.parser.LogicalPlanGenerator.query(LogicalPlanGenerator.java:421) at org.apache.pig.parser.QueryParserDriver.parse(QueryParserDriver.java:191) ... 16 more ================================================================================ /tmp/hsperfdata_hdfs/pig_1454087102249.log (END) ... View more