question Re: AvroStorage with mapreduce and java.lang.RuntimeException: could not instantiate in Archives of Support Questions (Read Only)

AvroStorage with mapreduce and java.lang.RuntimeException: could not instantiate

lenovomi — Fri, 29 Jan 2016 11:32:11 GMT

Hi,

I have simple Pig script. Im trying to load avro file or directory that contains avro file using AvroStorage in Mapreduce mode. I tried almost all the combinations (hdfs://, / , hdfs://ip:port/file ... ) but nothing works.

Using command below

set = load '/spool-dir/CustomerData-20160128-1501807/' USING org.apache.pig.piggybank.storage.avro.AvroStorage ();

I got error:

2016-01-29 00:10:08,439 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 1066: Unable to open iterator for alias sensitiveSet. Backend error : java.lang.RuntimeException: could not instantiate 'org.apache.pig.piggybank.storage.avro.AvroStorage' with arguments 'null' 2016-01-29 00:10:08,439 [main] WARN org.apache.pig.tools.grunt.Grunt - There is no log file to write to. 2016-01-29 00:10:08,439 [main] ERROR org.apache.pig.tools.grunt.Grunt - org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1066: Unable to open iterator for alias sensitiveSet. Backend error : java.lang.RuntimeException: could not instantiate 'org.apache.pig.piggybank.storage.avro.AvroStorage' with arguments 'null' at org.apache.pig.PigServer.openIterator(PigServer.java:925) at org.apache.pig.tools.grunt.GruntParser.processDump(GruntParser.java:754)

or using command with argument

set = load '/spool-dir/CustomerData-20160128-1501807/' USING org.apache.pig.piggybank.storage.avro.AvroStorage('no_schema_check');

2016-01-29 00:25:02,767 [main] ERROR org.apache.pig.tools.grunt.Grunt - org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1066: Unable to open iterator for alias sensitiveSet. Backend error :

java.lang.RuntimeException: could not instantiate 'org.apache.pig.piggybank.storage.avro.AvroStorage' with arguments

'[no_schema_check]' at org.apache.pig.PigServer.openIterator(PigServer.java:925

My samples are almost identical with the ones on the avrostorage documentation, but i really cant see where the problem is.

The problem is partially described on stackexchange also.

Thank you

Re: AvroStorage with mapreduce and java.lang.RuntimeException: could not instantiate

aervits — Fri, 29 Jan 2016 12:01:48 GMT

@John Smith

firstly set is a reserved word, change set to another alias, you can also refer to avro simply by AvroStorage no need to write out full package name. If all else fails, add register piggybank.jar command.

Re: AvroStorage with mapreduce and java.lang.RuntimeException: could not instantiate

lenovomi — Fri, 29 Jan 2016 17:50:02 GMT

Hi, sorry the set was my mistypo here:

outSet = load 'hdfs:///CustomerData-20160128-1501807.avro' USING AvroStorage();

This command works, which is ODD, because whats the different when you call it as AvroStorage() or using full package path

org.apache.pig.piggybank.storage.avro.AvroStorage()

Re: AvroStorage with mapreduce and java.lang.RuntimeException: could not instantiate

lenovomi — Fri, 29 Jan 2016 17:52:08 GMT

I have another issue with STORE now ....

STORE outputSet INTO 'hdfs:///avro-dest/-CustomerData-20160128-1501807'>> USING AvroStorage('no_schema_check', 'schema', '{"type":"record","name":"xxx","fields":[{"name":"name","type":"string","title":"Customer name","description":"non Surrogate Key for joining files on the BDP"}, ....]}');

error below:

2016-01-29 09:48:42,211 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 1200: Pig script failed to parse:

<line 20, column 0> pig script failed to validate: java.lang.RuntimeException: could not instantiate 'AvroStorage' with arguments '[no_schema_check, schema, {"type":"record",

Re: AvroStorage with mapreduce and java.lang.RuntimeException: could not instantiate

lenovomi — Fri, 29 Jan 2016 18:16:12 GMT

ok so STORE works only with

org.apache.pig.piggybank.storage.avro.AvroStorage(.... )

But there are still issues while trying to write output file

2016-01-29 10:09:28,406 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - HadoopJobId: job_1454023575813_0018
2016-01-29 10:09:28,406 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Processing aliases outputSet
2016-01-29 10:09:28,406 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - detailed locations: M:  C:  R: outputSet[19,12]
2016-01-29 10:10:03,931 [main] WARN  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Ooops! Some job has failed! Specify -stop_on_failure if you want Pig to stop immediately on failure.
2016-01-29 10:10:03,931 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - job job_1454023575813_0018 has failed! Stop running all dependent jobs
2016-01-29 10:10:03,931 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 100% complete
2016-01-29 10:10:06,256 [main] INFO  org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl - Timeline service address: http://sandbox.hortonworks.com:8188/ws/v1/timeline/
2016-01-29 10:10:06,257 [main] INFO  org.apache.hadoop.yarn.client.RMProxy - Connecting to ResourceManager at sandbox.hortonworks.com/10.0.1.47:8050
2016-01-29 10:10:07,417 [main] INFO  org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl - Timeline service address: http://sandbox.hortonworks.com:8188/ws/v1/timeline/
2016-01-29 10:10:07,417 [main] INFO  org.apache.hadoop.yarn.client.RMProxy - Connecting to ResourceManager at sandbox.hortonworks.com/10.0.1.47:8050
2016-01-29 10:10:07,577 [main] ERROR org.apache.pig.tools.pigstats.mapreduce.MRPigStatsUtil - 1 map reduce job(s) failed!
2016-01-29 10:10:07,585 [main] INFO  org.apache.pig.tools.pigstats.mapreduce.SimplePigStats - Script Statistics:

Failed Jobs:
JobId  Alias  Feature Message Outputs
job_1454023575813_0018  outputSet  DISTINCT  Message: Job failed!  hdfs:///avro-dest/CustomerData-20160128-1501807,

Output(s) Failed to produce result in "hdfs:///avro-dest/CustomerData-20160128-1501807"

Well i really dont understand whats going on here ... no proper documentation, for me random behavior its really hard to use the tool like that.

Re: AvroStorage with mapreduce and java.lang.RuntimeException: could not instantiate

lenovomi — Fri, 29 Jan 2016 18:30:47 GMT

And now it says ... that i cant read data.. both files are there ... even previous run was Successful with reading the source data... Well im so desperate, this is like working with random turing machine. ;-(

How it can fail to read data .... i can easily dump both relations that read data from those input files.

Input(s):

Failed to read data from "hdfs:///CustomerData-20160128-1501807.avro"

Failed to read data from "hdfs:///CustomerData-20160128-1501807.avro"

Output(s):
Failed to produce result in "hdfs:///CustomerData-20160128-1501807"

Re: AvroStorage with mapreduce and java.lang.RuntimeException: could not instantiate

lenovomi — Fri, 29 Jan 2016 18:54:47 GMT

Still failing ;-(

Failed Jobs:

JobId Alias Feature Message Outputs

job_1454023575813_0027 outputSet DISTINCT Message: Job failed! /CustomerData-20160128-1501807,

Input(s):

Successfully read 100 records from: "/CustomerData-20160128-1501807-l.avro"

Successfully read 100 records from: "/CustomerData-20160128-1501807-t.avro"

Output(s): Failed to produce result in "/avro-dest/CustomerData-20160128-1501807"

Re: AvroStorage with mapreduce and java.lang.RuntimeException: could not instantiate

lenovomi — Fri, 29 Jan 2016 19:04:14 GMT

here is the full log: log

Re: AvroStorage with mapreduce and java.lang.RuntimeException: could not instantiate

aervits — Fri, 29 Jan 2016 20:26:03 GMT

@John Smith AvroStorage may have different package now though I confirmed with javadoc and I was the same as yours, it may be packaged differently in HDP, classpath may differ, don't know for sure. Please accept this answer.

Re: AvroStorage with mapreduce and java.lang.RuntimeException: could not instantiate

aervits — Fri, 29 Jan 2016 20:28:48 GMT

@John Smith I'll review and let you know.

Re: AvroStorage with mapreduce and java.lang.RuntimeException: could not instantiate

lenovomi — Fri, 29 Jan 2016 20:42:14 GMT

Ok, waiting on your results! Thank you

Re: AvroStorage with mapreduce and java.lang.RuntimeException: could not instantiate

lenovomi — Fri, 29 Jan 2016 20:45:24 GMT

@Artem Ervits here you can find one more output - > sources were read successfully but output failed,

http://paste.debian.net/377433/

Re: AvroStorage with mapreduce and java.lang.RuntimeException: could not instantiate

lenovomi — Fri, 29 Jan 2016 21:52:25 GMT

it is failing to write output avro file but log says:

web-log says

Application Overview

User: hdfs

Name: PigLatin:pigMerger.pig

Application Type: MAPREDUCE

Application Tags:

YarnApplicationState: FINISHED

Queue: default

FinalStatus Reported by AM: SUCCEEDED

Started: Fri Jan 29 12:59:25 +0000 2016

Elapsed: 4mins, 29sec

Tracking URL: History

Log Aggregation Status SUCCEEDED

Diagnostics:

Re: AvroStorage with mapreduce and java.lang.RuntimeException: could not instantiate

lenovomi — Fri, 29 Jan 2016 21:55:31 GMT

one more log:

log

Re: AvroStorage with mapreduce and java.lang.RuntimeException: could not instantiate

aervits — Fri, 29 Jan 2016 21:56:47 GMT

can you paste sample dataset and pig script, I'll try to reproduce sometime today on my machine. It's hard to see the issue from the logs. @John Smith

Re: AvroStorage with mapreduce and java.lang.RuntimeException: could not instantiate

lenovomi — Fri, 29 Jan 2016 22:21:39 GMT

@Artem Ervits source files and pig script are included Data , thanks

Re: AvroStorage with mapreduce and java.lang.RuntimeException: could not instantiate

aervits — Fri, 29 Jan 2016 22:44:05 GMT

@John Smith I highly recommend you develop your scripts in Pig Grunt shell. This is what happened with your script as I was trying to execute it one by one.

grunt> sensitiveSet = load '/user/root/Test-20160129-1401822-ttp.avro' using AvroStorage();
grunt> outputSet = join sensitiveSet by Row_ID, nonSensSet by Row_ID;
grunt> outputSet = distinct outputSet;
grunt> outputSet = foreach outputSet generate nonSensSet::name,nonSensSet::customerId,sensitiveSet::VIN,sensitiveSet::Birthdate,nonSensSet::Mileage,nonSensSet::Fuel_Consumption;
2016-01-29 14:41:59,228 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 1025:
<line 5, column 79> Invalid field projection. Projected field [sensitiveSet::VIN] does not exist in schema: sensitiveSet::Row_ID:long,sensitiveSet::name:chararray,sensitiveSet::customerId:chararray,sensitiveSet::Mileage:chararray,sensitiveSet::Fuel_Consumption:chararray,nonSensSet::Row_ID:long,nonSensSet::name:chararray,nonSensSet::customerId:chararray,nonSensSet::Mileage:chararray,nonSensSet::Fuel_Consumption:chararray.
Details at logfile: /root/pig-upload/pig_1454078371113.log


grunt> describe sensitiveSet;
sensitiveSet: {Row_ID: long,name: chararray,customerId: chararray,Mileage: chararray,Fuel_Consumption: chararray}

Re: AvroStorage with mapreduce and java.lang.RuntimeException: could not instantiate

aervits — Fri, 29 Jan 2016 22:45:48 GMT

@John Smith birthdate also doesn't exist

<line 5, column 79> Invalid field projection. Projected field [sensitiveSet::Birthdate] does not exist in schema: sensitiveSet::Row_ID:long,sensitiveSet::name:chararray,sensitiveSet::customerId:chararray,sensitiveSet::Mileage:chararray,sensitiveSet::Fuel_Consumption:chararray,nonSensSet::Row_ID:long,nonSensSet::name:chararray,nonSensSet::customerId:chararray,nonSensSet::Mileage:chararray,nonSensSet::Fuel_Consumption:chararray.
Details at logfile: /root/pig-upload/pig_1454078371113.log

-- this works

grunt> outputSet = foreach outputSet generate nonSensSet::name,nonSensSet::customerId,nonSensSet::Mileage,nonSensSet::Fuel_Consumption;

Re: AvroStorage with mapreduce and java.lang.RuntimeException: could not instantiate

aervits — Fri, 29 Jan 2016 23:06:16 GMT

@John Smith

ok it was a problem with me, I copied the same file twice. nevermind that issue, I'm still looking.

Re: AvroStorage with mapreduce and java.lang.RuntimeException: could not instantiate

lenovomi — Fri, 29 Jan 2016 23:09:55 GMT

thats strange... it works for me.

grunt> sensitiveSet = load '/t-spool-dir/Test-20160129-1401822-ttp.avro' USING AvroStorage();

grunt> nonSensSet = load '/d-spool-dir/Test-20160129-1401822-lake.avro' USING AvroStorage();

grunt> outputSet = join sensitiveSet by Row_ID, nonSensSet by Row_ID;grunt> outputSet = distinct outputSet;

grunt> outputSet = foreach outputSet generate nonSensSet::name,nonSensSet::customerId,sensitiveSet::VIN,sensitiveSet::Birthdate,nonSensSet::Mileage,nonSensSet::Fuel_Consumption;grunt> 

dump outputSet;

("Kina Buttars",12452346,"WBA32649710927373","1968-08-14",68,10.551)

("Caren Rodman",18853438,"WBA56064572124841","1987-01-24",96,6.779)

("Tierra Bork",89673290,"WBA69315467645466","1958-11-22",52,10.109)

("Thelma Steve",97170856,"WBA73739033913927","1985-12-03",98,5.081)

.....