Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

AvroStorage with mapreduce and java.lang.RuntimeException: could not instantiate

Solved Go to solution

Re: AvroStorage with mapreduce and java.lang.RuntimeException: could not instantiate

Expert Contributor

Ok, waiting on your results! Thank you

Re: AvroStorage with mapreduce and java.lang.RuntimeException: could not instantiate

Expert Contributor

@Artem Ervits here you can find one more output - > sources were read successfully but output failed,

http://paste.debian.net/377433/

Re: AvroStorage with mapreduce and java.lang.RuntimeException: could not instantiate

Expert Contributor

it is failing to write output avro file but log says:

web-log says

Application Overview

User: hdfs

Name: PigLatin:pigMerger.pig

Application Type: MAPREDUCE

Application Tags:

YarnApplicationState: FINISHED

Queue: default

FinalStatus Reported by AM: SUCCEEDED

Started: Fri Jan 29 12:59:25 +0000 2016

Elapsed: 4mins, 29sec

Tracking URL: History

Log Aggregation Status SUCCEEDED

Diagnostics:

Re: AvroStorage with mapreduce and java.lang.RuntimeException: could not instantiate

Mentor

can you paste sample dataset and pig script, I'll try to reproduce sometime today on my machine. It's hard to see the issue from the logs. @John Smith

Re: AvroStorage with mapreduce and java.lang.RuntimeException: could not instantiate

Expert Contributor

@Artem Ervits source files and pig script are included Data , thanks

Re: AvroStorage with mapreduce and java.lang.RuntimeException: could not instantiate

Expert Contributor

one more log:

log

Re: AvroStorage with mapreduce and java.lang.RuntimeException: could not instantiate

Mentor

@John Smith I highly recommend you develop your scripts in Pig Grunt shell. This is what happened with your script as I was trying to execute it one by one.

grunt> sensitiveSet = load '/user/root/Test-20160129-1401822-ttp.avro' using AvroStorage();
grunt> outputSet = join sensitiveSet by Row_ID, nonSensSet by Row_ID;
grunt> outputSet = distinct outputSet;
grunt> outputSet = foreach outputSet generate nonSensSet::name,nonSensSet::customerId,sensitiveSet::VIN,sensitiveSet::Birthdate,nonSensSet::Mileage,nonSensSet::Fuel_Consumption;
2016-01-29 14:41:59,228 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 1025:
<line 5, column 79> Invalid field projection. Projected field [sensitiveSet::VIN] does not exist in schema: sensitiveSet::Row_ID:long,sensitiveSet::name:chararray,sensitiveSet::customerId:chararray,sensitiveSet::Mileage:chararray,sensitiveSet::Fuel_Consumption:chararray,nonSensSet::Row_ID:long,nonSensSet::name:chararray,nonSensSet::customerId:chararray,nonSensSet::Mileage:chararray,nonSensSet::Fuel_Consumption:chararray.
Details at logfile: /root/pig-upload/pig_1454078371113.log


grunt> describe sensitiveSet;
sensitiveSet: {Row_ID: long,name: chararray,customerId: chararray,Mileage: chararray,Fuel_Consumption: chararray}

Re: AvroStorage with mapreduce and java.lang.RuntimeException: could not instantiate

Expert Contributor

i do develop everything in grunt,... you are missing one line in that script

nonSensSet = load '/d-spool-dir/Test-20160129-1401822-lake.avro' USING AvroStorage();

Re: AvroStorage with mapreduce and java.lang.RuntimeException: could not instantiate

Mentor

@John Smith birthdate also doesn't exist

<line 5, column 79> Invalid field projection. Projected field [sensitiveSet::Birthdate] does not exist in schema: sensitiveSet::Row_ID:long,sensitiveSet::name:chararray,sensitiveSet::customerId:chararray,sensitiveSet::Mileage:chararray,sensitiveSet::Fuel_Consumption:chararray,nonSensSet::Row_ID:long,nonSensSet::name:chararray,nonSensSet::customerId:chararray,nonSensSet::Mileage:chararray,nonSensSet::Fuel_Consumption:chararray.
Details at logfile: /root/pig-upload/pig_1454078371113.log

-- this works

grunt> outputSet = foreach outputSet generate nonSensSet::name,nonSensSet::customerId,nonSensSet::Mileage,nonSensSet::Fuel_Consumption;

Re: AvroStorage with mapreduce and java.lang.RuntimeException: could not instantiate

Mentor
@John Smith

ok it was a problem with me, I copied the same file twice. nevermind that issue, I'm still looking.

Don't have an account?
Coming from Hortonworks? Activate your account here