Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Pig 0.12 + CDH 5.3 Compatibility

avatar
Champion Alumni

Hello,

 

I'm trying to do an EvalFunc  UDF for PIG and I'm getting the folowing error:

 

java.lang.IncompatibleClassChangeError: Found interface org.apache.hadoop.mapreduce.JobContext, but class was expected
	at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat.setupUdfEnvAndStores(PigOutputFormat.java:225)
	at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat.getOutputCommitter(PigOutputFormat.java:275)
	at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.createOutputCommitter(MRAppMaster.java:465)
	at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceInit(MRAppMaster.java:368)
	at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
	at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$1.run(MRAppMaster.java:1477)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:415)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1642)
	at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1474)
	at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1407)

 

After searching I found that the mandatory requirements for CDH may not be fulfilled:

http://pig.apache.org/docs/r0.12.0/start.html 

http://stackoverflow.com/questions/21873050/pig-found-interface-org-apache-hadoop-mapreduce-jobconte...

 

 

En fact, now,  I can't see how was this working until now... How is possible to have pig 0.12 with Hadoop 2.5?

 

 

Thank you!!

 

 

 

GHERMAN Alina
1 ACCEPTED SOLUTION

avatar
Champion Alumni

 

Note:

- the job is run with Map Reduce 2 only if I add a new library to the UDF  ==> I get this error only when I add an external library to the UDF. 

 

The job run with MR1 (and a lot of warnings) when I added the library with register in the pig script. 

 

Alina

 

 

GHERMAN Alina

View solution in original post

1 REPLY 1

avatar
Champion Alumni

 

Note:

- the job is run with Map Reduce 2 only if I add a new library to the UDF  ==> I get this error only when I add an external library to the UDF. 

 

The job run with MR1 (and a lot of warnings) when I added the library with register in the pig script. 

 

Alina

 

 

GHERMAN Alina