Support Questions
Find answers, ask questions, and share your expertise

CDH5 Hive UDTF KryoException: Class cannot be created (missing no-arg constructor)

New Contributor

Hello,

 

I have written a GenericUDTF (SentimentUDF) that uses OpenNLP's sentence detector. I have included the OpenNLP libraries in the Hive and Hadoop lib directories. This works flawless using the CDH 4.4 version of Hive and Hadoop (current VM version). In CDH 5 (Hive 0.12.0+cdh5.0.0+308, Hadoop 2.3.0+cdh5.0.0+548) on AWS, I get the following RuntimeException:

 

java.lang.RuntimeException: org.apache.hive.com.esotericsoftware.kryo.KryoException: Class cannot be created (missing no-arg constructor): opennlp.tools.sentdetect.SentenceDetectorME
Serialization trace:
sentenceDetector (de.bankmark.bigbench.queries.SentimentUDF)
genericUDTF (org.apache.hadoop.hive.ql.plan.UDTFDesc)
conf (org.apache.hadoop.hive.ql.exec.UDTFOperator)
childOperators (org.apache.hadoop.hive.ql.exec.SelectOperator)
childOperators (org.apache.hadoop.hive.ql.exec.TableScanOperator)
aliasToWork (org.apache.hadoop.hive.ql.plan.MapWork)
at org.apache.hadoop.hive.ql.exec.Utilities.getBaseWork(Utilities.java:320)
at org.apache.hadoop.hive.ql.exec.Utilities.getMapWork(Utilities.java:259)
at org.apache.hadoop.hive.ql.io.HiveInputFormat.init(HiveInputFormat.java:251)
at org.apache.hadoop.hive.ql.io.HiveInputFormat.pushProjectionsAndFilters(HiveInputFormat.java:378)
at org.apache.hadoop.hive.ql.io.HiveInputFormat.pushProjectionsAndFilters(HiveInputFormat.java:371)
at org.apache.hadoop.hive.ql.io.CombineHiveInputFormat.getRecordReader(CombineHiveInputFormat.java:556)
at org.apache.hadoop.mapred.MapTask$TrackedRecordReader.<init>(MapTask.java:168)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:409)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)

 

Is there a difference in how to write UDFs in the CDH 5 version of Hive or is this a bug?

 

As a follow-up question, is there a better way to deploy jars like OpenNLP that are used inside the UDF? Add jars did not work for me, neither did just adding them to the hive lib directory.

 

Thanks,

Tilmann

1 REPLY 1

Re: CDH5 Hive UDTF KryoException: Class cannot be created (missing no-arg constructor)

Master Guru
CDH5's Apache Hive version has indeed changed the way we handle UDF construction (it now uses a Kyro based serialiser). This appears to be a bug, so I'd recommend taking it up upstream on the Apache Hive lists or JIRA to evaluate what can be done at the Hive layer to handle classes from 3rd party dependencies that would not be carrying default constructors that Kyro appears to rely on.