java.lang.NoClassDefFoundError using Hbase Storage Handler on Hive

scubMUC — Fri, 16 Sep 2022 09:01:26 GMT

Hi,

i want to create an external hive tebale mapped on hbase in my oozie workflow. on cdh4 it works without problems but using the script on cdh5 i got an "java.lang.NoClassDefFoundError" exception. I have added all jars like in the documentation:

http://www.cloudera.com/content/cloudera-content/cloudera-docs/CDH5/latest/CDH5-Installation-Guide/cdh5ig_hive_hbase.html

I also added the jars with the "hive.aux.jars.path" property at the hive-site.xml

Hive script:

CREATE EXTERNAL TABLE IF NOT EXISTS hive_table(
 hash BIGINT,
 f1 STRING
 )STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler' WITH SERDEPROPERTIES (
 'hbase.columns.mapping' =':key,cf1:f1')TBLPROPERTIES('hbase.table.name' = 'hbase_table');

Error:

3029 [main] ERROR hive.ql.exec.DDLTask  - java.lang.NoClassDefFoundError: org/apache/hadoop/hbase/mapreduce/TableInputFormatBase
  at java.lang.ClassLoader.defineClass1(Native Method)
  at java.lang.ClassLoader.defineClass(ClassLoader.java:800)
  at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
  at java.net.URLClassLoader.defineClass(URLClassLoader.java:449)
  at java.net.URLClassLoader.access$100(URLClassLoader.java:71)
  at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
  at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
  at java.security.AccessController.doPrivileged(Native Method)
  at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
  at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
  at java.lang.ClassLoader.loadClass(ClassLoader.java:412)
  at java.lang.ClassLoader.loadClass(ClassLoader.java:412)
  at java.lang.ClassLoader.loadClass(ClassLoader.java:412)
  at java.lang.ClassLoader.loadClass(ClassLoader.java:412)
  at java.lang.ClassLoader.loadClass(ClassLoader.java:412)
  at java.lang.ClassLoader.loadClass(ClassLoader.java:412)
  at java.lang.ClassLoader.loadClass(ClassLoader.java:412)
  at java.lang.ClassLoader.loadClass(ClassLoader.java:412)
  at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
  at java.lang.Class.forName0(Native Method)
  at java.lang.Class.forName(Class.java:270)
  at org.apache.hadoop.hive.ql.metadata.Table.getInputFormatClass(Table.java:305)
  at org.apache.hadoop.hive.ql.metadata.Table.<init>(Table.java:98)
  at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:991)
  at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:894)
  at org.apache.hadoop.hive.ql.exec.DDLTask.dropTable(DDLTask.java:3308)
  at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:278)
  at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:151)
  at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:65)
  at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1485)
  at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1263)
  at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1091)
  at org.apache.hadoop.hive.ql.Driver.run(Driver.java:931)
  at org.apache.hadoop.hive.ql.Driver.run(Driver.java:921)
  at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:268)
  at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:220)
  at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:422)
  at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:357)
  at org.apache.hadoop.hive.cli.CliDriver.processReader(CliDriver.java:455)
  at org.apache.hadoop.hive.cli.CliDriver.processFile(CliDriver.java:465)
  at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:746)
  at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:684)
  at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:623)
  at org.apache.oozie.action.hadoop.HiveMain.runHive(HiveMain.java:315)
  at org.apache.oozie.action.hadoop.HiveMain.run(HiveMain.java:276)
  at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:38)
  at org.apache.oozie.action.hadoop.HiveMain.main(HiveMain.java:66)
  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  at java.lang.reflect.Method.invoke(Method.java:606)
  at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:226)
  at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
  at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:430)
  at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342)
  at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
  at java.security.AccessController.doPrivileged(Native Method)
  at javax.security.auth.Subject.doAs(Subject.java:415)
  at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
  at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)
  Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.hbase.mapreduce.TableInputFormatBase
  at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
  at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
  at java.security.AccessController.doPrivileged(Native Method)
  at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
  at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
  at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
  ... 60 more

My Cluster is configured with CDH5.0.2 and CDM5 with Yarn MRv2

I hope you can help me which jar file is missing

Re: java.lang.NoClassDefFoundError using Hbase Storage Handler on Hive

scubMUC — Tue, 01 Jul 2014 12:39:31 GMT

Has anyone an idea what library is missing?

It would be great if someone could help me briefly?

thanks a lot 🙂

BG,

Re: java.lang.NoClassDefFoundError using Hbase Storage Handler on Hive

SRAI — Fri, 31 Oct 2014 16:07:36 GMT

I'm having the same problem. Did you find a fix?

Re: java.lang.NoClassDefFoundError using Hbase Storage Handler on Hive

SRAI — Fri, 31 Oct 2014 17:51:20 GMT

I was able to fix this issue by copying the following jars located at /opt/cloudera/parcels/CDH/lib/hive/lib to /user/oozie/share/lib/lib_{ts}/hive/

hbase-client.jar
hbase-common.jar
hbase-hadoop2-compat.jar
hbase-hadoop-compat.jar
hbase-protocol.jar
hbase-server.jar
hive-hbase-handler-0.12.0-cdh5.1.3.jar
hive-hbase-handler.jar

htrace-core.jar

I'm using Hadoop 2.3.0-cdh5.1.3 and CDH manager 5.

- Saimon

question java.lang.NoClassDefFoundError using Hbase Storage Handler on Hive in Archives of Support Questions (Read Only)

java.lang.NoClassDefFoundError using Hbase Storage Handler on Hive

Re: java.lang.NoClassDefFoundError using Hbase Storage Handler on Hive

Re: java.lang.NoClassDefFoundError using Hbase Storage Handler on Hive

Re: java.lang.NoClassDefFoundError using Hbase Storage Handler on Hive