Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

unable to use hive and hbase integration with oozie

Highlighted

unable to use hive and hbase integration with oozie

Rising Star

Hi all,

I'm trying to execute an oozie script that read data from an hbase table mapped in hive. If I execute the script via beeline it runs well, but using oozie I get the following error:

13795 [main] ERROR hive.log  - error in initSerDe: java.lang.ClassNotFoundException Class org.apache.hadoop.hive.hbase.HBaseSerDe not found
java.lang.ClassNotFoundException: Class org.apache.hadoop.hive.hbase.HBaseSerDe not found
	at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2101)
	at org.apache.hadoop.hive.metastore.MetaStoreUtils.getDeserializer(MetaStoreUtils.java:395)
	at org.apache.hadoop.hive.ql.metadata.Table.getDeserializerFromMetaStore(Table.java:276)
	at org.apache.hadoop.hive.ql.metadata.Table.getDeserializer(Table.java:258)
	at org.apache.hadoop.hive.ql.metadata.Table.getCols(Table.java:605)

In the oozie shared library I added the hive-hbase-handler jar, under this path:

/user/oozie/share/lib/lib_20151020075400/hive/hive-hbase-handler-1.2.1.2.3.2.0-2950.jar

I also added hive-site.xml and hbase-site.xml files in the oozie shared library

This is my job.properties file:

nameNode=hdfs://masterHA
jobTracker=master03:8032
queueName=HOYA
oozie.use.system.libpath=true
#oozie.libpath=/user/oozie/share/lib/lib_20151020075400
oozie.wf.application.path=${nameNode}/user/root/workflow.xml

By the way, in the container path on the slave, I'm unable to find the jar

Current (local) dir = /grid/5/hadoop/yarn/local/usercache/root/appcache/application_1464329584635_67853/container_e70_1464329584635_67853_01_000002
------------------------
  tmp
  container_tokens
  .container_tokens.crc
  launch_container.sh
  .launch_container.sh.crc
  default_container_executor_session.sh
  .default_container_executor_session.sh.crc
  default_container_executor.sh
  .default_container_executor.sh.crc
  json-simple-1.1.jar
  curator-client-2.6.0.jar
  servlet-api-2.5.jar
  slf4j-api-1.6.6.jar
  curator-recipes-2.5.0.jar
  joda-time-2.1.jar
  jackson-databind-2.2.3.jar
  stax-api-1.0.1.jar
  jersey-client-1.9.jar
  parquet-hadoop-bundle-1.6.0.jar
  jdo-api-3.0.1.jar
  libthrift-0.9.2.jar
  commons-compress-1.4.1.jar
  httpclient-4.2.5.jar
  hadoop-annotations-2.7.1.2.3.2.0-2950.jar
  xz-1.0.jar
  job.xml
  commons-codec-1.4.jar
  jetty-all-7.6.0.v20120127.jar
  hadoop-yarn-common-2.7.1.2.3.2.0-2950.jar
  bonecp-0.8.0.RELEASE.jar
  antlr-runtime-3.4.jar
  guice-3.0.jar
  hive-shims-1.2.1.2.3.2.0-2950.jar
  stringtemplate-3.2.1.jar
  commons-httpclient-3.1.jar
  libfb303-0.9.2.jar
  commons-io-2.4.jar
  hive-contrib-1.2.1.2.3.2.0-2950.jar
  oozie-hadoop-utils-hadoop-2-4.2.0.2.3.2.0-2950.jar
  calcite-avatica-1.2.0.2.3.2.0-2950.jar
  jline-2.12.jar
  geronimo-jta_1.1_spec-1.1.1.jar
  objenesis-2.1.jar
  commons-collections4-4.0.jar
  hive-shims-scheduler-1.2.1.2.3.2.0-2950.jar
  aopalliance-1.0.jar
  commons-collections-3.2.1.jar
  activation-1.1.jar
  jersey-json-1.9.jar
  opencsv-2.3.jar
  jersey-core-1.9.jar
  oozie-sharelib-hive-4.2.0.2.3.2.0-2950.jar
  hive-shims-0.20S-1.2.1.2.3.2.0-2950.jar
  avro-1.7.5.jar
  aws-java-sdk-1.7.4.jar
  netty-3.6.2.Final.jar
  commons-pool-1.5.4.jar
  tez-runtime-internals-0.7.0.2.3.2.0-2950.jar
  commons-cli-1.2.jar
  curator-framework-2.6.0.jar
  ant-1.9.1.jar
  hadoop-azure-2.7.1.2.3.2.0-2950.jar
  guava-11.0.2.jar
  jackson-core-2.2.3.jar
  hive-shims-common-1.2.1.2.3.2.0-2950.jar
  ST4-4.0.4.jar
  datanucleus-rdbms-3.2.9.jar
  httpcore-4.2.4.jar
  asm-tree-3.1.jar
  javassist-3.18.1-GA.jar
  slf4j-log4j12-1.6.6.jar
  tez-api-0.7.0.2.3.2.0-2950.jar
  hive-service-1.2.1.2.3.2.0-2950.jar
  hive-cli-1.2.1.2.3.2.0-2950.jar
  ivy-2.4.0.jar
  jaxb-impl-2.2.3-1.jar
  pentaho-aggdesigner-algorithm-5.1.5-jhyde.jar
  commons-lang3-3.3.2.jar
  mr-framework
  jpam-1.1.jar
  asm-commons-3.1.jar
  jetty-6.1.14.jar
  datanucleus-api-jdo-3.2.6.jar
  asm-3.1.jar
  hive-exec-1.2.1.2.3.2.0-2950.jar
  calcite-linq4j-1.2.0.2.3.2.0-2950.jar
  hive-serde-1.2.1.2.3.2.0-2950.jar
  log4j-1.2.16.jar
  datanucleus-core-3.2.10.jar
  hadoop-yarn-server-web-proxy-2.7.1.2.3.2.0-2950.jar
  mail-1.4.jar
  refreshCurrencyList.sql
  jackson-xc-1.9.13.jar
  jetty-util-6.1.26.hwx.jar
  commons-logging-1.1.jar
  guice-servlet-3.0.jar
  snappy-java-1.0.5.jar
  protobuf-java-2.5.0.jar
  tez-common-0.7.0.2.3.2.0-2950.jar
  janino-2.7.6.jar
  tez-runtime-library-0.7.0.2.3.2.0-2950.jar
  tez-mapreduce-0.7.0.2.3.2.0-2950.jar
  hadoop-yarn-api-2.7.1.2.3.2.0-2950.jar
  oro-2.0.8.jar
  fst-2.24.jar
  tez-dag-0.7.0.2.3.2.0-2950.jar
  jsr305-2.0.3.jar
  commons-lang-2.4.jar
  groovy-all-2.1.6.jar
  json-20090211.jar
  hive-shims-0.23-1.2.1.2.3.2.0-2950.jar
  hive-common-1.2.1.2.3.2.0-2950.jar
  hadoop-yarn-server-resourcemanager-2.7.1.2.3.2.0-2950.jar
  calcite-core-1.2.0.2.3.2.0-2950.jar
  jackson-jaxrs-1.9.13.jar
  hadoop-yarn-registry-2.7.1.2.3.2.0-2950.jar
  geronimo-annotation_1.0_spec-1.1.1.jar
  hadoop-aws-2.7.1.2.3.2.0-2950.jar
  apache-curator-2.6.0.pom
  oozie-sharelib-oozie-4.2.0.2.3.2.0-2950.jar
  hadoop-yarn-server-applicationhistoryservice-2.7.1.2.3.2.0-2950.jar
  eigenbase-properties-1.1.5.jar
  hadoop-yarn-server-common-2.7.1.2.3.2.0-2950.jar
  zookeeper-3.4.6.2.3.2.0-2950-tests.jar
  geronimo-jaspic_1.0_spec-1.0.jar
  commons-compiler-2.7.6.jar
  hive-ant-1.2.1.2.3.2.0-2950.jar
  zookeeper-3.4.6.2.3.2.0-2950.jar
  tez-yarn-timeline-history-0.7.0.2.3.2.0-2950.jar
  jettison-1.3.4.jar
  hive-metastore-1.2.1.2.3.2.0-2950.jar
  ant-launcher-1.9.1.jar
  servlet-api-2.5-6.1.14.jar
  jta-1.1.jar
  jackson-annotations-2.2.3.jar
  commons-dbcp-1.4.jar
  derby-10.10.1.1.jar
  velocity-1.5.jar
  antlr-2.7.7.jar
  jaxb-api-2.2.2.jar
  paranamer-2.3.jar
  jersey-guice-1.9.jar
  azure-storage-2.2.0.jar
  javax.inject-1.jar
  stax-api-1.0-2.jar
  tez-yarn-timeline-history-with-acls-0.7.0.2.3.2.0-2950.jar
  apache-log4j-extras-1.1.jar
  leveldbjni-all-1.8.jar
  .job.xml.crc
  action.xml
  .action.xml.crc
  propagation-conf.xml
  hive-site.xml
  hive-log4j.properties
  hive-exec-log4j.properties

I'm running hbase on slider.

Do you have any tips?

Thank you!

3 REPLIES 3

Re: unable to use hive and hbase integration with oozie

Mentor

In your job.properties add the following

oozie.action.sharelib.for.hive=hive,hbase

Re: unable to use hive and hbase integration with oozie

Rising Star

tried to add it, but nothing changed

nameNode=hdfs://masterHA
jobTracker=master03:8032
queueName=HOYA
oozie.use.system.libpath=true
oozie.action.sharelib.for.hive=hive,hbase
91928 [main] ERROR hive.log  - error in initSerDe: java.lang.ClassNotFoundException Class org.apache.hadoop.hive.hbase.HBaseSerDe not found
java.lang.ClassNotFoundException: Class org.apache.hadoop.hive.hbase.HBaseSerDe not found
	at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2101)
	at org.apache.hadoop.hive.metastore.MetaStoreUtils.getDeserializer(MetaStoreUtils.java:395)
	at org.apache.hadoop.hive.ql.metadata.Table.getDeserializerFromMetaStore(Table.java:276)
	at org.apache.hadoop.hive.ql.metadata.Table.getDeserializer(Table.java:258)
	at org.apache.hadoop.hive.ql.metadata.Table.getCols(Table.java:605)
	at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.handleInsertStatementSpecPhase1(SemanticAnalyzer.java:1459)
	at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.doPhase1(SemanticAnalyzer.java:1187)

Re: unable to use hive and hbase integration with oozie

New Contributor

copy the following jars into hive folder. And then into Oozie hive sharelib folder.

/usr/hdp/2.3.2.0-2950/hive/lib/guava-14.0.1.jar,/usr/hdp/2.3.2.0-2950/hive/lib/zookeeper-3.4.6.2.3.2.0-2950.jar,/usr/hdp/2.3.2.0-2950/hive/lib/hive-hbase-handler.jar ,/usr/hdp/2.3.2.0-2950/hbase/lib/htrace-core-3.1.0-incubating.jar,/usr/hdp/2.3.2.0-2950/hbase/lib/hbase-client.jar,/usr/hdp/2.3.2.0-2950/hbase/lib/hbase-common.jar,/usr/hdp/2.3.2.0-2950/hbase/lib/hbase-server.jar,/usr/hdp/2.3.2.0-2950/hbase/lib/hbase-protocol.jar,/usr/hdp/2.3.2.0-2950/hbase/lib/hbase-hadoop-compat.jar,/usr/hdp/2.3.2.0-2950/hbase/lib/hbase-hadoop2-compat.jar,/usr/hdp/2.3.2.0-2950/hbase/lib/netty-3.2.4.Final.jar

Oozie Update shaerlib.