Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Error while inserting from Hive to Hbase

Error while inserting from Hive to Hbase

New Contributor

I am using CDH 4.7.1 cluster. Map seems completed 100% and failing the reduce part. I have added the below part to hive-site.xml. Actual error message is pasted in the last part of this post. Thanks. Any help is appreciated.

<property>
    <name>hive.aux.jars.path</name>
    <value> file:///opt/cloudera/parcels/CDH/lib/hbase/hbase.jar, 
        file:///opt/cloudera/parcels/CDH-4.7.1-1.cdh4.7.1.p0.47/lib/hive/lib/hive-hbase-handler-0.10.0-cdh4.7.1.jar,
        file:///opt/cloudera/parcels/CDH-4.7.1-1.cdh4.7.1.p0.47/lib/hive/lib/zookeeper.jar,
        file:///opt/cloudera/parcels/CDH-4.7.1-1.cdh4.7.1.p0.47/lib/hive/lib/guava-11.0.2.jar
    </value>
</property>

ERROR:

java.lang.ClassNotFoundException: org.apache.hadoop.hive.hbase.HBaseSerDe
Continuing ...
java.lang.ClassNotFoundException: org.apache.hadoop.hive.hbase.HiveHBaseTableInputFormat
Continuing ...
java.lang.ClassNotFoundException: org.apache.hadoop.hive.hbase.HiveHBaseTableOutputFormat
Continuing ...
java.lang.NullPointerException
Continuing ...
java.lang.NullPointerException
at org.apache.hadoop.hive.ql.exec.FileSinkOperator.initializeOp(FileSinkOperator.java:315)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:360)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:436)
at org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:392)
at org.apache.hadoop.hive.ql.exec.Operator.initializeOp(Operator.java:377)
at org.apache.hadoop.hive.ql.exec.LimitOperator.initializeOp(LimitOperator.java:41)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:360)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:436)
at org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:392)
at org.apache.hadoop.hive.ql.exec.ExtractOperator.initializeOp(ExtractOperator.java:40)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:360)
at org.apache.hadoop.hive.ql.exec.ExecReducer.configure(ExecReducer.java:150)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:106)
at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:75)
at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)
at org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:469)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:447)
at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1438)
at org.apache.hadoop.mapred.Child.main(Child.java:262)
2 REPLIES 2

Re: Error while inserting from Hive to Hbase

New Contributor

Additionally, I did the following :

Set the value of HIVE_AUX_JARS_PATH in your hive-env.sh. If you could access to hive shell do the following:

hive> add jar /opt/cloudera/parcels/CDH/lib/hbase/hbase.jar; 
hive> add jar /opt/cloudera/parcels/CDH-4.7.1-1.cdh4.7.1.p0.47/lib/hive/lib/hive-hbase-handler-0.10.0-cdh4.7.1.jar;
hive> add jar /opt/cloudera/parcels/CDH-4.7.1 1.cdh4.7.1.p0.47/lib/hive/lib/zookeeper.jar;
hive> add jar /opt/cloudera/parcels/CDH-4.7.1-1.cdh4.7.1.p0.47/lib/hive/lib/guava-11.0.2.jar;

2 things I can see now.

One: In the detailed log - 15/02/19 05:20:56 INFO zookeeper.ZooKeeper: Client environment: section shows all these jars along with a lot of other Jars. But this section mention only a single jar.15/02/19 05:21:01 INFO exec.ExecDriver: adding libjars: file:///opt/cloudera/parcels/CDH-4.7.1-1.cdh4.7.1.p0.47/lib/hive/lib/hive-builti‌​ns-0.10.0-cdh4.7.1.jar.

Two: when I try "LIST JARS" in hive cli in each node, the jar listing is not consistent. Sometimes it shows only a single jar otherwise all the 5 added jars.

 

Re: Error while inserting from Hive to Hbase

New Contributor

More error log from reduce task:

2015-02-19 16:20:09,624 WARN org.apache.hadoop.mapred.Child: Error running child
java.lang.RuntimeException: Error in configuring object
	at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:109)
	at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:75)
	at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)
	at org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:469)
	at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:447)
	at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:396)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1438)
	at org.apache.hadoop.mapred.Child.main(Child.java:262)
Caused by: java.lang.reflect.InvocationTargetException
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:106)
	... 9 more
Caused by: java.lang.RuntimeException: Reduce operator initialization failed
	at org.apache.hadoop.hive.ql.exec.ExecReducer.configure(ExecReducer.java:157)
	... 14 more
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.NullPointerException
	at org.apache.hadoop.hive.ql.exec.FileSinkOperator.initializeOp(FileSinkOperator.java:373)
	at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:360)
	at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:436)
	at org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:392)
	at org.apache.hadoop.hive.ql.exec.Operator.initializeOp(Operator.java:377)
	at org.apache.hadoop.hive.ql.exec.LimitOperator.initializeOp(LimitOperator.java:41)
	at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:360)
	at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:436)
	at org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:392)
	at org.apache.hadoop.hive.ql.exec.ExtractOperator.initializeOp(ExtractOperator.java:40)
	at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:360)
	at org.apache.hadoop.hive.ql.exec.ExecReducer.configure(ExecReducer.java:150)
	... 14 more
Caused by: java.lang.NullPointerException
	at org.apache.hadoop.hive.ql.exec.FileSinkOperator.initializeOp(FileSinkOperator.java:315)
	... 25 more