Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Problem in inserting data into hbase table through hive

avatar
Contributor
Trying to insert data into hbase table through hive.:

Create table query:
CREATE TABLE hbase_testing(id int, name string, password string) STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler' WITH SERDEPROPERTIES ("hbase.columns.mapping" = ":key,cf1:val") TBLPROPERTIES ("hbase.table.name" = "hbase_testing");

Insert Query:

INSERT OVERWRITE TABLE hbase_test2 SELECT * FROM user WHERE key=1;

 

Error:

java.lang.RuntimeException: Error in configuring object at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:106) at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:72) at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:130) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:413) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:332) at org.apache.hadoop.mapred.Child$4.run(Child.java:268) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408) at org.apache.hadoop.mapred.Child.main(Child.java:262) Caused by: java.lang.reflect.InvocationTargetException at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.ja java.lang.RuntimeException: Error in configuring object at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:106) at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:72) at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:130) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:413) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:332) at org.apache.hadoop.mapred.Child$4.run(Child.java:268) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408) at org.apache.hadoop.mapred.Child.main(Child.java:262) Caused by: java.lang.reflect.InvocationTargetException at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.ja java.lang.RuntimeException: Error in configuring object at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:106) at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:72) at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:130) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:413) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:332) at org.apache.hadoop.mapred.Child$4.run(Child.java:268) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408) at org.apache.hadoop.mapred.Child.main(Child.java:262) Caused by: java.lang.reflect.InvocationTargetException at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.ja java.lang.RuntimeException: Error in configuring object at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:106) at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:72) at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:130) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:413) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:332) at org.apache.hadoop.mapred.Child$4.run(Child.java:268) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408) at org.apache.hadoop.mapred.Child.main(Child.java:262) Caused by: java.lang.reflect.InvocationTargetException at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.ja
1 ACCEPTED SOLUTION

avatar
Expert Contributor

Some of the jars are not added in the hive --auxpath command.

 

Also add /hbase-0.94.1/hbase-0.94.6.jar and hive_contrib.jar. The version may be different.

View solution in original post

6 REPLIES 6

avatar
Expert Contributor

For Hive add the following JARs (version may be different) to classpath.


 

/opt/hive/lib/hive-hbase-handler-0.8.0-SNAPSHOT.jar
/opt/hive/lib/hbase-0.90.1-SNAPSHOT.jar
/opt/hive/lib/zookeeper-3.3.1.jar

 

 

Add to classpath either by adding to the Hadoop lib dir (/usr/lib/hadoop/lib) or with the add jar command.

 

add jar /opt/hive/lib/hive-hbase-handler-0.8.0-SNAPSHOT.jar
add jar /opt/hive/lib/hbase-0.90.1-SNAPSHOT.jar
add jar /opt/hive/lib/zookeeper-3.3.1.jar

avatar
Contributor

Thnks but nothing happened...same error again.

 

Hey can you please mention steps of how to insert data into hbase table through hive. I think i am wrong somewhere.

 

Please help !!!!!

 

avatar
Expert Contributor

1.


 

bin/hive --auxpath /hadoop/projects/hive-0.9.0/lib/hive-hbase-handler-0.9.0.jar,/hadoop/projects/hive-0.9.0/lib/hbase-0.92.0.jar,/hadoop/projects/hive-0.9.0/lib/zookeeper-3.3.4.jar,/hadoop/projects/hive-0.9.0/lib/guava-r09.jar -hiveconf hbase.master=localhost:60000

 

 

2.
 

hive> CREATE TABLE hbase_table_1(key int, value string) STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler' WITH SERDEPROPERTIES ("hbase.columns.mapping" = ":key,cf1:val") TBLPROPERTIES ("hbase.table.name" = "xyz");

 

 

3.


 

hive> INSERT OVERWRITE TABLE hbase_table_1 SELECT * FROM pokes WHERE foo=1;

 

avatar
Contributor

I have followed same steps as you have mentioned. But still error came. Please verify it & please provide me some another solution.

 

  • bin/hive --auxpath /usr/lib/hive/lib/hive-hbase-handler-0.1 1.0.1.3.0.0-107.jar, /usr/lib/hive/lib/hbase-0.94.6.1.3.0.0-107-security.jar, /u sr/lib/hive/lib/zookeeper-3.4.5.1.3.0.0-107.jar, /usr/lib/hive/lib/guava-11.0.2. jar, -hiveconf hbase.master=localhost:60000


13/08/15 22:42:42 WARN conf.HiveConf: DEPRECATED: Configuration property hive.me tastore.local no longer has any effect. Make sure to provide a valid value for h ive.metastore.uris if you are connecting to a remote metastore.

Logging initialized using configuration in jar:file:/usr/lib/hive/lib/hive-commo n-0.11.0.1.3.0.0-107.jar!/hive-log4j.properties

 

  • hive> CREATE TABLE hbase_table_5(key int, value string) STORED BY 'org.apache.ha doop.hive.hbase.HBaseStorageHandler' WITH SERDEPROPERTIES ("hbase.columns.mappin g" = ":key,cf1:val") TBLPROPERTIES ("hbase.table.name" = "xxx");

OK
Time taken: 4.99 seconds

 

  • hive> INSERT OVERWRITE TABLE hbase_table_5 SELECT * FROM user WHERE key=1;

Total MapReduce jobs = 1
Launching Job 1 out of 1
Number of reduce tasks is set to 0 since there's no reduce operator
Starting Job = job_201307292222_0028, Tracking URL = http://cloudera:50030/jobdet ails.jsp?jobid=job_201307292222_0028
Kill Command = /usr/lib/hadoop/libexec/../bin/hadoop job -kill job_201307292222 _0028
Hadoop job information for Stage-0: number of mappers: 1; number of reducers: 0
2013-08-15 22:44:08,760 Stage-0 map = 0%, reduce = 0%
2013-08-15 22:44:29,849 Stage-0 map = 100%, reduce = 100%
Ended Job = job_201307292222_0028 with errors
Error during job, obtaining debugging information...
Job Tracking URL: http://cloudera:50030/jobdetails.jsp?jobid=job_201307292222_002 8
Examining task ID: task_201307292222_0028_m_000002 (and more) from job job_20130 7292222_0028

Task with the most failures(4):
-----
Task ID:
task_201307292222_0028_m_000000

URL:
http://cloudera:50030/taskdetails.jsp?jobid=job_201307292222_0028&tipid=task_20 1307292222_0028_m_000000
-----
Diagnostic Messages for this Task:
java.lang.RuntimeException: Error in configuring object
at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.jav a:93)
at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:6 4)
at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.ja va:117)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:425)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:365)
at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInforma tion.java:1232)
at org.apache.hadoop.mapred.Child.main(Child.java:249)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl. java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces sorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.jav a:88)
... 9 more
Caused by: java.lang.RuntimeException: Error in configuring object
at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.jav a:93)
at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:6 4)
at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.ja va:117)
at org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:34)
... 14 more
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl. java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces sorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.jav a:88)
... 17 more
Caused by: java.lang.RuntimeException: Map operator initialization failed
at org.apache.hadoop.hive.ql.exec.ExecMapper.configure(ExecMapper.java:1 21)
... 22 more
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.NullPoint erException
at org.apache.hadoop.hive.ql.exec.FileSinkOperator.initializeOp(FileSink Operator.java:385)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:375)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:451)
at org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.j ava:407)
at org.apache.hadoop.hive.ql.exec.SelectOperator.initializeOp(SelectOper ator.java:62)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:375)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:451)
at org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.j ava:407)
at org.apache.hadoop.hive.ql.exec.FilterOperator.initializeOp(FilterOper ator.java:78)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:375)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:451)
at org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.j ava:407)
at org.apache.hadoop.hive.ql.exec.TableScanOperator.initializeOp(TableSc anOperator.java:186)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:375)
at org.apache.hadoop.hive.ql.exec.MapOperator.initializeOp(MapOperator.j ava:543)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:375)
at org.apache.hadoop.hive.ql.exec.ExecMapper.configure(ExecMapper.java:1 00)
... 22 more
Caused by: java.lang.NullPointerException
at org.apache.hadoop.hive.ql.exec.FileSinkOperator.initializeOp(FileSink Operator.java:322)
... 38 more


FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.MapRe dTask
MapReduce Jobs Launched:
Job 0: Map: 1 HDFS Read: 0 HDFS Write: 0 FAIL
Total MapReduce CPU Time Spent: 0 msec

avatar
Expert Contributor

Some of the jars are not added in the hive --auxpath command.

 

Also add /hbase-0.94.1/hbase-0.94.6.jar and hive_contrib.jar. The version may be different.

avatar
Contributor

Above problem has been solved by adding hbase.jar in /usr/lib/hadoop/lib & add some jar files like hive-hbase-handler.jar, hbase.jar, guava.jar, zookeeper.jar, hive-contrib.jar in auxpth in hive-site.xml..

 

Thanks

Surbhi