Reply
Explorer
Posts: 16
Registered: ‎08-13-2013
Accepted Solution

Problem in inserting data into hbase table through hive

Trying to insert data into hbase table through hive.:

Create table query:
CREATE TABLE hbase_testing(id int, name string, password string) STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler' WITH SERDEPROPERTIES ("hbase.columns.mapping" = ":key,cf1:val") TBLPROPERTIES ("hbase.table.name" = "hbase_testing");

Insert Query:

INSERT OVERWRITE TABLE hbase_test2 SELECT * FROM user WHERE key=1;

 

Error:

java.lang.RuntimeException: Error in configuring object at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:106) at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:72) at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:130) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:413) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:332) at org.apache.hadoop.mapred.Child$4.run(Child.java:268) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408) at org.apache.hadoop.mapred.Child.main(Child.java:262) Caused by: java.lang.reflect.InvocationTargetException at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.ja java.lang.RuntimeException: Error in configuring object at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:106) at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:72) at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:130) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:413) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:332) at org.apache.hadoop.mapred.Child$4.run(Child.java:268) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408) at org.apache.hadoop.mapred.Child.main(Child.java:262) Caused by: java.lang.reflect.InvocationTargetException at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.ja java.lang.RuntimeException: Error in configuring object at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:106) at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:72) at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:130) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:413) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:332) at org.apache.hadoop.mapred.Child$4.run(Child.java:268) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408) at org.apache.hadoop.mapred.Child.main(Child.java:262) Caused by: java.lang.reflect.InvocationTargetException at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.ja java.lang.RuntimeException: Error in configuring object at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:106) at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:72) at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:130) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:413) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:332) at org.apache.hadoop.mapred.Child$4.run(Child.java:268) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408) at org.apache.hadoop.mapred.Child.main(Child.java:262) Caused by: java.lang.reflect.InvocationTargetException at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.ja
Expert Contributor
Posts: 63
Registered: ‎08-06-2013

Re: Problem in inserting data into hbase table through hive

For Hive add the following JARs (version may be different) to classpath.


 

/opt/hive/lib/hive-hbase-handler-0.8.0-SNAPSHOT.jar
/opt/hive/lib/hbase-0.90.1-SNAPSHOT.jar
/opt/hive/lib/zookeeper-3.3.1.jar

 

 

Add to classpath either by adding to the Hadoop lib dir (/usr/lib/hadoop/lib) or with the add jar command.

 

add jar /opt/hive/lib/hive-hbase-handler-0.8.0-SNAPSHOT.jar
add jar /opt/hive/lib/hbase-0.90.1-SNAPSHOT.jar
add jar /opt/hive/lib/zookeeper-3.3.1.jar

Explorer
Posts: 16
Registered: ‎08-13-2013

Re: Problem in inserting data into hbase table through hive

Thnks but nothing happened...same error again.

 

Hey can you please mention steps of how to insert data into hbase table through hive. I think i am wrong somewhere.

 

Please help !!!!!

 

Highlighted
Expert Contributor
Posts: 63
Registered: ‎08-06-2013

Re: Problem in inserting data into hbase table through hive

1.


 

bin/hive --auxpath /hadoop/projects/hive-0.9.0/lib/hive-hbase-handler-0.9.0.jar,/hadoop/projects/hive-0.9.0/lib/hbase-0.92.0.jar,/hadoop/projects/hive-0.9.0/lib/zookeeper-3.3.4.jar,/hadoop/projects/hive-0.9.0/lib/guava-r09.jar -hiveconf hbase.master=localhost:60000

 

 

2.
 

hive> CREATE TABLE hbase_table_1(key int, value string) STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler' WITH SERDEPROPERTIES ("hbase.columns.mapping" = ":key,cf1:val") TBLPROPERTIES ("hbase.table.name" = "xyz");

 

 

3.


 

hive> INSERT OVERWRITE TABLE hbase_table_1 SELECT * FROM pokes WHERE foo=1;

 

Explorer
Posts: 16
Registered: ‎08-13-2013

Re: Problem in inserting data into hbase table through hive

[ Edited ]

I have followed same steps as you have mentioned. But still error came. Please verify it & please provide me some another solution.

 

  • bin/hive --auxpath /usr/lib/hive/lib/hive-hbase-handler-0.1 1.0.1.3.0.0-107.jar, /usr/lib/hive/lib/hbase-0.94.6.1.3.0.0-107-security.jar, /u sr/lib/hive/lib/zookeeper-3.4.5.1.3.0.0-107.jar, /usr/lib/hive/lib/guava-11.0.2. jar, -hiveconf hbase.master=localhost:60000


13/08/15 22:42:42 WARN conf.HiveConf: DEPRECATED: Configuration property hive.me tastore.local no longer has any effect. Make sure to provide a valid value for h ive.metastore.uris if you are connecting to a remote metastore.

Logging initialized using configuration in jar:file:/usr/lib/hive/lib/hive-commo n-0.11.0.1.3.0.0-107.jar!/hive-log4j.properties

 

  • hive> CREATE TABLE hbase_table_5(key int, value string) STORED BY 'org.apache.ha doop.hive.hbase.HBaseStorageHandler' WITH SERDEPROPERTIES ("hbase.columns.mappin g" = ":key,cf1:val") TBLPROPERTIES ("hbase.table.name" = "xxx");

OK
Time taken: 4.99 seconds

 

  • hive> INSERT OVERWRITE TABLE hbase_table_5 SELECT * FROM user WHERE key=1;

Total MapReduce jobs = 1
Launching Job 1 out of 1
Number of reduce tasks is set to 0 since there's no reduce operator
Starting Job = job_201307292222_0028, Tracking URL = http://cloudera:50030/jobdet ails.jsp?jobid=job_201307292222_0028
Kill Command = /usr/lib/hadoop/libexec/../bin/hadoop job -kill job_201307292222 _0028
Hadoop job information for Stage-0: number of mappers: 1; number of reducers: 0
2013-08-15 22:44:08,760 Stage-0 map = 0%, reduce = 0%
2013-08-15 22:44:29,849 Stage-0 map = 100%, reduce = 100%
Ended Job = job_201307292222_0028 with errors
Error during job, obtaining debugging information...
Job Tracking URL: http://cloudera:50030/jobdetails.jsp?jobid=job_201307292222_002 8
Examining task ID: task_201307292222_0028_m_000002 (and more) from job job_20130 7292222_0028

Task with the most failures(4):
-----
Task ID:
task_201307292222_0028_m_000000

URL:
http://cloudera:50030/taskdetails.jsp?jobid=job_201307292222_0028&tipid=task_20 1307292222_0028_m_000000
-----
Diagnostic Messages for this Task:
java.lang.RuntimeException: Error in configuring object
at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.jav a:93)
at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:6 4)
at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.ja va:117)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:425)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:365)
at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInforma tion.java:1232)
at org.apache.hadoop.mapred.Child.main(Child.java:249)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl. java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces sorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.jav a:88)
... 9 more
Caused by: java.lang.RuntimeException: Error in configuring object
at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.jav a:93)
at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:6 4)
at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.ja va:117)
at org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:34)
... 14 more
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl. java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces sorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.jav a:88)
... 17 more
Caused by: java.lang.RuntimeException: Map operator initialization failed
at org.apache.hadoop.hive.ql.exec.ExecMapper.configure(ExecMapper.java:1 21)
... 22 more
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.NullPoint erException
at org.apache.hadoop.hive.ql.exec.FileSinkOperator.initializeOp(FileSink Operator.java:385)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:375)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:451)
at org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.j ava:407)
at org.apache.hadoop.hive.ql.exec.SelectOperator.initializeOp(SelectOper ator.java:62)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:375)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:451)
at org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.j ava:407)
at org.apache.hadoop.hive.ql.exec.FilterOperator.initializeOp(FilterOper ator.java:78)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:375)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:451)
at org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.j ava:407)
at org.apache.hadoop.hive.ql.exec.TableScanOperator.initializeOp(TableSc anOperator.java:186)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:375)
at org.apache.hadoop.hive.ql.exec.MapOperator.initializeOp(MapOperator.j ava:543)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:375)
at org.apache.hadoop.hive.ql.exec.ExecMapper.configure(ExecMapper.java:1 00)
... 22 more
Caused by: java.lang.NullPointerException
at org.apache.hadoop.hive.ql.exec.FileSinkOperator.initializeOp(FileSink Operator.java:322)
... 38 more


FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.MapRe dTask
MapReduce Jobs Launched:
Job 0: Map: 1 HDFS Read: 0 HDFS Write: 0 FAIL
Total MapReduce CPU Time Spent: 0 msec

Expert Contributor
Posts: 63
Registered: ‎08-06-2013

Re: Problem in inserting data into hbase table through hive

Some of the jars are not added in the hive --auxpath command.

 

Also add /hbase-0.94.1/hbase-0.94.6.jar and hive_contrib.jar. The version may be different.

Explorer
Posts: 16
Registered: ‎08-13-2013

Re: Problem in inserting data into hbase table through hive

Above problem has been solved by adding hbase.jar in /usr/lib/hadoop/lib & add some jar files like hive-hbase-handler.jar, hbase.jar, guava.jar, zookeeper.jar, hive-contrib.jar in auxpth in hive-site.xml..

 

Thanks

Surbhi