Reply
Explorer
Posts: 16
Registered: ‎08-13-2013

Hive hbase integration

[ Edited ]

I have integrated hive & hbase successfully. The below query run successfully & created table in both hive & hbase.

 

CREATE TABLE hbase_test1(key int, value string)
STORED BY ‘org.apache.hadoop.hive.hbase.HBaseStorageHandler’
WITH SERDEPROPERTIES (“hbase.columns.mapping” = “:key,cf1:val”)
TBLPROPERTIES (“hbase.table.name” = “test1″);

 

Now i want to insert data into hbase table through hive. I have created a hive table name user.

 

create table user(key int,value string)
ROW FORMAT DELIMITED FIELDS TERMINATED BY ‘,’ LINES TERMINATED BY ‘\n’ STORED AS TEXTFILE;

Successfully inserted data into hive table through this query.
LOAD DATA LOCAL INPATH ‘/home/user.txt’ INTO TABLE user;

 

Now i run this query into hive for inserting data into hbase table.

INSERT OVERWRITE TABLE hbase_test1 SELECT * FROM user where key =1;

 

Below error came when i run insert command:

 

hive> INSERT OVERWRITE TABLE hbase_test2 SELECT * FROM user WHERE key=1;

Total MapReduce jobs = 1
Launching Job 1 out of 1
Number of reduce tasks is set to 0 since there's no reduce operator
Starting Job = job_201308130336_0008, Tracking URL = http://localhost.localdomai                                                                         ... n:50030/jobdetails.jsp?jobid=job_201308130336_0008
Kill Command = /usr/lib/hadoop/bin/hadoop job  -kill job_201308130336_0008
Hadoop job information for Stage-0: number of mappers: 1; number of reducers: 0
2013-08-14 02:10:54,638 Stage-0 map = 0%,  reduce = 0%
2013-08-14 02:11:29,929 Stage-0 map = 100%,  reduce = 100%
Ended Job = job_201308130336_0008 with errors
Error during job, obtaining debugging information...
Job Tracking URL: http://localhost.localdomain:50030/jobdetails.jsp?jobid=job_201308130336_0008
Examining task ID: task_201308130336_0008_m_000002 (and more) from job job_201308130336_0008

Task with the most failures(4):
-----
Task ID:
  task_201308130336_0008_m_000000

URL:
  http://localhost.localdomain:50030/taskdetails.jsp?jobid=job_201308130336_0008&tipid=task_2013081303...
-----
Diagnostic Messages for this Task:
java.lang.RuntimeException: Error in configuring object
        at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:106)
        at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:72)
        at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:130)
        at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:413)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:332)
        at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:396)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
        at org.apache.hadoop.mapred.Child.main(Child.java:262)
Caused by: java.lang.reflect.InvocationTargetException
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.ja

FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.MapRedTask
MapReduce Jobs Launched:
Job 0: Map: 1   HDFS Read: 0 HDFS Write: 0 FAIL
Total MapReduce CPU Time Spent: 0 msec
hive>

 

Please help me by providing solution as soon as possible.

 

Thanks & Regards
Surbhi Singh

 

Expert Contributor
Posts: 63
Registered: ‎08-06-2013

Re: Hive hbase integration

For Hive HBase integration:

 

1.
bin/hive --auxpath /hadoop/projects/hive-0.9.0/lib/hive-hbase-handler-0.9.0.jar,/hadoop/projects/hive-0.9.0/lib/hbase-0.92.0.jar,/hadoop/projects/hive-0.9.0/lib/zookeeper-3.3.4.jar,/hadoop/projects/hive-0.9.0/lib/guava-r09.jar -hiveconf hbase.master=localhost:60000

2.
hive> CREATE TABLE hbase_table_1(key int, value string) STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler' WITH SERDEPROPERTIES ("hbase.columns.mapping" = ":key,cf1:val") TBLPROPERTIES ("hbase.table.name" = "xyz");

3.
hive> INSERT OVERWRITE TABLE hbase_table_1 SELECT * FROM pokes WHERE foo=1;

Highlighted
New Contributor
Posts: 7
Registered: ‎07-30-2013

Re: Hive hbase integration

FWIW I reproduced errors using the steps provided using quickstart VM 4.2.0:

 

CREATE TABLE hbase_test1(key int, value string)
STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'
WITH SERDEPROPERTIES ("hbase.columns.mapping" = ":key,cf1:val")
TBLPROPERTIES ("hbase.table.name" = "test1");

create table user(key int,value string)
ROW FORMAT DELIMITED FIELDS TERMINATED BY ',' 
LINES TERMINATED BY '\n' 
STORED AS TEXTFILE;

LOAD DATA LOCAL INPATH '/home/cloudera/user.txt' INTO TABLE user;

INSERT OVERWRITE TABLE hbase_test1 SELECT * FROM user where key =1;

 

Here's the output:

 

hive> create table user(key int,value string)
    > ROW FORMAT DELIMITED FIELDS TERMINATED BY ','
    > LINES TERMINATED BY '\n'
    > STORED AS TEXTFILE;
OK
Time taken: 0.121 seconds
hive> LOAD DATA LOCAL INPATH '/home/cloudera/user.txt' INTO TABLE user;
Copying data from file:/home/cloudera/user.txt
Copying file: file:/home/cloudera/user.txt
Loading data to table default.user
Table default.user stats: [num_partitions: 0, num_files: 1, num_rows: 0, total_size: 14, raw_data_size: 0]
OK
Time taken: 1.453 seconds
hive> select * from user;
OK
1       val1
2       val2
Time taken: 0.353 seconds
hive> INSERT OVERWRITE TABLE hbase_test1 SELECT * FROM user where key =1;
Total MapReduce jobs = 1
Launching Job 1 out of 1
Number of reduce tasks is set to 0 since there's no reduce operator
Starting Job = job_201308151236_0003, Tracking URL = http://localhost.localdomain:50030/jobdetails.jsp?jobid=job_201308151236_0003
Kill Command = /usr/lib/hadoop/bin/hadoop job  -kill job_201308151236_0003
Hadoop job information for Stage-0: number of mappers: 1; number of reducers: 0
2013-08-15 13:09:50,193 Stage-0 map = 0%,  reduce = 0%
2013-08-15 13:10:26,530 Stage-0 map = 100%,  reduce = 100%
Ended Job = job_201308151236_0003 with errors
Error during job, obtaining debugging information...
Job Tracking URL: http://localhost.localdomain:50030/jobdetails.jsp?jobid=job_201308151236_0003
Examining task ID: task_201308151236_0003_m_000002 (and more) from job job_201308151236_0003

Task with the most failures(4):
-----
Task ID:
  task_201308151236_0003_m_000000

URL:
  http://localhost.localdomain:50030/taskdetails.jsp?jobid=job_201308151236_0003&tipid=task_2013081512...
-----
Diagnostic Messages for this Task:
java.lang.RuntimeException: Error in configuring object
        at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:106)
        at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:72)
        at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:130)
        at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:414)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:333)
        at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:396)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
        at org.apache.hadoop.mapred.Child.main(Child.java:262)
Caused by: java.lang.reflect.InvocationTargetException
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.ja

FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.MapRedTask
MapReduce Jobs Launched:
Job 0: Map: 1   HDFS Read: 0 HDFS Write: 0 FAIL
Total MapReduce CPU Time Spent: 0 msec

 

Mat.