question Re: Loading data into HBase using Pig script in Archives of Support Questions (Read Only)

Loading data into HBase using Pig script

maheshmsh88 — Tue, 04 Oct 2016 20:36:12 GMT

Hi, I'm trying to load simple dataset into HBase using Pig script. I have referred few websites but some are using org.apache.pig.backend.hadoop.hbase.HBaseStorage and in som website they used org.apache.hadoop.hive.hbase.HBaseStorageHandler. Can someone please let me know which is the correct method and what is the difference between these two.?

Re: Loading data into HBase using Pig script

sunile_manjee — Tue, 04 Oct 2016 21:31:25 GMT

@Mahesh Mallikarjunappa When you use pig to load into hbase use org.apache.pig.backend.hadoop.hbase.HBaseStorage, when you use hive to load into hbase use used org.apache.hadoop.hive.hbase.HBaseStorageHandler. Both are for those specific technolgoies.

Re: Loading data into HBase using Pig script

StevenONeill — Wed, 05 Oct 2016 05:48:27 GMT

@Mahesh Mallikarjunappa The Hive serde can be used to load data into HBase via Hive. A simple example follows:

-- Create a hive-managed HBase table
CREATE TABLE MyHBaseTable(MyKey string, Col1 string, Col2 string)
   STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'
   WITH SERDEPROPERTIES ("hbase.columns.mapping" = ":key,colfam:col1,colfam:col2")
   TBLPROPERTIES("hbase.table.name" = "MyNamespace:MyTable")
;

-- Insert data into it
INSERT INTO TABLE MyHBaseTable
   SELECT SourceKey, SourceCol1, SourceCol2
   FROM SourceHiveTable
;

And from Pig, you can read form that same source and write to that same target using the HBase serde as follows:

pig -useHCatalog -f script.pig

Where script.pig is as below

RawData = LOAD 'SourceHiveTable'
          USING org.apache.hive.hcatalog.pig.HCatLoader();

KeepColumns = FOREACH RawData
              GENERATE SourceKey, SourceCol1, SourceCol2;

STORE KeepColumns
INTO 'hbase://MyNamespace:MyTable'
USING org.apache.pig.backend.hadoop.hbase.HBaseStorage('colfam:col1,colfam:col2');

Note: You don't specify the key in the STORE statement - the first column is always the key.

Hope this helps!

Re: Loading data into HBase using Pig script

amit_dass — Wed, 05 Oct 2016 20:41:05 GMT

TBLPROPERTIES("hbase.table.name"="MyNamespace:MyTable")

I am getting "FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. MetaException(message:org.apache.hadoop.hbase.NamespaceNotFoundException: org.apache.hadoop.hbase.NamespaceNotFoundException: MyNamespace" Please help us here

Re: Loading data into HBase using Pig script

maheshmsh88 — Wed, 05 Oct 2016 21:25:00 GMT

@Amit Dass, first create table in Hive inside database and that table name should match with TBLPROPERTIES table name.

Re: Loading data into HBase using Pig script

smedoji — Tue, 03 Jul 2018 12:25:52 GMT

In hbase, create a namespace and then create a table to avoid this error.