Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Error when copying table from Hive to HBase.

avatar
Explorer

Hello. I am trying to copy an existing table from Hive to a new table in HBase with the following query:

--------------------------------------------------

CREATE TABLE hbase_lineitem (part_key int, item_id int, category_id int, dept_id int, quantity int, price double, discount double, tax double, flag_1 string, flag_2 string)

STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'

WITH SERDEPROPERTIES ("hbase.columns.mapping" = ":part_key,lineitem:item_id,lineitem:category_id,lineitem:dept_id,lineitem:quantity,lineitem:price,lineitem:discount,lineitem:tax,lineitem:flag_1,lineitem:flag_2");

INSERT OVERWRITE TABLE hbase_lineitem SELECT * FROM lineitem;

---------------------------------------------------

However, I am receiving the following error:

Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. java.lang.RuntimeException: MetaException(message:org.apache.hadoop.hive.serde2.SerDeException org.apache.hadoop.hive.hbase.HBaseSerDe: columns has 10 elements while hbase.columns.mapping has 11 elements (counting the key if implicit))

I only count 10 elements in both of my lists, so can anyone help me figure out what mistake I am making? Thanks.

1 ACCEPTED SOLUTION

avatar
Super Guru

I would guess that the problem is that you are not explicitly providing a :key column which makes your value actually have 10+1 entries instead of just 10.

You will want to map one of the Hive columns you are selecting into the HBase rowKey. This is done by including a ":key" entry in the value for "hbase.columns.mapping". This is what defines uniqueness in each record.

See https://cwiki.apache.org/confluence/display/Hive/HBaseIntegration#HBaseIntegration-ColumnMapping for more information.

View solution in original post

3 REPLIES 3

avatar
Super Guru

I would guess that the problem is that you are not explicitly providing a :key column which makes your value actually have 10+1 entries instead of just 10.

You will want to map one of the Hive columns you are selecting into the HBase rowKey. This is done by including a ":key" entry in the value for "hbase.columns.mapping". This is what defines uniqueness in each record.

See https://cwiki.apache.org/confluence/display/Hive/HBaseIntegration#HBaseIntegration-ColumnMapping for more information.

avatar
Explorer

This fixed the issue, appreciate it!

avatar
Super Collaborator

If I correctly understand, you need to change :part_key to :key.