Our Community is getting an upgrade! To get everything ready for the relaunch, we’ll be placing the site in read-only mode starting September 21st.
We really appreciate your understanding while we get things set up behind the scenes. Catch up on all the exciting details about the move here.
Need help or have questions? Drop us a line at [email protected]
Created 05-28-2024 09:45 AM
Hello there!
We create a hive external table that refers to a HBase table:
CREATE EXTERNAL TABLE default.c_COUNTRIES_AP
( key BINARY,
CF1_COUNTRY_ID_1 BINARY,
CF1_COUNTRY_NAME_2 BINARY,
CF1_COUNTRY_REGION_3 BINARY,
CF1_COUNTRY_SUBREGION_4 BINARY )
STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'
WITH SERDEPROPERTIES (
"hbase.columns.mapping" = ":key,CF1:COUNTRY_ID#s,CF1:COUNTRY_NAME#s,CF1:COUNTRY_REGION#s,CF1:COUNTRY_SUBREGION#s"
)
TBLPROPERTIES ("hbase.table.name" = "COUNTRIES_TGT")
In Hive, we insert some data into the hive external table:
insert into c_countries_ap values ('ZA', 'ZA', 'South Africa', 'Africa', 'Africa');
select * from c_countries_ap shows correct data is inserted
But when we go to HBase shell, and do a SCAN ‘COUNTRIES_TGT’, it shows:
hbase:063:0> scan 'COUNTRIES_TGT'
ROW COLUMN+CELL
WkE= column=CF1:COUNTRY_ID, timestamp=1716416439777, value=WkE=
WkE= column=CF1:COUNTRY_NAME, timestamp=1716416439777, value=U291dGggQWZyaWNh
WkE= column=CF1:COUNTRY_REGION, timestamp=1716416439777, value=QWZyaWNh
WkE= column=CF1:COUNTRY_SUBREGION, timestamp=1716416439777, value=QWZyaWNh
The data in HBase is all base64 encoded.
If we do SQOOP import into HBase table, the data in HBase table is not encoded.
This only happens when data is inserted from Hive table through SerDe.
We are wondering if there are any configuration parameters need to changed?
Created 05-28-2024 11:06 AM
@Marks_08 The encoding is being performed by HBase SerDe, for the binary data. Could you change the col data type to STRING if your data does not really require binary storage?
Created 05-28-2024 11:06 AM
@Marks_08 The encoding is being performed by HBase SerDe, for the binary data. Could you change the col data type to STRING if your data does not really require binary storage?
Created 05-28-2024 12:55 PM
Thanks @smruti
I changed data type to String and data is stored without encoding as I want.
Is there any documentation where I can find "Why the encoding is being performed by HBase SerDe, for the binary data"?
regards,
Marks