I'm plying around with HBase tables that are managed by Hive. Therefore I run the following commands in Zeppelin: %hbase
# Create Table with 1 CF and 10 Regions
create 'my_test', {NAME => 'cf1', VERSIONS => 3}, {SPLITS => ['00', '01', '02', '03', '04', '05', '06', '07', '08', '09']}
After creating the HBase table, I create a external Hive Table with HBaseStorageHandler: %hive
-- CREATE EXTERNAL HIVE MANAGED HBASE TABLE!
CREATE EXTERNAL TABLE dmueller.my_test(
key String,
hashvalue int,
valuelist String
)
STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'
WITH SERDEPROPERTIES ("hbase.columns.mapping" = ":key,cf1:hashValue,cf1:valueList")
TBLPROPERTIES("hbase.table.name" = "my_test", "hbase.mapred.output.outputtable" = "my_test")
Then I fill the HBase table by reading data (1000 rows) from another Hive table into this external Hive Table: %hive
-- INSERT DATA INTO THE MANAGED TABLE
INSERT OVERWRITE TABLE dmueller.my_test SELECT concat_ws("_", testname, lotnrc, testnumber, teststufe) as key, hashvalue, valuelist from dmueller.hivetable limit 1000
The problem here is, that the INSERT statement is too slow. It takes around 4 minutes to insert the new values (1000 rows with 6 column to read)! The dmueller.hivetable is a partitioned ORC table with around 60 mio rows and ~500 GB of ORC file size in HDFS. How can this INSERT statement made faster? What am I doing wrong? Update - Some more information: I'm using Hive with Tez. Writing less data takes less time, e.g. one row takes already ~30 sec. (so it seems to be an exponential runtime). Reading the data (e.g. the 1000 rows) directly via Zeppelin Hive (JDBC) interpreter takes ~2 sec. Re-running the same INSERT statement (e.g. with 1000 rows) takes always the same time (+/- a few seconds of course).
... View more
