Hi, I tried to create a table in Hive with DruidStorageHandler using the following command: CREATE TABLE druid_table (`__time` timestamp,`userid`string,`num_l`float)STORED BY 'org.apache.hadoop.hive.druid.DruidStorageHandler'TBLPROPERTIES ("druid.segment.granularity" = "DAY", "druid.query.granularity" = "DAY") as select * from poc.test; It failed with the below error: at org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.processVectorGroup(ReduceRecordSource.java:489) at org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.pushRecordVector(ReduceRecordSource.java:397) ... 18 more Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.NullPointerException: Data source name is null at org.apache.hadoop.hive.ql.exec.FileSinkOperator.createBucketFiles(FileSinkOperator.java:564) at org.apache.hadoop.hive.ql.exec.FileSinkOperator.process(FileSinkOperator.java:664) at org.apache.hadoop.hive.ql.exec.vector.VectorFileSinkOperator.process(VectorFileSinkOperator.java:101) at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:897) at org.apache.hadoop.hive.ql.exec.vector.VectorSelectOperator.process(VectorSelectOperator.java:145) at org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.processVectorGroup(ReduceRecordSource.java:478) ... 19 more Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.NullPointerException: Data source name is null at org.apache.hadoop.hive.ql.io.HiveFileFormatUtils.getHiveRecordWriter(HiveFileFormatUtils.java:272) at org.apache.hadoop.hive.ql.exec.FileSinkOperator.createBucketForFileIdx(FileSinkOperator.java:609) at org.apache.hadoop.hive.ql.exec.FileSinkOperator.createBucketFiles(FileSinkOperator.java:553) ... 24 more Caused by: java.lang.NullPointerException: Data source name is null at org.apache.hive.druid.com.google.common.base.Preconditions.checkNotNull(Preconditions.java:229) at org.apache.hadoop.hive.druid.io.DruidOutputFormat.getHiveRecordWriter(DruidOutputFormat.java:187) at org.apache.hadoop.hive.ql.io.HiveFileFormatUtils.getRecordWriter(HiveFileFormatUtils.java:284) at org.apache.hadoop.hive.ql.io.HiveFileFormatUtils.getHiveRecordWriter(HiveFileFormatUtils.java:269) ... 26 more ]], Vertex did not succeed due to OWN_TASK_FAILURE, failedTasks:1 killedTasks:0, Vertex vertex_1525394716493_0623_23_02 [Reducer 3] killed/failed due to:OWN_TASK_FAILURE]DAG did not succeed due to VERTEX_FAILURE. failedVertices:1 killedVertices:0 (state=08S01,code=2) I added druid.datasource property to TBLPROPERTIES and ran it once again: CREATE TABLE druid_table (`__time` timestamp,`userid`string,`num_l`float)STORED BY 'org.apache.hadoop.hive.druid.DruidStorageHandler'TBLPROPERTIES ("druid.segment.granularity" = "DAY", "druid.query.granularity" = "DAY", "druid.datasource"="dummy") as select * from poc.test; This was successful. Now I want to increment this table by adding new data. insert into table druid_table select * from poc.test2; This fails with the following error: Error: Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. MetaException(message:INSERT INTO statement is not allowed by druid storage handler) (state=08S01,code=1) The documentation says that INSERT INTO statements are supported. Am I missing something here? Please let me know in case any additional details are required. Many Thanks. ... View more

I'm trying to create a table in hive with DruidStorageHandler as follows: CREATE TABLE druid_table (`__time` timestamp,`userid`string,`num_l`float)STORED BY 'org.apache.hadoop.hive.druid.DruidStorageHandler'TBLPROPERTIES ("druid.segment.granularity"="DAY"); It fails with the following error: Error: Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. java.lang.RuntimeException: java.io.FileNotFoundException: File /tmp/workingDirectory/.staging-hive_20180507130925_227f2e48-d049-464e-b2cd-43009b3398b3/segmentsDescriptorDir does not exist. (state=08S01,code=1) I went through the Source code of DruidStorageHandler.Java and to my surprise I wasn't able to see any dfs.mkdir call for the above "segmentsDescriptorDir" directory. I have got this syntax straight from the documentation. CREATE TABLE druid_table_1 (`__time` TIMESTAMP, `dimension1` STRING, `dimension2` STRING, `metric1` INT, `metric2` FLOAT) STORED BY 'org.apache.hadoop.hive.druid.DruidStorageHandler'; Has someone faced a similar problem earlier? Thanks in Advance. Megh ... View more

Hello, I would like to know if there is any other solution to Hive metadata replication across clusters other than Apache Falcon Hive Mirror. 1) How does Hive Mirroring by Falcon work internally (On a High level)? 2) Can the same be achieved by backup and restore of metastore DB on a different server? 3) How can HDFS storage dependencies for tables be managed in case of Metastore DB backup and restore? Any help on the above questions is much appreciated. Thanks & Regards, Megh ... View more