Member since
01-05-2016
15
Posts
0
Kudos Received
0
Solutions
09-24-2018
07:32 AM
Hi @Xue Chen! I have a couple of questions. 1) How long do you see it in initiated mode? 2) Does the compaction ever happen? Normally, I've faced issues with compaction if I run the compaction with a user other than hive. Also, the owner of the table should also be hive. You could try setting the owner of the table to hive and run the compactions as hive user if not done already. Please let me know if that helps you! Regards, Megh
... View more
05-10-2018
10:29 AM
@ln chari can we give this a try in your setup? The OP has reverted and installed 2.6.2.0 because 2.6.4.0 didn't work.
... View more
05-09-2018
12:29 PM
@Rahul Pathak can you please let me know if Hive also provides capability of deleting/dropping a particular segment from a table with druidstoragehandler? Since we're not having the latest HDP stack (2.6.4) with latest Druid version, we're unable to test the same. Would really appreciate your help in this regard.
... View more
05-09-2018
07:17 AM
Hi @Rahul Pathak, I have deployed HDP 2.6.1 and running the query using hiveServer2 Interactive. The table poc.test2 is in parquet format.
... View more
05-08-2018
09:11 AM
Hi, I tried to create a table in Hive with DruidStorageHandler using the following command: CREATE TABLE druid_table (`__time` timestamp,`userid`string,`num_l`float)STORED BY 'org.apache.hadoop.hive.druid.DruidStorageHandler'TBLPROPERTIES ("druid.segment.granularity" = "DAY", "druid.query.granularity" = "DAY") as select * from poc.test; It failed with the below error: at org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.processVectorGroup(ReduceRecordSource.java:489)
at org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.pushRecordVector(ReduceRecordSource.java:397)
... 18 more
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.NullPointerException: Data source name is null
at org.apache.hadoop.hive.ql.exec.FileSinkOperator.createBucketFiles(FileSinkOperator.java:564)
at org.apache.hadoop.hive.ql.exec.FileSinkOperator.process(FileSinkOperator.java:664)
at org.apache.hadoop.hive.ql.exec.vector.VectorFileSinkOperator.process(VectorFileSinkOperator.java:101)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:897)
at org.apache.hadoop.hive.ql.exec.vector.VectorSelectOperator.process(VectorSelectOperator.java:145)
at org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.processVectorGroup(ReduceRecordSource.java:478)
... 19 more
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.NullPointerException: Data source name is null
at org.apache.hadoop.hive.ql.io.HiveFileFormatUtils.getHiveRecordWriter(HiveFileFormatUtils.java:272)
at org.apache.hadoop.hive.ql.exec.FileSinkOperator.createBucketForFileIdx(FileSinkOperator.java:609)
at org.apache.hadoop.hive.ql.exec.FileSinkOperator.createBucketFiles(FileSinkOperator.java:553)
... 24 more
Caused by: java.lang.NullPointerException: Data source name is null
at org.apache.hive.druid.com.google.common.base.Preconditions.checkNotNull(Preconditions.java:229)
at org.apache.hadoop.hive.druid.io.DruidOutputFormat.getHiveRecordWriter(DruidOutputFormat.java:187)
at org.apache.hadoop.hive.ql.io.HiveFileFormatUtils.getRecordWriter(HiveFileFormatUtils.java:284)
at org.apache.hadoop.hive.ql.io.HiveFileFormatUtils.getHiveRecordWriter(HiveFileFormatUtils.java:269)
... 26 more
]], Vertex did not succeed due to OWN_TASK_FAILURE, failedTasks:1 killedTasks:0, Vertex vertex_1525394716493_0623_23_02 [Reducer 3] killed/failed due to:OWN_TASK_FAILURE]DAG did not succeed due to VERTEX_FAILURE. failedVertices:1 killedVertices:0 (state=08S01,code=2)
I added druid.datasource property to TBLPROPERTIES and ran it once again: CREATE TABLE druid_table (`__time` timestamp,`userid`string,`num_l`float)STORED BY 'org.apache.hadoop.hive.druid.DruidStorageHandler'TBLPROPERTIES ("druid.segment.granularity" = "DAY", "druid.query.granularity" = "DAY", "druid.datasource"="dummy") as select * from poc.test; This was successful. Now I want to increment this table by adding new data. insert into table druid_table select * from poc.test2; This fails with the following error: Error: Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. MetaException(message:INSERT INTO statement is not allowed by druid storage handler) (state=08S01,code=1) The documentation says that INSERT INTO statements are supported. Am I missing something here? Please let me know in case any additional details are required. Many Thanks.
... View more
Labels:
- Labels:
-
Apache Hive
05-08-2018
06:34 AM
@Slim Thanks for your response. The version of Druid that we are using is 0.9.2 which is available as technical preview with HDP 2.6.1. As per your solution I was able to use CTAS successfully. However, I'm not able to append data in the table. insert into table poc_druid.druid_table select * from poc.test; This command fails with the following error. Error: Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. MetaException(message:INSERT INTO statement is not allowed by druid storage handler) (state=08S01,code=1) The documentation says that INSERT statements are supported. Am I missing something here? Many Thanks.
... View more
05-08-2018
05:30 AM
@ln chari
... View more
05-07-2018
03:52 PM
I'm trying to create a table in hive with DruidStorageHandler as follows:
CREATE TABLE druid_table (`__time` timestamp,`userid`string,`num_l`float)STORED BY 'org.apache.hadoop.hive.druid.DruidStorageHandler'TBLPROPERTIES ("druid.segment.granularity"="DAY");
It fails with the following error:
Error: Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. java.lang.RuntimeException: java.io.FileNotFoundException: File /tmp/workingDirectory/.staging-hive_20180507130925_227f2e48-d049-464e-b2cd-43009b3398b3/segmentsDescriptorDir does not exist. (state=08S01,code=1)
I went through the Source code of DruidStorageHandler.Java and to my surprise I wasn't able to see any dfs.mkdir call for the above "segmentsDescriptorDir" directory.
I have got this syntax straight from the documentation. CREATE TABLE druid_table_1
(`__time` TIMESTAMP, `dimension1` STRING, `dimension2` STRING, `metric1` INT, `metric2` FLOAT)
STORED BY 'org.apache.hadoop.hive.druid.DruidStorageHandler'; Has someone faced a similar problem earlier? Thanks in Advance. Megh
... View more
Labels:
- Labels:
-
Apache Hive
05-07-2018
03:22 PM
Hi, Can you please share more details about how you got it to work? I'm facing the same problem while trying to create a new table with Druid storage handler.
... View more
10-26-2017
06:40 AM
@Abdelkrim Hadjidj Hive export and import is working fine with normal tables. In case of partitioned and bucketed ORC tables, we are facing issues as there are further delta directories. Somehow Hive is not able to detect the data present in the table after import.
... View more
10-26-2017
05:41 AM
@Abdelkrim Hadjidj Any other existing solution or workaround for the same?
... View more
10-26-2017
05:39 AM
@Abdelkrim Hadjidj Thanks for this info 🙂
... View more
10-25-2017
09:06 AM
Hello, I would like to know if there is any other solution to Hive metadata replication across clusters other than Apache Falcon Hive Mirror. 1) How does Hive Mirroring by Falcon work internally (On a High level)? 2) Can the same be achieved by backup and restore of metastore DB on a different server? 3) How can HDFS storage dependencies for tables be managed in case of Metastore DB backup and restore? Any help on the above questions is much appreciated. Thanks & Regards, Megh
... View more
Labels:
- Labels:
-
Apache Hive