Re: Druid kafka ingestion from Hive - HDP 3.0

sbouguerra · ‎02-09-2019

I bet your column names have upper cases in Druid tables, thus Hive will query the wrong columns since it is always down-casing the names

sbouguerra · ‎11-12-2018

Can you please explain what you mean with rebuild ? What is the Source of truth ? is it an append? is it removing some rows ? Can you please add an example ?

sbouguerra · ‎11-09-2018

You see that exception because MySql is the default metadata Driver for Druid-Hive integration. you need to set hive.druid.metadata.db.type=derby Also i want to make 2 points. First Derby is only used for integration testing and will only work on one host and did not get any chance to test is outside of that scope. Second Please keep in mind that Hive is not case sensitive and lowers all your columns name while Druid is case sensitive thus recommend to lower case all the column names.

sbouguerra · ‎11-07-2018

Correct @Matthieu Lamairesse Druid is case sensitive while Hive is not, thus, to make it work you need to make sure that all the columns are in lowercase format.

sbouguerra · ‎11-07-2018

you can set the kerberos credentials as part of the consumer properties, look at this thread and let me know if it works for you. https://groups.google.com/forum/#!topic/druid-user/W2SiPnNsy0U

sbouguerra · ‎11-07-2018

Druid Hive Handler is not using MR, but it is using whatever Hive is using as execution engine eg TEZ.

sbouguerra · ‎11-06-2018

Looked at the code and seems like the current state of the art the timestamp column is hard coded to be __time, thus that is why you are getting the exceptions since your column is called `timestamp`. https://github.com/apache/hive/blob/a51e6aeaf816bdeea5e91ba3a0fab8a31b3a496d/druid-handler/src/java/org/apache/hadoop/hive/druid/DruidStorageHandler.java#L301 If this is the case this is a serious limitation and need to be fixed. @Nishant Bangarwa what you think?

sbouguerra · ‎11-06-2018

This is a known bug, when you try to create an empty table. Quick fix will be to Create the table and insert data in the same time using Create table as Select statement. Or update to version with the actual bug fix https://issues.apache.org/jira/browse/HIVE-16677

sbouguerra · ‎10-17-2018

@Nishant Bangarwa can you please take a look at this?

sbouguerra · ‎09-26-2018

Can you please add the Logs of Coordinator and Overlord plus the Task?

sbouguerra · ‎08-31-2018

If you are interested on rolling up SUM and Count then you output raw data as is and "druid.query.granularity"="HOUR" (FYI not segment granularity) will do the rollup for you. If you want to compute other rollups metrics like using MIN/MAX/AVG etc then you need to do the rollup before. If you share an example of your use cases i can help explaining more. Thanks.

sbouguerra · ‎08-31-2018

seems like having issue connecting to jdbc:mysql://${DRUID_HOST}/druid; are you sure the values of DRUID_HOST is the actual hostname port of Mysql DB? Can you share the logs of HS2?

sbouguerra · ‎08-31-2018

Please take a look at this page HDP-2.6-Hive-Druid How much pre-processing is needed on the hive table creation? should i "clean" the data such that there is not further aggregation on druid or is the granularity settings will take care of aggregation on the druid side? and if so, where should those aggregations be defined? (so if i want "HOUR" granularity should i pre-process the table to group by the hours already and do all the aggregations within Hive)? Use "druid.query.granularity" = "HOUR" Is there any support for "HyperUnique" in this workflow? looking on doing something like "unique user ids" sorry NO complex metrics One of my challenges is that new metrics are added at a weekly/monthly basis. How will i support that if i need to load the data daily into druid? How would you handle schema evolution? Use Insert Into for newer data and Alter table to add new columns.

sbouguerra · ‎05-07-2018

This is a fixed bug looks like you older version fix is here https://issues.apache.org/jira/browse/HIVE-16677 The workaround is to use a CTAS insert at least one row.

sbouguerra · ‎05-02-2018

The speed at which the indexes are generated depends on Hive LLAP workers not Druid it self. You might try to change the "druid.segment.granularity" = "MONTH" to "DAY" that will give you more parallelism thus might run faster if LLAP has enough resources.

sbouguerra · ‎05-02-2018

use Inset into statement create table test_table(`timecolumn` timestamp, `userid` string, `num_l` float); insert into test_table values ('2015-01-08 00:00:00', 'i1-start', 4); CREATE TABLE druid_table (`__time` timestamp, `userid` string, `num_l` float) STORED BY 'org.apache.hadoop.hive.druid.DruidStorageHandler' TBLPROPERTIES ("druid.segment.granularity" = "DAY"); INSERT INTO TABLE druid_table select cast(`timecolumn` as timestamp) as `__time`, `userid`, `num_l` FROM test_table;

sbouguerra · ‎05-02-2018

It is coming from Hive conf as default. You can set it from Ambari to make it global for all sessions hive.druid.storage.storageDirectory https://github.com/b-slim/hive/blob/f8bc4868eced2ca83113579b626e279bbe6d5b13/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java#L2626

sbouguerra · ‎04-28-2018

@ln Chari the properties you need are set hive.druid.metadata.db.type=postgresql; set hive.druid.metadata.username=${DRUID_USERNAME}; set hive.druid.metadata.password=${DRUID_PASSWORD}; set hive.druid.metadata.uri=jdbc:postgresql://${METADATA_DRUID_HOST}:5432/druid; You were missing hive.druid.metadata.db.type Also make sure that ${DRUID_USERNAME} are replaced with actual values You do not need all the properties starting with druid.*

sbouguerra · ‎04-03-2018

In this case increasing the Ulimit of number of processes to the Druid user will fix this issue. This Article explains more the issue and best way to fix such issue. Exception stack. 2018-04-02T23:41:56,827 ERROR [main] io.druid.cli.CliBroker-Errorwhen starting up.Failing.java.lang.OutOfMemoryError: unable to create newnative thread at java.lang.Thread.start0(NativeMethod)~[?:1.8.0_40]at java.lang.Thread.start(Thread.java:714)~[?:1.8.0_40]at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:950)~[?:1.8.0_40]at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1368)~[?:1.8.0_40]at org.jboss.netty.util.internal.DeadLockProofWorker.start(DeadLockProofWorker.java:38)~[netty-3.10.6.Final.jar:?]at org.jboss.netty.channel.socket.nio.AbstractNioSelector.openSelector(AbstractNioSelector.java:368)~[netty-3.10.6.Final.jar:?]at org.jboss.netty.channel.socket.nio.AbstractNioSelector.<init>(AbstractNioSelector.java:100)~[netty-3.10.6.Final.jar:?]at org.jboss.netty.channel.socket.nio.AbstractNioWorker.<init>(AbstractNioWorker.java:52)~[netty-3.10.6.Final.jar:?]at org.jboss.netty.channel.socket.nio.NioWorker.<init>(NioWorker.java:45)~[netty-3.10.6.Final.jar:?]at org.jboss.netty.channel.socket.nio.NioWorkerPool.newWorker(NioWorkerPool.java:44)~[netty-3.10.6.Final.jar:?]at org.jboss.netty.channel.socket.nio.NioWorkerPool.newWorker(NioWorkerPool.java:28)

sbouguerra · ‎03-21-2018

I think your classpath is missing the HDFS module that is under extensions directory...

sbouguerra · ‎03-06-2018

i believe HS2 logs are under /var/log/hive/hiveserver2.log

sbouguerra · ‎03-01-2018

do you see this only with druid indexing jobs ?

sbouguerra · ‎02-27-2018

here is a step by step tutorial http://druid.io/docs/0.11.0/tutorials/tutorial-kafka.html

sbouguerra · ‎02-27-2018

If this is the first run then it takes some time for Superset to discover the data (it has to wait for the first handoff data going from real-time nodes to historicals) then it should work. I guess the default handoff window is 15 mins

sbouguerra · ‎12-22-2017

Hi, Thanks for trying out this new Hive Feature. The Scan query was very recently released as part of the core Druid 0.11.0 core engine. Yes we are working on the integration of scan query, stage one is done here https://github.com/apache/calcite/pull/577 and stage 2 will follow shortly https://issues.apache.org/jira/browse/HIVE-17627. We are also working on pushing more computations down to Druid, as you can see here https://github.com/apache/calcite/pulls?utf8=%E2%9C%93&q=druid and https://issues.apache.org/jira/browse/HIVE/component/12330863/?selectedTab=com.atlassian.jira.jira-projects-plugin:component-summary-panel. For the question In Tableau I see there is way to define custom queries ,but this more of an extract rather than a live connection.Is there a better for live connections in Hive using custom queries? Am not really sure if there is a better way, i hope that Tableau experts can answer this. @Carter Shanklin any chance you can dive in?

sbouguerra · ‎12-05-2017

can you create a new issue and attach the logs. it is hard to see what is going on without logs. One way to check this is to first issue an explain query that will show you the druid query then you can copy that query and try it your self via curl command.

sbouguerra · ‎11-30-2017

FYI Derby is a local instance DB used only for testing. For production please use Mysql or Postgres.

sbouguerra · ‎11-27-2017

Most of those CLIs you are referring to are deprecated. Please refer to this post to read about the differences https://community.hortonworks.com/questions/135182/hive-cli-vs-beeline.html

sbouguerra · ‎11-27-2017

It all depends on how you are building your schema but it is doable. IMO depending on the cardinality of your customer base and/or the number of products. Also you might be able to use Sketches if approximation is okay for your use cases.

sbouguerra · ‎11-24-2017

This is related to this question. https://community.hortonworks.com/questions/102905/hive-druid-handler-javalangnoclassdeffounderror-or.html

Online	Offline
Last Visited	‎09-07-2019 05:22 AM

Date Registered	‎08-05-2016 05:14 PM
Last Visited	‎09-07-2019 05:22 AM
Total Messages Posted	76
Total Kudos Received	10

Re: Druid kafka ingestion from Hive - HDP 3.0

Re: Hive to Druid Methodology

Re: Druid on Hive LLAP - HDP2.6.1

Re: Druid integration with Hive LLAP

Re: Druid 0.10.1 on Ambari

Re: Hive/Druid all Queries yield null columns

Re: Is it possible to rebuild druid segment from h...

Re: hive druid create table failing due to mysql

Re: Druid kafka ingestion from Hive - HDP 3.0

Re: How can I set Druid to ingest streaming data f...

Re: Druid-Hive integration

Re: Druid kafka ingestion from Hive - HDP 3.0

Re: Hive LLAP and druid - not able to create table

Re: not able to select or drop table which was cre...

Re: how to load csv file in Druid

Re: Hive to Druid Methodology

Re: SandBox HDP 2.6.5.0 - Hive and Druid Integrati...

Re: Hive to Druid Methodology

Re: Create table in Hive with DruidStorageHadler f...

Re: Memory configuraiton for better performance on...

Re: How to load delta data on daily basis into Hiv...

Re: Druid on Hive LLAP - HDP2.6.1

Re: Druid integration with Hive LLAP

Druid node failing with OOM "java.lang.OutOfMemory...

Re: What would be the right command to start Druid...

Re: How to debug Hive2 beeline ?

Re: MapReduce failure when using Druid indexing

Re: In DRUID, unable to access Druid after install...

Re: SAM application not writing data to Druid Data...

Re: Druid-Hive-Tableau integration query types

Re: Stuck as Hive- DruidIntegration

Re: Stuck as Hive- DruidIntegration

Re: Hive druid Integration Issue -org.apache.hadoo...

Re: Data Store for Customer Segmentation, Druid ?

Re: Hive druid Integration Issue -org.apache.hadoo...