Thanks for update , the new version of Druid 0.17.0 is not at release. Is this feature available in Druid 0.16.0? ... View more

@Nishant Bangarwa Thanks for reply. I have cleaned all the extensions using clean pull-deps command and then added these extensions ["druid-datasketches", "druid-hdfs-storage", "druid-kafka-indexing-service", "mysql-metadata-storage", "druid-avro-extensions", "druid-parquet-extensions"] using pull-deps command. I am using HDP 3.0.1 with Druid 0.12.1 java -classpath "/my/druid/lib/*" org.apache.druid.cli.Main tools pull-deps --clean -c org.apache.druid.extensions: io.druid.extensions:druid-hdfs-storage:0.12.1 ... View more

Hi @Nishant Bangarwa, Thank you so much for your answers, in the first link http://druid.io/docs/latest/development/extensions-core/kafka-ingestion.html it is the way we can ingest data to druid data from kafka using supervisor (I done it with kafka). However, I dont think I can do similar with Rabbitmq or other streaming as Spark, Storm ... Therefore, I want to use realtime node to ingest data to druid and test more with Rabbitmq, Spark, Storm ... Do you have any suggestion for me to create realtime node and run it? Many thanks ... View more

Thank you @Nishant Bangarwa! "No module psycopg2" solved! Though I am getting another error now: Error: Fatal: no pg_hba.conf entry for host "x.x.x.x", user "x", database "x", SSL off. Plus, I am not sure about the user & password for my Postgres database ... View more

Hi Nishant, I have an update, so utiliizing your suggestion of adding the "columns": [ ... ] solved my problem and now I am able to successfully ingest my data into Druid. For users who want to see the ingestion spec, I have included it below: { "type" : "index_hadoop", "spec" : { "dataSchema" : { "dataSource" : "usgs", "parser" : { "type" : "hadoopyString", "parseSpec" : { "format" : "tsv", "timestampSpec" : { "column" : "dates", "format" : "auto" }, "dimensionsSpec" : { "dimensions": ["staid","val","dates"], "dimensionExclusions" : [], "spatialDimensions" : [] }, "columns" : ["staid","val","dates"] } }, "metricsSpec" : [ { "type" : "count", "name" : "count" }, { "type" : "doubleSum", "name" : "avgFlowCuFtsec", "fieldName" : "val" } ], "granularitySpec" : { "type" : "uniform", "segmentGranularity" : "MONTH", "queryGranularity" : "NONE", "intervals" : [ "1963-01-01/2013-12-31" ] } }, "ioConfig" : { "type" : "hadoop", "inputSpec" : { "type" : "static", "paths" : "/tmp/druid/napa-flow.tsv.gz" } }, "tuningConfig" : { "type": "hadoop", "targetPartitionSize" : 10000, "maxRowsInMemory" : 75000 } } } Note: I will add the "index" ingestion spec soon too, so users can ingest data into Druid from their local file system or they can go with hadoop file system. ... View more

Hi Jasper, Changing the visualization type to Big Number should work and display it as a Big Number, After that you will need to click on "Save As" button and add this slice to any dashboard of your choice. ... View more

@Dileep Kumar Chiguruvada When i set this property from Hive Client, throwing below error.But in Ambari Hive confs, this property was already set. SET hive.druid.broker.address.default=localhost:8082; /usr/hdp/2.6.3.0-235/hive/bin/hive.distro: line 106: [: /usr/hdp/2.6.3.0-235/hive/lib/hive-metastore-1.2.1000.2.6.3.0-235.jar: binary operator expected log4j:WARN No such property [maxFileSize] in org.apache.log4j.DailyRollingFileAppender. Logging initialized using configuration in file:/etc/hive/2.6.3.0-235/0/hive-log4j.properties hive> SET hive.druid.broker.address.default=localhost:8082; Query returned non-zero code: 1, cause: hive configuration hive.druid.broker.address.default does not exists. hive> Please help-- ... View more

Thanks a lot! java -Xmx256m -Duser.timezone=UTC -Dfile.encoding=UTF-8 -Dhdp.version=2.6.3.0-235 -Ddruid.storage.storageDirectory=hdfs://`hostname -f`:8020/tmp/data/index/output -Ddruid.storage.type=hdfs -classpath /usr/hdp/current/druid-overlord/extensions/druid-hdfs-storage/*:/usr/hdp/current/druid-overlord/lib/*:/usr/hdp/current/druid-overlord/conf/_common:/etc/hadoop/conf/ io.druid.cli.Main index hadoop ./hadoop_index_spec.json Above worked. Mine is sandbox so using `hostname -f`. ... View more

Hi when i launch tranquility it stops on its own : ... 2018-06-19 10:30:11,753 [main] INFO k.c.ZookeeperConsumerConnector - [tranquility-kafka_FR-001SLLI129-1529404211282-a8c8a12a], end rebalancing consumer tranquility-kafka_FR-001SLLI129-1529404211282-a8c8a12a try #0 2018-06-19 10:30:11,755 [main] INFO k.c.ZookeeperConsumerConnector - [tranquility-kafka_FR-001SLLI129-1529404211282-a8c8a12a], Creating topic event watcher for topics (couldwork) 2018-06-19 10:30:11,764 [main] INFO k.c.ZookeeperConsumerConnector - [tranquility-kafka_FR-001SLLI129-1529404211282-a8c8a12a], Topics to consume = List(couldwork) 2018-06-19 10:30:11,768 [tranquility-kafka_FR-001SLLI129-1529404211282-a8c8a12a-leader-finder-thread] INFO kafka.utils.VerifiableProperties - Verifying properties 2018-06-19 10:30:11,769 [tranquility-kafka_FR-001SLLI129-1529404211282-a8c8a12a-leader-finder-thread] INFO kafka.utils.VerifiableProperties - Property client.id is overridden to tranquility-kafka 2018-06-19 10:30:11,769 [tranquility-kafka_FR-001SLLI129-1529404211282-a8c8a12a-leader-finder-thread] INFO kafka.utils.VerifiableProperties - Property metadata.broker.list is overridden to fr-001slli124.groupinfra.com:6667 2018-06-19 10:30:11,769 [tranquility-kafka_FR-001SLLI129-1529404211282-a8c8a12a-leader-finder-thread] INFO kafka.utils.VerifiableProperties - Property request.timeout.ms is overridden to 30000 2018-06-19 10:30:11,787 [tranquility-kafka_FR-001SLLI129-1529404211282-a8c8a12a-leader-finder-thread] INFO kafka.client.ClientUtils$ - Fetching metadata from broker id:1001,host:fr-001slli124.groupinfra.com,port:6667 with correlation id 0 for 1 topic(s) Set(couldwork) 2018-06-19 10:30:11,790 [tranquility-kafka_FR-001SLLI129-1529404211282-a8c8a12a-leader-finder-thread] INFO kafka.producer.SyncProducer - Connected to fr-001slli124.groupinfra.com:6667 for producing 2018-06-19 10:30:11,808 [tranquility-kafka_FR-001SLLI129-1529404211282-a8c8a12a-leader-finder-thread] INFO kafka.producer.SyncProducer - Disconnecting from fr-001slli124.groupinfra.com:6667 2018-06-19 10:30:11,855 [ConsumerFetcherThread-tranquility-kafka_FR-001SLLI129-1529404211282-a8c8a12a-0-1001] INFO kafka.consumer.ConsumerFetcherThread - [ConsumerFetcherThread-tranquility-kafka_FR-001SLLI129-1529404211282-a8c8a12a-0-1001], Starting 2018-06-19 10:30:11,858 [tranquility-kafka_FR-001SLLI129-1529404211282-a8c8a12a-leader-finder-thread] INFO k.consumer.ConsumerFetcherManager - [ConsumerFetcherManager-1529404211455] Added fetcher for partitions ArrayBuffer([[couldwork,0], initOffset 45 to broker id:1001,host:fr-001slli124.groupinfra.com,port:6667] ) 2018-06-19 10:30:13,278 [Thread-4] INFO c.metamx.tranquility.kafka.KafkaMain - Initiating shutdown... 2018-06-19 10:30:13,278 [Thread-4] INFO c.m.tranquility.kafka.KafkaConsumer - Shutting down - attempting to flush buffers and commit final offsets 2018-06-19 10:30:13,281 [Thread-4] INFO k.c.ZookeeperConsumerConnector - [tranquility-kafka_FR-001SLLI129-1529404211282-a8c8a12a], ZKConsumerConnector shutting down 2018-06-19 10:30:13,288 [Thread-4] INFO k.c.ZookeeperTopicEventWatcher - Shutting down topic event watcher. 2018-06-19 10:30:13,288 [Thread-4] INFO k.consumer.ConsumerFetcherManager - [ConsumerFetcherManager-1529404211455] Stopping leader finder thread 2018-06-19 10:30:13,288 [Thread-4] INFO k.c.ConsumerFetcherManager$LeaderFinderThread - [tranquility-kafka_FR-001SLLI129-1529404211282-a8c8a12a-leader-finder-thread], Shutting down 2018-06-19 10:30:13,289 [tranquility-kafka_FR-001SLLI129-1529404211282-a8c8a12a-leader-finder-thread] INFO k.c.ConsumerFetcherManager$LeaderFinderThread - [tranquility-kafka_FR-001SLLI129-1529404211282-a8c8a12a-leader-finder-thread], Stopped 2018-06-19 10:30:13,289 [Thread-4] INFO k.c.ConsumerFetcherManager$LeaderFinderThread - [tranquility-kafka_FR-001SLLI129-1529404211282-a8c8a12a-leader-finder-thread], Shutdown completed 2018-06-19 10:30:13,289 [Thread-4] INFO k.consumer.ConsumerFetcherManager - [ConsumerFetcherManager-1529404211455] Stopping all fetchers 2018-06-19 10:30:13,290 [Thread-4] INFO kafka.consumer.ConsumerFetcherThread - [ConsumerFetcherThread-tranquility-kafka_FR-001SLLI129-1529404211282-a8c8a12a-0-1001], Shutting down 2018-06-19 10:30:13,291 [ConsumerFetcherThread-tranquility-kafka_FR-001SLLI129-1529404211282-a8c8a12a-0-1001] INFO kafka.consumer.SimpleConsumer - Reconnect due to socket error: java.nio.channels.ClosedByInterruptException 2018-06-19 10:30:13,291 [ConsumerFetcherThread-tranquility-kafka_FR-001SLLI129-1529404211282-a8c8a12a-0-1001] INFO kafka.consumer.ConsumerFetcherThread - [ConsumerFetcherThread-tranquility-kafka_FR-001SLLI129-1529404211282-a8c8a12a-0-1001], Stopped 2018-06-19 10:30:13,291 [Thread-4] INFO kafka.consumer.ConsumerFetcherThread - [ConsumerFetcherThread-tranquility-kafka_FR-001SLLI129-1529404211282-a8c8a12a-0-1001], Shutdown completed 2018-06-19 10:30:13,292 [Thread-4] INFO k.consumer.ConsumerFetcherManager - [ConsumerFetcherManager-1529404211455] All connections stopped 2018-06-19 10:30:13,294 [ZkClient-EventThread-14-10.80.145.201:2181,10.80.145.200:2181,10.80.145.199:2181] INFO org.I0Itec.zkclient.ZkEventThread - Terminate ZkClient event thread. 2018-06-19 10:30:13,298 [Thread-4] INFO org.apache.zookeeper.ZooKeeper - Session: 0x1641734cd6b0000 closed 2018-06-19 10:30:13,298 [main-EventThread] INFO org.apache.zookeeper.ClientCnxn - EventThread shut down 2018-06-19 10:30:13,298 [Thread-4] INFO k.c.ZookeeperConsumerConnector - [tranquility-kafka_FR-001SLLI129-1529404211282-a8c8a12a], ZKConsumerConnector shutdown completed in 17 ms 2018-06-19 10:30:13,298 [KafkaConsumer-CommitThread] INFO c.m.tranquility.kafka.KafkaConsumer - Commit thread interrupted. 2018-06-19 10:30:13,299 [Thread-4] INFO c.m.tranquility.kafka.KafkaConsumer - Finished clean shutdown. ... View more

There is no option "hive.druid.broker.address.default"? Which version is required? ... View more

@ Avraha Zilberman Hi I did change the port forwarding in both VM and sandbox container. After that to be sure i execute "docker ps" . It also shows the new configuration. But even though Superset UI is not runiing in browser. What might be the issue? Please help me. ... View more

I was able to resolve this issue. Please refer to the link below: https://community.hortonworks.com/questions/141715/router-error-in-druid-with-kerberos.html?childToView=142342#answer-142342 ... View more

https://issues.apache.org/jira/browse/HIVE-16576 should fix this. ... View more

That worked! ... View more