Member since
05-30-2016
14
Posts
6
Kudos Received
3
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
17470 | 05-25-2018 03:18 PM | |
47356 | 03-29-2018 04:13 PM | |
663 | 04-14-2017 09:30 PM |
12-03-2019
10:44 PM
Thanks for update , the new version of Druid 0.17.0 is not at release. Is this feature available in Druid 0.16.0?
... View more
02-06-2019
05:47 PM
@Nishant Bangarwa Thanks for reply. I have cleaned all the extensions using clean pull-deps command and then added these extensions ["druid-datasketches", "druid-hdfs-storage", "druid-kafka-indexing-service", "mysql-metadata-storage", "druid-avro-extensions", "druid-parquet-extensions"] using pull-deps command. I am using HDP 3.0.1 with Druid 0.12.1 java -classpath "/my/druid/lib/*" org.apache.druid.cli.Main tools pull-deps --clean -c org.apache.druid.extensions:
io.druid.extensions:druid-hdfs-storage:0.12.1
... View more
09-07-2018
09:33 PM
Hi @Nishant Bangarwa, Thank you so much for your answers, in the first link http://druid.io/docs/latest/development/extensions-core/kafka-ingestion.html it is the way we can ingest data to druid data from kafka using supervisor (I done it with kafka). However, I dont think I can do similar with Rabbitmq or other streaming as Spark, Storm ... Therefore, I want to use realtime node to ingest data to druid and test more with Rabbitmq, Spark, Storm ... Do you have any suggestion for me to create realtime node and run it? Many thanks
... View more
05-28-2018
12:23 PM
Thank you @Nishant Bangarwa! "No module psycopg2" solved! Though I am getting another error now: Error: Fatal: no pg_hba.conf entry for host "x.x.x.x", user "x", database "x", SSL off. Plus, I am not sure about the user & password for my Postgres database
... View more
04-01-2018
07:33 PM
Hi Nishant, I have an update, so utiliizing your suggestion of adding the "columns": [ ... ] solved my problem and now I am able to successfully ingest my data into Druid. For users who want to see the ingestion spec, I have included it below: {
"type" : "index_hadoop",
"spec" : {
"dataSchema" : {
"dataSource" : "usgs",
"parser" : {
"type" : "hadoopyString",
"parseSpec" : {
"format" : "tsv",
"timestampSpec" : {
"column" : "dates",
"format" : "auto"
},
"dimensionsSpec" : {
"dimensions": ["staid","val","dates"],
"dimensionExclusions" : [],
"spatialDimensions" : []
},
"columns" : ["staid","val","dates"]
}
},
"metricsSpec" : [
{
"type" : "count",
"name" : "count"
},
{
"type" : "doubleSum",
"name" : "avgFlowCuFtsec",
"fieldName" : "val"
}
],
"granularitySpec" : {
"type" : "uniform",
"segmentGranularity" : "MONTH",
"queryGranularity" : "NONE",
"intervals" : [ "1963-01-01/2013-12-31" ]
}
},
"ioConfig" : {
"type" : "hadoop",
"inputSpec" : {
"type" : "static",
"paths" : "/tmp/druid/napa-flow.tsv.gz"
}
},
"tuningConfig" : {
"type": "hadoop",
"targetPartitionSize" : 10000,
"maxRowsInMemory" : 75000
}
}
}
Note: I will add the "index" ingestion spec soon too, so users can ingest data into Druid from their local file system or they can go with hadoop file system.
... View more
03-27-2018
03:02 PM
Hi Jasper, Changing the visualization type to Big Number should work and display it as a Big Number, After that you will need to click on "Save As" button and add this slice to any dashboard of your choice.
... View more
03-28-2018
07:15 AM
@Dileep Kumar Chiguruvada When i set this property from Hive Client, throwing below error.But in Ambari Hive confs, this property was already set. SET hive.druid.broker.address.default=localhost:8082; /usr/hdp/2.6.3.0-235/hive/bin/hive.distro: line 106: [: /usr/hdp/2.6.3.0-235/hive/lib/hive-metastore-1.2.1000.2.6.3.0-235.jar: binary operator expected
log4j:WARN No such property [maxFileSize] in org.apache.log4j.DailyRollingFileAppender.
Logging initialized using configuration in file:/etc/hive/2.6.3.0-235/0/hive-log4j.properties
hive> SET hive.druid.broker.address.default=localhost:8082; Query returned non-zero code: 1, cause: hive configuration hive.druid.broker.address.default does not exists. hive> Please help--
... View more
03-22-2018
03:18 AM
Thanks a lot! java -Xmx256m -Duser.timezone=UTC -Dfile.encoding=UTF-8 -Dhdp.version=2.6.3.0-235 -Ddruid.storage.storageDirectory=hdfs://`hostname -f`:8020/tmp/data/index/output -Ddruid.storage.type=hdfs -classpath /usr/hdp/current/druid-overlord/extensions/druid-hdfs-storage/*:/usr/hdp/current/druid-overlord/lib/*:/usr/hdp/current/druid-overlord/conf/_common:/etc/hadoop/conf/ io.druid.cli.Main index hadoop ./hadoop_index_spec.json Above worked. Mine is sandbox so using `hostname -f`.
... View more
06-19-2018
01:25 PM
Hi when i launch tranquility it stops on its own : ...
2018-06-19 10:30:11,753 [main] INFO k.c.ZookeeperConsumerConnector - [tranquility-kafka_FR-001SLLI129-1529404211282-a8c8a12a], end rebalancing consumer tranquility-kafka_FR-001SLLI129-1529404211282-a8c8a12a try #0
2018-06-19 10:30:11,755 [main] INFO k.c.ZookeeperConsumerConnector - [tranquility-kafka_FR-001SLLI129-1529404211282-a8c8a12a], Creating topic event watcher for topics (couldwork)
2018-06-19 10:30:11,764 [main] INFO k.c.ZookeeperConsumerConnector - [tranquility-kafka_FR-001SLLI129-1529404211282-a8c8a12a], Topics to consume = List(couldwork)
2018-06-19 10:30:11,768 [tranquility-kafka_FR-001SLLI129-1529404211282-a8c8a12a-leader-finder-thread] INFO kafka.utils.VerifiableProperties - Verifying properties
2018-06-19 10:30:11,769 [tranquility-kafka_FR-001SLLI129-1529404211282-a8c8a12a-leader-finder-thread] INFO kafka.utils.VerifiableProperties - Property client.id is overridden to tranquility-kafka
2018-06-19 10:30:11,769 [tranquility-kafka_FR-001SLLI129-1529404211282-a8c8a12a-leader-finder-thread] INFO kafka.utils.VerifiableProperties - Property metadata.broker.list is overridden to fr-001slli124.groupinfra.com:6667
2018-06-19 10:30:11,769 [tranquility-kafka_FR-001SLLI129-1529404211282-a8c8a12a-leader-finder-thread] INFO kafka.utils.VerifiableProperties - Property request.timeout.ms is overridden to 30000
2018-06-19 10:30:11,787 [tranquility-kafka_FR-001SLLI129-1529404211282-a8c8a12a-leader-finder-thread] INFO kafka.client.ClientUtils$ - Fetching metadata from broker id:1001,host:fr-001slli124.groupinfra.com,port:6667 with correlation id 0 for 1 topic(s) Set(couldwork)
2018-06-19 10:30:11,790 [tranquility-kafka_FR-001SLLI129-1529404211282-a8c8a12a-leader-finder-thread] INFO kafka.producer.SyncProducer - Connected to fr-001slli124.groupinfra.com:6667 for producing
2018-06-19 10:30:11,808 [tranquility-kafka_FR-001SLLI129-1529404211282-a8c8a12a-leader-finder-thread] INFO kafka.producer.SyncProducer - Disconnecting from fr-001slli124.groupinfra.com:6667
2018-06-19 10:30:11,855 [ConsumerFetcherThread-tranquility-kafka_FR-001SLLI129-1529404211282-a8c8a12a-0-1001] INFO kafka.consumer.ConsumerFetcherThread - [ConsumerFetcherThread-tranquility-kafka_FR-001SLLI129-1529404211282-a8c8a12a-0-1001], Starting
2018-06-19 10:30:11,858 [tranquility-kafka_FR-001SLLI129-1529404211282-a8c8a12a-leader-finder-thread] INFO k.consumer.ConsumerFetcherManager - [ConsumerFetcherManager-1529404211455] Added fetcher for partitions ArrayBuffer([[couldwork,0], initOffset 45 to broker id:1001,host:fr-001slli124.groupinfra.com,port:6667] )
2018-06-19 10:30:13,278 [Thread-4] INFO c.metamx.tranquility.kafka.KafkaMain - Initiating shutdown...
2018-06-19 10:30:13,278 [Thread-4] INFO c.m.tranquility.kafka.KafkaConsumer - Shutting down - attempting to flush buffers and commit final offsets
2018-06-19 10:30:13,281 [Thread-4] INFO k.c.ZookeeperConsumerConnector - [tranquility-kafka_FR-001SLLI129-1529404211282-a8c8a12a], ZKConsumerConnector shutting down
2018-06-19 10:30:13,288 [Thread-4] INFO k.c.ZookeeperTopicEventWatcher - Shutting down topic event watcher.
2018-06-19 10:30:13,288 [Thread-4] INFO k.consumer.ConsumerFetcherManager - [ConsumerFetcherManager-1529404211455] Stopping leader finder thread
2018-06-19 10:30:13,288 [Thread-4] INFO k.c.ConsumerFetcherManager$LeaderFinderThread - [tranquility-kafka_FR-001SLLI129-1529404211282-a8c8a12a-leader-finder-thread], Shutting down
2018-06-19 10:30:13,289 [tranquility-kafka_FR-001SLLI129-1529404211282-a8c8a12a-leader-finder-thread] INFO k.c.ConsumerFetcherManager$LeaderFinderThread - [tranquility-kafka_FR-001SLLI129-1529404211282-a8c8a12a-leader-finder-thread], Stopped
2018-06-19 10:30:13,289 [Thread-4] INFO k.c.ConsumerFetcherManager$LeaderFinderThread - [tranquility-kafka_FR-001SLLI129-1529404211282-a8c8a12a-leader-finder-thread], Shutdown completed
2018-06-19 10:30:13,289 [Thread-4] INFO k.consumer.ConsumerFetcherManager - [ConsumerFetcherManager-1529404211455] Stopping all fetchers
2018-06-19 10:30:13,290 [Thread-4] INFO kafka.consumer.ConsumerFetcherThread - [ConsumerFetcherThread-tranquility-kafka_FR-001SLLI129-1529404211282-a8c8a12a-0-1001], Shutting down
2018-06-19 10:30:13,291 [ConsumerFetcherThread-tranquility-kafka_FR-001SLLI129-1529404211282-a8c8a12a-0-1001] INFO kafka.consumer.SimpleConsumer - Reconnect due to socket error: java.nio.channels.ClosedByInterruptException
2018-06-19 10:30:13,291 [ConsumerFetcherThread-tranquility-kafka_FR-001SLLI129-1529404211282-a8c8a12a-0-1001] INFO kafka.consumer.ConsumerFetcherThread - [ConsumerFetcherThread-tranquility-kafka_FR-001SLLI129-1529404211282-a8c8a12a-0-1001], Stopped
2018-06-19 10:30:13,291 [Thread-4] INFO kafka.consumer.ConsumerFetcherThread - [ConsumerFetcherThread-tranquility-kafka_FR-001SLLI129-1529404211282-a8c8a12a-0-1001], Shutdown completed
2018-06-19 10:30:13,292 [Thread-4] INFO k.consumer.ConsumerFetcherManager - [ConsumerFetcherManager-1529404211455] All connections stopped
2018-06-19 10:30:13,294 [ZkClient-EventThread-14-10.80.145.201:2181,10.80.145.200:2181,10.80.145.199:2181] INFO org.I0Itec.zkclient.ZkEventThread - Terminate ZkClient event thread.
2018-06-19 10:30:13,298 [Thread-4] INFO org.apache.zookeeper.ZooKeeper - Session: 0x1641734cd6b0000 closed
2018-06-19 10:30:13,298 [main-EventThread] INFO org.apache.zookeeper.ClientCnxn - EventThread shut down
2018-06-19 10:30:13,298 [Thread-4] INFO k.c.ZookeeperConsumerConnector - [tranquility-kafka_FR-001SLLI129-1529404211282-a8c8a12a], ZKConsumerConnector shutdown completed in 17 ms
2018-06-19 10:30:13,298 [KafkaConsumer-CommitThread] INFO c.m.tranquility.kafka.KafkaConsumer - Commit thread interrupted.
2018-06-19 10:30:13,299 [Thread-4] INFO c.m.tranquility.kafka.KafkaConsumer - Finished clean shutdown.
... View more
12-13-2017
02:29 PM
There is no option "hive.druid.broker.address.default"? Which version is required?
... View more
04-09-2018
04:58 PM
@ Avraha Zilberman Hi I did change the port forwarding in both VM and sandbox container. After that to be sure i execute "docker ps" . It also shows the new configuration. But even though Superset UI is not runiing in browser. What might be the issue? Please help me.
... View more
10-24-2017
01:06 PM
I was able to resolve this issue. Please refer to the link below: https://community.hortonworks.com/questions/141715/router-error-in-druid-with-kerberos.html?childToView=142342#answer-142342
... View more
06-08-2017
07:08 PM
https://issues.apache.org/jira/browse/HIVE-16576 should fix this.
... View more