Thank you @Nishant Bangarwa! "No module psycopg2" solved! Though I am getting another error now: Error: Fatal: no pg_hba.conf entry for host "x.x.x.x", user "x", database "x", SSL off. Plus, I am not sure about the user & password for my Postgres database ... View more

Hi Nishant, I have an update, so utiliizing your suggestion of adding the "columns": [ ... ] solved my problem and now I am able to successfully ingest my data into Druid. For users who want to see the ingestion spec, I have included it below: { "type" : "index_hadoop", "spec" : { "dataSchema" : { "dataSource" : "usgs", "parser" : { "type" : "hadoopyString", "parseSpec" : { "format" : "tsv", "timestampSpec" : { "column" : "dates", "format" : "auto" }, "dimensionsSpec" : { "dimensions": ["staid","val","dates"], "dimensionExclusions" : [], "spatialDimensions" : [] }, "columns" : ["staid","val","dates"] } }, "metricsSpec" : [ { "type" : "count", "name" : "count" }, { "type" : "doubleSum", "name" : "avgFlowCuFtsec", "fieldName" : "val" } ], "granularitySpec" : { "type" : "uniform", "segmentGranularity" : "MONTH", "queryGranularity" : "NONE", "intervals" : [ "1963-01-01/2013-12-31" ] } }, "ioConfig" : { "type" : "hadoop", "inputSpec" : { "type" : "static", "paths" : "/tmp/druid/napa-flow.tsv.gz" } }, "tuningConfig" : { "type": "hadoop", "targetPartitionSize" : 10000, "maxRowsInMemory" : 75000 } } } Note: I will add the "index" ingestion spec soon too, so users can ingest data into Druid from their local file system or they can go with hadoop file system. ... View more

Thanks a lot! java -Xmx256m -Duser.timezone=UTC -Dfile.encoding=UTF-8 -Dhdp.version=2.6.3.0-235 -Ddruid.storage.storageDirectory=hdfs://`hostname -f`:8020/tmp/data/index/output -Ddruid.storage.type=hdfs -classpath /usr/hdp/current/druid-overlord/extensions/druid-hdfs-storage/*:/usr/hdp/current/druid-overlord/lib/*:/usr/hdp/current/druid-overlord/conf/_common:/etc/hadoop/conf/ io.druid.cli.Main index hadoop ./hadoop_index_spec.json Above worked. Mine is sandbox so using `hostname -f`. ... View more

Hi when i launch tranquility it stops on its own : ... 2018-06-19 10:30:11,753 [main] INFO k.c.ZookeeperConsumerConnector - [tranquility-kafka_FR-001SLLI129-1529404211282-a8c8a12a], end rebalancing consumer tranquility-kafka_FR-001SLLI129-1529404211282-a8c8a12a try #0 2018-06-19 10:30:11,755 [main] INFO k.c.ZookeeperConsumerConnector - [tranquility-kafka_FR-001SLLI129-1529404211282-a8c8a12a], Creating topic event watcher for topics (couldwork) 2018-06-19 10:30:11,764 [main] INFO k.c.ZookeeperConsumerConnector - [tranquility-kafka_FR-001SLLI129-1529404211282-a8c8a12a], Topics to consume = List(couldwork) 2018-06-19 10:30:11,768 [tranquility-kafka_FR-001SLLI129-1529404211282-a8c8a12a-leader-finder-thread] INFO kafka.utils.VerifiableProperties - Verifying properties 2018-06-19 10:30:11,769 [tranquility-kafka_FR-001SLLI129-1529404211282-a8c8a12a-leader-finder-thread] INFO kafka.utils.VerifiableProperties - Property client.id is overridden to tranquility-kafka 2018-06-19 10:30:11,769 [tranquility-kafka_FR-001SLLI129-1529404211282-a8c8a12a-leader-finder-thread] INFO kafka.utils.VerifiableProperties - Property metadata.broker.list is overridden to fr-001slli124.groupinfra.com:6667 2018-06-19 10:30:11,769 [tranquility-kafka_FR-001SLLI129-1529404211282-a8c8a12a-leader-finder-thread] INFO kafka.utils.VerifiableProperties - Property request.timeout.ms is overridden to 30000 2018-06-19 10:30:11,787 [tranquility-kafka_FR-001SLLI129-1529404211282-a8c8a12a-leader-finder-thread] INFO kafka.client.ClientUtils$ - Fetching metadata from broker id:1001,host:fr-001slli124.groupinfra.com,port:6667 with correlation id 0 for 1 topic(s) Set(couldwork) 2018-06-19 10:30:11,790 [tranquility-kafka_FR-001SLLI129-1529404211282-a8c8a12a-leader-finder-thread] INFO kafka.producer.SyncProducer - Connected to fr-001slli124.groupinfra.com:6667 for producing 2018-06-19 10:30:11,808 [tranquility-kafka_FR-001SLLI129-1529404211282-a8c8a12a-leader-finder-thread] INFO kafka.producer.SyncProducer - Disconnecting from fr-001slli124.groupinfra.com:6667 2018-06-19 10:30:11,855 [ConsumerFetcherThread-tranquility-kafka_FR-001SLLI129-1529404211282-a8c8a12a-0-1001] INFO kafka.consumer.ConsumerFetcherThread - [ConsumerFetcherThread-tranquility-kafka_FR-001SLLI129-1529404211282-a8c8a12a-0-1001], Starting 2018-06-19 10:30:11,858 [tranquility-kafka_FR-001SLLI129-1529404211282-a8c8a12a-leader-finder-thread] INFO k.consumer.ConsumerFetcherManager - [ConsumerFetcherManager-1529404211455] Added fetcher for partitions ArrayBuffer([[couldwork,0], initOffset 45 to broker id:1001,host:fr-001slli124.groupinfra.com,port:6667] ) 2018-06-19 10:30:13,278 [Thread-4] INFO c.metamx.tranquility.kafka.KafkaMain - Initiating shutdown... 2018-06-19 10:30:13,278 [Thread-4] INFO c.m.tranquility.kafka.KafkaConsumer - Shutting down - attempting to flush buffers and commit final offsets 2018-06-19 10:30:13,281 [Thread-4] INFO k.c.ZookeeperConsumerConnector - [tranquility-kafka_FR-001SLLI129-1529404211282-a8c8a12a], ZKConsumerConnector shutting down 2018-06-19 10:30:13,288 [Thread-4] INFO k.c.ZookeeperTopicEventWatcher - Shutting down topic event watcher. 2018-06-19 10:30:13,288 [Thread-4] INFO k.consumer.ConsumerFetcherManager - [ConsumerFetcherManager-1529404211455] Stopping leader finder thread 2018-06-19 10:30:13,288 [Thread-4] INFO k.c.ConsumerFetcherManager$LeaderFinderThread - [tranquility-kafka_FR-001SLLI129-1529404211282-a8c8a12a-leader-finder-thread], Shutting down 2018-06-19 10:30:13,289 [tranquility-kafka_FR-001SLLI129-1529404211282-a8c8a12a-leader-finder-thread] INFO k.c.ConsumerFetcherManager$LeaderFinderThread - [tranquility-kafka_FR-001SLLI129-1529404211282-a8c8a12a-leader-finder-thread], Stopped 2018-06-19 10:30:13,289 [Thread-4] INFO k.c.ConsumerFetcherManager$LeaderFinderThread - [tranquility-kafka_FR-001SLLI129-1529404211282-a8c8a12a-leader-finder-thread], Shutdown completed 2018-06-19 10:30:13,289 [Thread-4] INFO k.consumer.ConsumerFetcherManager - [ConsumerFetcherManager-1529404211455] Stopping all fetchers 2018-06-19 10:30:13,290 [Thread-4] INFO kafka.consumer.ConsumerFetcherThread - [ConsumerFetcherThread-tranquility-kafka_FR-001SLLI129-1529404211282-a8c8a12a-0-1001], Shutting down 2018-06-19 10:30:13,291 [ConsumerFetcherThread-tranquility-kafka_FR-001SLLI129-1529404211282-a8c8a12a-0-1001] INFO kafka.consumer.SimpleConsumer - Reconnect due to socket error: java.nio.channels.ClosedByInterruptException 2018-06-19 10:30:13,291 [ConsumerFetcherThread-tranquility-kafka_FR-001SLLI129-1529404211282-a8c8a12a-0-1001] INFO kafka.consumer.ConsumerFetcherThread - [ConsumerFetcherThread-tranquility-kafka_FR-001SLLI129-1529404211282-a8c8a12a-0-1001], Stopped 2018-06-19 10:30:13,291 [Thread-4] INFO kafka.consumer.ConsumerFetcherThread - [ConsumerFetcherThread-tranquility-kafka_FR-001SLLI129-1529404211282-a8c8a12a-0-1001], Shutdown completed 2018-06-19 10:30:13,292 [Thread-4] INFO k.consumer.ConsumerFetcherManager - [ConsumerFetcherManager-1529404211455] All connections stopped 2018-06-19 10:30:13,294 [ZkClient-EventThread-14-10.80.145.201:2181,10.80.145.200:2181,10.80.145.199:2181] INFO org.I0Itec.zkclient.ZkEventThread - Terminate ZkClient event thread. 2018-06-19 10:30:13,298 [Thread-4] INFO org.apache.zookeeper.ZooKeeper - Session: 0x1641734cd6b0000 closed 2018-06-19 10:30:13,298 [main-EventThread] INFO org.apache.zookeeper.ClientCnxn - EventThread shut down 2018-06-19 10:30:13,298 [Thread-4] INFO k.c.ZookeeperConsumerConnector - [tranquility-kafka_FR-001SLLI129-1529404211282-a8c8a12a], ZKConsumerConnector shutdown completed in 17 ms 2018-06-19 10:30:13,298 [KafkaConsumer-CommitThread] INFO c.m.tranquility.kafka.KafkaConsumer - Commit thread interrupted. 2018-06-19 10:30:13,299 [Thread-4] INFO c.m.tranquility.kafka.KafkaConsumer - Finished clean shutdown. ... View more

@ Avraha Zilberman Hi I did change the port forwarding in both VM and sandbox container. After that to be sure i execute "docker ps" . It also shows the new configuration. But even though Superset UI is not runiing in browser. What might be the issue? Please help me. ... View more

That worked! ... View more