About jmedel

jmedel · ‎07-19-2018

@Divya Sodha, Can you add a photo of your ExecuteProcess processor properties tab that shows the path used to execute the python script? On your Raspberry Pi, does the location you have the python script match with the configuration specified in the MiNiFi processor? Perfect on the conversion part. For your ping issue, which Hortonworks Sandbox are you using Docker, VirtualBox or VMware? What is the Network Adapter used for the environment (Docker, VirtualBox, VMware) your Sandbox runs on? Also can your Sandbox ping google.com or another site?

jmedel · ‎07-17-2018

@Divya Sodha, Does the python script have the same name as "WeatherStation.py" or is it changed? Is the MiNiFi flow that was built in NiFi running on the NiFi UI or was the NiFi template (MiNiFi flow) converted to MiNiFi yaml file and transported over to the Raspberry Pi? Are you running the MiNiFi flow using ./bin/minifi.sh start? In your minifi-app log, I noticed your MiNiFi instance is having trouble communicating with the NiFi node on HDF, is that still occurring?

jmedel · ‎07-06-2018

@Divya Sodha, Can you share the python script you are using? Also did you write out the script or download it? Can you also send me your minifi logs: cd minifi-0.4.0/logs? Thanks, James

jmedel · ‎06-29-2018

Hey @rajat puchnanda First I would recommend extracting the attributes from JSON using EvaluateJSONPath processor, then utilize AttributesToCSV processor to represent the FlowFile attributes as CSV format. For seeing how to use evaluateJSONPath processor, check out this part of the tutorial series: https://hortonworks.com/tutorial/analyze-transit-patterns-with-apache-nifi/section/5/ AttributesToCSV processor is similar to AttributesToJSON processor, the following tutorial shows how to use AttributesToJSON, apply the similar idea to using AttributesToCSV processor: https://hortonworks.com/tutorial/analyze-transit-patterns-with-apache-nifi/section/6/ For info on AttributesToCSV processor: https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.7.0/org.apache.nifi.processors.standard.AttributesToCSV/index.html

jmedel · ‎04-01-2018

I realized that the problem was the input path didn't exist and that prevented me from submitting the wikiticker data into Druid. So I created the appropriate directories on HDFS, then stored the dataset into the appropriate data source path that was being referenced in the Druid Hadoop-based Ingestion Spec: { "type" : "index_hadoop", "spec" : { "dataSchema" : { "dataSource" : "wikiticker", "parser" : { "type" : "hadoopyString", "parseSpec" : { "format" : "json", "timestampSpec" : { "format" : "auto", "column" : "time" }, "dimensionsSpec" : { "dimensions" : [ "channel", "cityName", "comment", "countryIsoCode", "countryName", "isAnonymous", "isMinor", "isNew", "isRobot", "isUnpatrolled", "metroCode", "namespace", "page", "regionIsoCode", "regionName", "user" ] } } }, "metricsSpec" : [ { "name" : "count", "type" : "count" }, { "name" : "added", "type" : "longSum", "fieldName" : "added" }, { "name" : "deleted", "type" : "longSum", "fieldName" : "deleted" }, { "name" : "delta", "type" : "longSum", "fieldName" : "delta" }, { "name" : "user_unique", "type" : "hyperUnique", "fieldName" : "user" } ], "granularitySpec" : { "type" : "uniform", "segmentGranularity" : "day", "queryGranularity" : "none", "intervals" : ["2015-09-12/2015-09-13"] } }, "ioConfig" : { "type" : "hadoop", "inputSpec" : { "type" : "static", "paths" : "quickstart/wikiticker-2015-09-12-sampled.json.gz" } }, "tuningConfig" : { "type" : "hadoop", "partitionsSpec" : { "type" : "hashed", "targetPartitionSize" : 5000000 }, "jobProperties" : {} } } } On the HDP Sandbox, go into web shell client at http://sandbox-hdf.hortonworks.com:4200/ hdfs dfs -mkdir -p /user/druid/quickstart hdfs dfs -put /usr/hdp/2.6.4.0-91/druid/quickstart/wikiticker-2015-09-12-sampled.json.gz /user/druid/quickstart/ By adding this directory in HDFS, then submitting my Hadoop-based index spec to Druid Overlord that allowed for the task to be successfully submitted and data was ingested into Druid Run curl command to send spec over HTTP POST request to Druid Overlord: curl -X 'POST' -H 'Content-Type:application/json' -d @/usr/hdp/2.6.4.0-91/druid/quickstart/wikiticker-index.json localhost:8090/druid/indexer/v1/task

jmedel · ‎04-01-2018

Hi Nishant, I have an update, so utiliizing your suggestion of adding the "columns": [ ... ] solved my problem and now I am able to successfully ingest my data into Druid. For users who want to see the ingestion spec, I have included it below: { "type" : "index_hadoop", "spec" : { "dataSchema" : { "dataSource" : "usgs", "parser" : { "type" : "hadoopyString", "parseSpec" : { "format" : "tsv", "timestampSpec" : { "column" : "dates", "format" : "auto" }, "dimensionsSpec" : { "dimensions": ["staid","val","dates"], "dimensionExclusions" : [], "spatialDimensions" : [] }, "columns" : ["staid","val","dates"] } }, "metricsSpec" : [ { "type" : "count", "name" : "count" }, { "type" : "doubleSum", "name" : "avgFlowCuFtsec", "fieldName" : "val" } ], "granularitySpec" : { "type" : "uniform", "segmentGranularity" : "MONTH", "queryGranularity" : "NONE", "intervals" : [ "1963-01-01/2013-12-31" ] } }, "ioConfig" : { "type" : "hadoop", "inputSpec" : { "type" : "static", "paths" : "/tmp/druid/napa-flow.tsv.gz" } }, "tuningConfig" : { "type": "hadoop", "targetPartitionSize" : 10000, "maxRowsInMemory" : 75000 } } } Note: I will add the "index" ingestion spec soon too, so users can ingest data into Druid from their local file system or they can go with hadoop file system.

jmedel · ‎03-30-2018

overlord-log.txtHi Nishant, The input file doesn't have a header row.napa-flowtsv.txt But, I utilized your suggestion about adding the "hasHeaderRow = true" and that caused the task to complete successfully. Here is my latest Stack trace with the ERROR I noticed: 2018-03-29 15:54:37,879 pool-1-thread-1 ERROR Unable to register shutdown hook because JVM is shutting down. java.lang.IllegalStateException: Not started at io.druid.common.config.Log4jShutdown.addShutdownCallback(Log4jShutdown.java:46) at org.apache.logging.log4j.core.impl.Log4jContextFactory.addShutdownCallback(Log4jContextFactory.java:273) at org.apache.logging.log4j.core.LoggerContext.setUpShutdownHook(LoggerContext.java:256) at org.apache.logging.log4j.core.LoggerContext.start(LoggerContext.java:216) at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:145) at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:41) at org.apache.logging.log4j.LogManager.getContext(LogManager.java:182) at org.apache.logging.log4j.spi.AbstractLoggerAdapter.getContext(AbstractLoggerAdapter.java:103) at org.apache.logging.slf4j.Log4jLoggerFactory.getContext(Log4jLoggerFactory.java:43) at org.apache.logging.log4j.spi.AbstractLoggerAdapter.getLogger(AbstractLoggerAdapter.java:42) at org.apache.logging.slf4j.Log4jLoggerFactory.getLogger(Log4jLoggerFactory.java:29) at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:253) at org.apache.commons.logging.impl.SLF4JLogFactory.getInstance(SLF4JLogFactory.java:155) at org.apache.commons.logging.impl.SLF4JLogFactory.getInstance(SLF4JLogFactory.java:132) at org.apache.commons.logging.LogFactory.getLog(LogFactory.java:273) at org.apache.hadoop.hdfs.LeaseRenewer.<clinit>(LeaseRenewer.java:72) at org.apache.hadoop.hdfs.DFSClient.getLeaseRenewer(DFSClient.java:830) at org.apache.hadoop.hdfs.DFSClient.close(DFSClient.java:968) at org.apache.hadoop.hdfs.DistributedFileSystem.close(DistributedFileSystem.java:1214) at org.apache.hadoop.fs.FileSystem$Cache.closeAll(FileSystem.java:2886) at org.apache.hadoop.fs.FileSystem$Cache$ClientFinalizer.run(FileSystem.java:2903) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)

jmedel · ‎03-29-2018

I am developing an HDP sandbox tutorial that teaches users how they can use Druid's Batch Ingestion Method to load local files into Druid. After I store the USGS data into a local file and submit an ingestion spec referring to the local file to Druid Overlord, I get that the task fails meaning the data didn't load into Druid. Here is my batch ingestion spec: { "type" : "index", "spec" : { "dataSchema" : { "dataSource" : "usgs", "parser" : { "type" : "string", "parseSpec" : { "format" : "tsv", "timestampSpec" : { "column" : "dates", "format" : "auto" }, "dimensionsSpec" : { "dimensions": ["staid","val","dates"], "dimensionExclusions" : [], "spatialDimensions" : [] } } }, "metricsSpec" : [ { "type" : "count", "name" : "count" }, { "type" : "doubleSum", "name" : "avgFlowCuFtsec", "fieldName" : "val" } ], "granularitySpec" : { "type" : "uniform", "segmentGranularity" : "MONTH", "queryGranularity" : "NONE", "intervals" : [ "1963-01-01/2013-12-31" ] } }, "ioConfig" : { "type" : "index", "firehose" : { "type" : "local", "baseDir" : "/tmp/usgs/", "filter" : "napa-flow.tsv" } }, "tuningConfig" : { "type": "index", "targetPartitionSize" : 10000, "maxRowsInMemory" : 75000 } } } Here is the stack trace from Druid's Overlord: 2018-03-29T14:09:28,013 INFO [main] io.druid.guice.PropertiesModule - Loading properties from common.runtime.properties 2018-03-29T14:09:28,015 INFO [main] io.druid.guice.PropertiesModule - Loading properties from runtime.properties 2018-03-29T14:09:28,042 INFO [main] org.hibernate.validator.internal.util.Version - HV000001: Hibernate Validator 5.1.3.Final 2018-03-29T14:09:28,411 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.segment.NullValueHandlingConfig] from props[druid.null.handling.] as [io.druid.segment.NullValueHandlingConfig@6995bf68] 2018-03-29T14:09:28,460 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.guice.ExtensionsConfig] from props[druid.extensions.] as [ExtensionsConfig{searchCurrentClassloader=true, directory='/usr/hdp/current/druid-middlemanager/extensions', hadoopDependenciesDir='/usr/hdp/current/druid-middlemanager/hadoop-dependencies', hadoopContainerDruidClasspath='null', addExtensionsToHadoopContainer=false, loadList=[druid-datasketches, druid-hdfs-storage, druid-kafka-indexing-service]}] 2018-03-29T14:09:28,510 INFO [main] io.druid.initialization.Initialization - Loading extension [druid-datasketches] for class [interface io.druid.cli.CliCommandCreator] 2018-03-29T14:09:28,518 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-datasketches/druid-datasketches-0.10.1.2.6.4.0-91.jar] 2018-03-29T14:09:28,518 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-datasketches/sketches-core-0.8.4.jar] 2018-03-29T14:09:28,518 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-datasketches/memory-0.8.4.jar] 2018-03-29T14:09:28,520 INFO [main] io.druid.initialization.Initialization - Loading extension [druid-hdfs-storage] for class [interface io.druid.cli.CliCommandCreator] 2018-03-29T14:09:28,522 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/hadoop-common-2.7.3.2.6.4.0-91.jar] 2018-03-29T14:09:28,522 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/okio-1.4.0.jar] 2018-03-29T14:09:28,522 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/curator-client-2.11.0.jar] 2018-03-29T14:09:28,522 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/commons-beanutils-1.7.0.jar] 2018-03-29T14:09:28,522 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/jsp-api-2.1.jar] 2018-03-29T14:09:28,522 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/snappy-java-1.0.4.1.jar] 2018-03-29T14:09:28,523 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/stax-api-1.0-2.jar] 2018-03-29T14:09:28,523 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/servlet-api-2.5.jar] 2018-03-29T14:09:28,523 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/hadoop-client-2.7.3.2.6.4.0-91.jar] 2018-03-29T14:09:28,523 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/hadoop-mapreduce-client-jobclient-2.7.3.2.6.4.0-91.jar] 2018-03-29T14:09:28,523 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/okhttp-2.4.0.jar] 2018-03-29T14:09:28,523 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/commons-logging-1.1.1.jar] 2018-03-29T14:09:28,523 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/api-util-1.0.0-M20.jar] 2018-03-29T14:09:28,524 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/apacheds-i18n-2.0.0-M15.jar] 2018-03-29T14:09:28,524 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/curator-recipes-2.11.0.jar] 2018-03-29T14:09:28,524 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/netty-all-4.1.11.Final.jar] 2018-03-29T14:09:28,524 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/aws-java-sdk-kms-1.10.77.jar] 2018-03-29T14:09:28,524 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/druid-hdfs-storage-0.10.1.2.6.4.0-91.jar] 2018-03-29T14:09:28,525 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/nimbus-jose-jwt-3.9.jar] 2018-03-29T14:09:28,525 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/xmlenc-0.52.jar] 2018-03-29T14:09:28,525 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/htrace-core-3.1.0-incubating.jar] 2018-03-29T14:09:28,525 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/xz-1.0.jar] 2018-03-29T14:09:28,525 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/commons-beanutils-core-1.8.0.jar] 2018-03-29T14:09:28,525 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/gson-2.2.4.jar] 2018-03-29T14:09:28,526 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/jackson-core-2.4.6.jar] 2018-03-29T14:09:28,526 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/azure-storage-5.4.0.jar] 2018-03-29T14:09:28,526 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/curator-framework-2.11.0.jar] 2018-03-29T14:09:28,526 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/hadoop-mapreduce-client-core-2.7.3.2.6.4.0-91.jar] 2018-03-29T14:09:28,526 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/jackson-xc-1.9.13.jar] 2018-03-29T14:09:28,526 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/commons-configuration-1.6.jar] 2018-03-29T14:09:28,526 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/jackson-dataformat-cbor-2.5.3.jar] 2018-03-29T14:09:28,526 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/hadoop-yarn-api-2.7.3.2.6.4.0-91.jar] 2018-03-29T14:09:28,527 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/hadoop-yarn-client-2.7.3.2.6.4.0-91.jar] 2018-03-29T14:09:28,527 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/aws-java-sdk-core-1.10.77.jar] 2018-03-29T14:09:28,527 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/hadoop-hdfs-2.7.3.2.6.4.0-91.jar] 2018-03-29T14:09:28,527 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/hadoop-yarn-registry-2.7.3.2.6.4.0-91.jar] 2018-03-29T14:09:28,527 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/azure-keyvault-core-0.8.0.jar] 2018-03-29T14:09:28,527 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/hadoop-yarn-common-2.7.3.2.6.4.0-91.jar] 2018-03-29T14:09:28,527 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/api-asn1-api-1.0.0-M20.jar] 2018-03-29T14:09:28,528 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/hadoop-yarn-server-common-2.7.3.2.6.4.0-91.jar] 2018-03-29T14:09:28,528 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/commons-lang3-3.4.jar] 2018-03-29T14:09:28,528 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/hadoop-auth-2.7.3.2.6.4.0-91.jar] 2018-03-29T14:09:28,528 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/commons-net-3.1.jar] 2018-03-29T14:09:28,528 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/paranamer-2.3.jar] 2018-03-29T14:09:28,528 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/jersey-client-1.9.jar] 2018-03-29T14:09:28,528 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/hadoop-mapreduce-client-common-2.7.3.2.6.4.0-91.jar] 2018-03-29T14:09:28,529 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/avro-1.7.4.jar] 2018-03-29T14:09:28,529 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/jackson-databind-2.4.6.jar] 2018-03-29T14:09:28,529 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/jaxb-api-2.2.2.jar] 2018-03-29T14:09:28,529 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/jackson-annotations-2.4.6.jar] 2018-03-29T14:09:28,529 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/httpclient-4.5.1.jar] 2018-03-29T14:09:28,529 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/jackson-jaxrs-1.9.13.jar] 2018-03-29T14:09:28,529 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/xml-apis-1.3.04.jar] 2018-03-29T14:09:28,530 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/commons-compress-1.4.1.jar] 2018-03-29T14:09:28,530 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/xercesImpl-2.9.1.jar] 2018-03-29T14:09:28,530 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/joda-time-2.9.9.jar] 2018-03-29T14:09:28,530 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/jetty-sslengine-6.1.26.hwx.jar] 2018-03-29T14:09:28,530 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/commons-digester-1.8.jar] 2018-03-29T14:09:28,530 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/hadoop-aws-2.7.3.2.6.4.0-91.jar] 2018-03-29T14:09:28,530 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/json-smart-1.1.1.jar] 2018-03-29T14:09:28,530 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/apacheds-kerberos-codec-2.0.0-M15.jar] 2018-03-29T14:09:28,531 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/netty-3.6.2.Final.jar] 2018-03-29T14:09:28,531 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/aws-java-sdk-s3-1.10.77.jar] 2018-03-29T14:09:28,531 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/httpcore-4.4.3.jar] 2018-03-29T14:09:28,531 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/guava-16.0.1.jar] 2018-03-29T14:09:28,531 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/hadoop-mapreduce-client-shuffle-2.7.3.2.6.4.0-91.jar] 2018-03-29T14:09:28,531 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/commons-collections-3.2.2.jar] 2018-03-29T14:09:28,531 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/hadoop-mapreduce-client-app-2.7.3.2.6.4.0-91.jar] 2018-03-29T14:09:28,531 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/jcip-annotations-1.0.jar] 2018-03-29T14:09:28,532 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/commons-math3-3.6.1.jar] 2018-03-29T14:09:28,532 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/leveldbjni-all-1.8.jar] 2018-03-29T14:09:28,547 INFO [main] io.druid.initialization.Initialization - Loading extension [druid-kafka-indexing-service] for class [interface io.druid.cli.CliCommandCreator] 2018-03-29T14:09:28,547 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-kafka-indexing-service/snappy-java-1.1.2.6.jar] 2018-03-29T14:09:28,547 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-kafka-indexing-service/slf4j-api-1.6.4.jar] 2018-03-29T14:09:28,547 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-kafka-indexing-service/druid-kafka-indexing-service-0.10.1.2.6.4.0-91.jar] 2018-03-29T14:09:28,548 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-kafka-indexing-service/lz4-1.3.0.jar] 2018-03-29T14:09:28,548 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-kafka-indexing-service/kafka-clients-0.10.2.0.jar] 2018-03-29T14:09:28,616 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.guice.ModulesConfig] from props[druid.modules.] as [ModulesConfig{excludeList=[]}] 2018-03-29T14:09:28,772 INFO [main] io.druid.initialization.Initialization - Loading extension [druid-datasketches] for class [interface io.druid.initialization.DruidModule] 2018-03-29T14:09:28,775 INFO [main] io.druid.initialization.Initialization - Adding implementation [io.druid.query.aggregation.datasketches.theta.SketchModule] for class [interface io.druid.initialization.DruidModule] from local file system extension 2018-03-29T14:09:28,775 INFO [main] io.druid.initialization.Initialization - Adding implementation [io.druid.query.aggregation.datasketches.theta.oldapi.OldApiSketchModule] for class [interface io.druid.initialization.DruidModule] from local file system extension 2018-03-29T14:09:28,776 INFO [main] io.druid.initialization.Initialization - Loading extension [druid-hdfs-storage] for class [interface io.druid.initialization.DruidModule] 2018-03-29T14:09:28,778 INFO [main] io.druid.initialization.Initialization - Adding implementation [io.druid.storage.hdfs.HdfsStorageDruidModule] for class [interface io.druid.initialization.DruidModule] from local file system extension 2018-03-29T14:09:28,778 INFO [main] io.druid.initialization.Initialization - Loading extension [druid-kafka-indexing-service] for class [interface io.druid.initialization.DruidModule] 2018-03-29T14:09:28,779 INFO [main] io.druid.initialization.Initialization - Adding implementation [io.druid.indexing.kafka.KafkaIndexTaskModule] for class [interface io.druid.initialization.DruidModule] from local file system extension 2018-03-29T14:09:29,499 WARN [main] org.apache.hadoop.util.NativeCodeLoader - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2018-03-29T14:09:30,292 WARN [main] org.apache.hadoop.hdfs.shortcircuit.DomainSocketFactory - The short-circuit local reads feature cannot be used because libhadoop cannot be loaded. 2018-03-29T14:09:30,672 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.server.metrics.DruidMonitorSchedulerConfig] from props[druid.monitoring.] as [io.druid.server.metrics.DruidMonitorSchedulerConfig@24eeac69] 2018-03-29T14:09:30,679 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.server.metrics.MonitorsConfig] from props[druid.monitoring.] as [MonitorsConfig{monitors=[class com.metamx.metrics.JvmMonitor]}] 2018-03-29T14:09:30,698 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.server.DruidNode] from props[druid.] as [DruidNode{serviceName='druid/middlemanager', host='sandbox-hdp.hortonworks.com', port=8100}] 2018-03-29T14:09:30,774 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.query.lookup.LookupConfig] from props[druid.lookup.] as [LookupConfig{snapshotWorkingDir=''}] 2018-03-29T14:09:30,844 INFO [main] io.druid.server.metrics.MetricsModule - Adding monitor[com.metamx.metrics.JvmMonitor@485caa8f] 2018-03-29T14:09:30,847 INFO [main] io.druid.server.metrics.MetricsModule - Adding monitor[io.druid.query.ExecutorServiceMonitor@4d157493] 2018-03-29T14:09:30,848 INFO [main] io.druid.server.metrics.MetricsModule - Adding monitor[io.druid.server.initialization.jetty.JettyServerModule$JettyMonitor@76e3b45b] 2018-03-29T14:09:30,854 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.server.log.StartupLoggingConfig] from props[druid.startup.logging.] as [io.druid.server.log.StartupLoggingConfig@3e28fee1] 2018-03-29T14:09:30,854 INFO [main] io.druid.cli.CliPeon - Starting up with processors[4], memory[308,281,344], maxMemory[1,908,932,608]. 2018-03-29T14:09:30,857 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.storage.hdfs.HdfsKerberosConfig] from props[druid.hadoop.security.kerberos.] as [io.druid.storage.hdfs.HdfsKerberosConfig@0] 2018-03-29T14:09:30,864 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.indexing.common.config.TaskConfig] from props[druid.indexer.task.] as [io.druid.indexing.common.config.TaskConfig@2eda072] 2018-03-29T14:09:30,867 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.guice.http.DruidHttpClientConfig] from props[druid.global.http.] as [io.druid.guice.http.DruidHttpClientConfig@4091b9c3] 2018-03-29T14:09:30,928 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.client.indexing.IndexingServiceSelectorConfig] from props[druid.selectors.indexing.] as [io.druid.client.indexing.IndexingServiceSelectorConfig@7e7e962d] 2018-03-29T14:09:30,931 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.curator.CuratorConfig] from props[druid.zk.service.] as [io.druid.curator.CuratorConfig@7499eac7] 2018-03-29T14:09:30,934 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.curator.ExhibitorConfig] from props[druid.exhibitor.service.] as [io.druid.curator.ExhibitorConfig@d88f893] 2018-03-29T14:09:30,938 WARN [main] org.apache.curator.retry.ExponentialBackoffRetry - maxRetries too large (30). Pinning to 29 2018-03-29T14:09:30,959 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.server.initialization.CuratorDiscoveryConfig] from props[druid.discovery.curator.] as [io.druid.server.initialization.CuratorDiscoveryConfig@5b000fe6] 2018-03-29T14:09:30,980 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.indexing.common.RetryPolicyConfig] from props[druid.peon.taskActionClient.retry.] as [io.druid.indexing.common.RetryPolicyConfig@7186b202] 2018-03-29T14:09:30,982 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.storage.hdfs.HdfsDataSegmentPusherConfig] from props[druid.storage.] as [io.druid.storage.hdfs.HdfsDataSegmentPusherConfig@1a538ed8] 2018-03-29T14:09:30,983 INFO [main] io.druid.storage.hdfs.HdfsDataSegmentPusher - Configured HDFS as deep storage 2018-03-29T14:09:30,986 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.segment.loading.LocalDataSegmentPusherConfig] from props[druid.storage.] as [io.druid.segment.loading.LocalDataSegmentPusherConfig@270a620] 2018-03-29T14:09:30,991 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.client.DruidServerConfig] from props[druid.server.] as [io.druid.client.DruidServerConfig@4dfe8b37] 2018-03-29T14:09:30,994 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.server.initialization.BatchDataSegmentAnnouncerConfig] from props[druid.announcer.] as [io.druid.server.initialization.BatchDataSegmentAnnouncerConfig@3c232051] 2018-03-29T14:09:30,996 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.server.initialization.ZkPathsConfig] from props[druid.zk.paths.] as [io.druid.server.initialization.ZkPathsConfig@22e2266d] 2018-03-29T14:09:31,002 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[interface io.druid.server.coordination.DataSegmentAnnouncerProvider] from props[druid.announcer.] as [io.druid.server.coordination.BatchDataSegmentAnnouncerProvider@59fc6d05] 2018-03-29T14:09:31,004 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.client.coordinator.CoordinatorSelectorConfig] from props[druid.selectors.coordinator.] as [io.druid.client.coordinator.CoordinatorSelectorConfig@7e87ef9e] 2018-03-29T14:09:31,007 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.segment.realtime.plumber.CoordinatorBasedSegmentHandoffNotifierConfig] from props[druid.segment.handoff.] as [io.druid.segment.realtime.plumber.CoordinatorBasedSegmentHandoffNotifierConfig@7c241a9f] 2018-03-29T14:09:31,009 INFO [main] org.skife.config.ConfigurationObjectFactory - Assigning value [50000000] for [druid.processing.buffer.sizeBytes] on [io.druid.query.DruidProcessingConfig#intermediateComputeSizeBytes()] 2018-03-29T14:09:31,011 INFO [main] org.skife.config.ConfigurationObjectFactory - Using method itself for [druid.computation.buffer.poolCacheMaxCount, ${base_path}.buffer.poolCacheMaxCount] on [io.druid.query.DruidProcessingConfig#poolCacheMaxCount()] 2018-03-29T14:09:31,012 INFO [main] org.skife.config.ConfigurationObjectFactory - Using method itself for [${base_path}.fifo] on [io.druid.query.DruidProcessingConfig#isFifo()] 2018-03-29T14:09:31,012 INFO [main] org.skife.config.ConfigurationObjectFactory - Assigning value [2] for [druid.processing.numThreads] on [io.druid.query.DruidProcessingConfig#getNumThreadsConfigured()] 2018-03-29T14:09:31,012 INFO [main] org.skife.config.ConfigurationObjectFactory - Using method itself for [${base_path}.numMergeBuffers] on [io.druid.query.DruidProcessingConfig#getNumMergeBuffersConfigured()] 2018-03-29T14:09:31,013 INFO [main] org.skife.config.ConfigurationObjectFactory - Using method itself for [${base_path}.columnCache.sizeBytes] on [io.druid.query.DruidProcessingConfig#columnCacheSizeBytes()] 2018-03-29T14:09:31,013 INFO [main] org.skife.config.ConfigurationObjectFactory - Using method itself for [${base_path}.tmpDir] on [io.druid.query.DruidProcessingConfig#getTmpDir()] 2018-03-29T14:09:31,013 INFO [main] org.skife.config.ConfigurationObjectFactory - Assigning default value [processing-%s] for [${base_path}.formatString] on [io.druid.java.util.common.concurrent.ExecutorServiceConfig#getFormatString()] 2018-03-29T14:09:31,083 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[interface io.druid.client.cache.CacheProvider] from props[druid.cache.] as [io.druid.client.cache.LocalCacheProvider@182e7eda] 2018-03-29T14:09:31,091 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.client.cache.CacheConfig] from props[druid.realtime.cache.] as [io.druid.client.cache.CacheConfig@27dc627a] 2018-03-29T14:09:31,099 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.server.initialization.ServerConfig] from props[druid.server.http.] as [ServerConfig{numThreads=50, maxIdleTime=PT5M, defaultQueryTimeout=300000, maxScatterGatherBytes=9223372036854775807}] 2018-03-29T14:09:31,107 INFO [main] org.eclipse.jetty.util.log - Logging initialized @3834ms 2018-03-29T14:09:31,221 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.query.search.search.SearchQueryConfig] from props[druid.query.search.] as [io.druid.query.search.search.SearchQueryConfig@622d7e4] 2018-03-29T14:09:31,226 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.query.metadata.SegmentMetadataQueryConfig] from props[druid.query.segmentMetadata.] as [io.druid.query.metadata.SegmentMetadataQueryConfig@26582ca] 2018-03-29T14:09:31,229 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.query.groupby.GroupByQueryConfig] from props[druid.query.groupBy.] as [GroupByQueryConfig{defaultStrategy='v2', singleThreaded=false, maxIntermediateRows=50000, maxResults=500000, bufferGrouperMaxSize=2147483647, bufferGrouperMaxLoadFactor=0.0, bufferGrouperInitialBuckets=0, maxMergingDictionarySize=100000000, maxOnDiskStorage=0, forcePushDownLimit=false, forceHashAggregation=false}] 2018-03-29T14:09:31,244 INFO [main] io.druid.offheap.OffheapBufferGenerator - Allocating new intermediate processing buffer[0] of size[50,000,000] 2018-03-29T14:09:31,254 INFO [main] io.druid.offheap.OffheapBufferGenerator - Allocating new intermediate processing buffer[1] of size[50,000,000] 2018-03-29T14:09:31,263 INFO [main] io.druid.offheap.OffheapBufferGenerator - Allocating new result merging buffer[0] of size[50,000,000] 2018-03-29T14:09:31,314 INFO [main] io.druid.offheap.OffheapBufferGenerator - Allocating new result merging buffer[1] of size[50,000,000] 2018-03-29T14:09:31,370 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.query.scan.ScanQueryConfig] from props[druid.query.scan.] as [ScanQueryConfig{legacy=false}] 2018-03-29T14:09:31,373 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.query.select.SelectQueryConfig] from props[druid.query.select.] as [io.druid.query.select.SelectQueryConfig@707865bd] 2018-03-29T14:09:31,375 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.query.topn.TopNQueryConfig] from props[druid.query.topN.] as [io.druid.query.topn.TopNQueryConfig@3ebc955b] 2018-03-29T14:09:31,379 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[interface io.druid.server.log.RequestLoggerProvider] from props[druid.request.logging.] as [io.druid.server.log.NoopRequestLoggerProvider@6ffd4c0d] 2018-03-29T14:09:31,380 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.server.security.AuthConfig] from props[druid.auth.] as [AuthConfig{enabled=false}] 2018-03-29T14:09:31,388 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.query.lookup.LookupListeningAnnouncerConfig] from props[druid.lookup.] as [ListeningAnnouncerConfig{listenersPath='/druid/listeners'}] 2018-03-29T14:09:31,389 INFO [main] io.druid.java.util.common.lifecycle.Lifecycle$AnnotationBasedHandler - Invoking start method[public void io.druid.initialization.Log4jShutterDownerModule$Log4jShutterDowner.start()] on object[io.druid.initialization.Log4jShutterDownerModule$Log4jShutterDowner@3dbd7107]. 2018-03-29T14:09:31,391 INFO [main] io.druid.java.util.common.lifecycle.Lifecycle$AnnotationBasedHandler - Invoking start method[public void com.metamx.emitter.service.ServiceEmitter.start()] on object[com.metamx.emitter.service.ServiceEmitter@5a090f62]. 2018-03-29T14:09:31,392 INFO [main] io.druid.java.util.common.lifecycle.Lifecycle$AnnotationBasedHandler - Invoking start method[public void io.druid.query.lookup.LookupReferencesManager.start()] on object[io.druid.query.lookup.LookupReferencesManager@89caf47]. 2018-03-29T14:09:31,393 INFO [main] io.druid.query.lookup.LookupReferencesManager - LookupReferencesManager is starting. 2018-03-29T14:09:31,393 INFO [main] io.druid.query.lookup.LookupReferencesManager - LookupReferencesManager is started. 2018-03-29T14:09:31,394 INFO [main] io.druid.java.util.common.lifecycle.Lifecycle$AnnotationBasedHandler - Invoking start method[public void com.metamx.metrics.MonitorScheduler.start()] on object[com.metamx.metrics.MonitorScheduler@be164d8]. 2018-03-29T14:09:31,403 INFO [main] io.druid.java.util.common.lifecycle.Lifecycle$AnnotationBasedHandler - Invoking start method[public void io.druid.storage.hdfs.HdfsStorageAuthentication.authenticate()] on object[io.druid.storage.hdfs.HdfsStorageAuthentication@2e56b4d]. 2018-03-29T14:09:31,405 INFO [main] com.metamx.common.lifecycle.Lifecycle$AnnotationBasedHandler - Invoking start method[public void com.metamx.http.client.NettyHttpClient.start()] on object[com.metamx.http.client.NettyHttpClient@36ad229b]. 2018-03-29T14:09:31,405 INFO [main] io.druid.curator.CuratorModule - Starting Curator 2018-03-29T14:09:31,405 INFO [main] org.apache.curator.framework.imps.CuratorFrameworkImpl - Starting 2018-03-29T14:09:31,413 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:zookeeper.version=3.4.6-91--1, built on 01/04/2018 09:27 GMT 2018-03-29T14:09:31,413 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:host.name=sandbox-hdp.hortonworks.com 2018-03-29T14:09:31,413 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:java.version=1.8.0_161 2018-03-29T14:09:31,413 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:java.vendor=Oracle Corporation 2018-03-29T14:09:31,413 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:java.home=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.161-3.b14.el6_9.x86_64/jre 2018-03-29T14:09:31,413 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:java.class.path=/usr/hdp/current/druid-middlemanager/conf/_common:/usr/hdp/current/druid-middlemanager/conf/middleManager:/usr/hdp/current/druid-middlemanager/lib/antlr4-runtime-4.5.1.jar:/usr/hdp/current/druid-middlemanager/lib/druid-sql-0.10.1.2.6.4.0-91.jar:/usr/hdp/current/druid-middlemanager/lib/plexus-interpolation-1.19.jar:/usr/hdp/current/druid-middlemanager/lib/guice-servlet-4.1.0.jar:/usr/hdp/current/druid-middlemanager/lib/jackson-jaxrs-smile-provider-2.4.6.jar:/usr/hdp/current/druid-middlemanager/lib/curator-client-2.11.0.jar:/usr/hdp/current/druid-middlemanager/lib/tesla-aether-0.0.5.jar:/usr/hdp/current/druid-middlemanager/lib/wagon-provider-api-2.4.jar:/usr/hdp/current/druid-middlemanager/lib/extendedset-0.10.1.2.6.4.0-91.jar:/usr/hdp/current/druid-middlemanager/lib/jackson-dataformat-smile-2.4.6.jar:/usr/hdp/current/druid-middlemanager/lib/maven-model-3.1.1.jar:/usr/hdp/current/druid-middlemanager/lib/json-path-2.1.0.jar:/usr/hdp/current/druid-middlemanager/lib/jetty-util-9.3.19.v20170502.jar:/usr/hdp/current/druid-middlemanager/lib/commons-compiler-2.7.6.jar:/usr/hdp/current/druid-middlemanager/lib/log4j-api-2.5.jar:/usr/hdp/current/druid-middlemanager/lib/jcl-over-slf4j-1.7.12.jar:/usr/hdp/current/druid-middlemanager/lib/druid-server-0.10.1.2.6.4.0-91.jar:/usr/hdp/current/druid-middlemanager/lib/jetty-proxy-9.3.19.v20170502.jar:/usr/hdp/current/druid-middlemanager/lib/commons-pool-1.6.jar:/usr/hdp/current/druid-middlemanager/lib/jsr311-api-1.1.1.jar:/usr/hdp/current/druid-middlemanager/lib/jsr305-2.0.1.jar:/usr/hdp/current/druid-middlemanager/lib/druid-indexing-service-0.10.1.2.6.4.0-91.jar:/usr/hdp/current/druid-middlemanager/lib/commons-logging-1.1.1.jar:/usr/hdp/current/druid-middlemanager/lib/okhttp-1.0.2.jar:/usr/hdp/current/druid-middlemanager/lib/RoaringBitmap-0.5.18.jar:/usr/hdp/current/druid-middlemanager/lib/http-client-1.1.0.jar:/usr/hdp/current/druid-middlemanager/lib/curator-recipes-2.11.0.jar:/usr/hdp/current/druid-middlemanager/lib/aopalliance-1.0.jar:/usr/hdp/current/druid-middlemanager/lib/netty-all-4.1.11.Final.jar:/usr/hdp/current/druid-middlemanager/lib/derbyclient-10.11.1.1.jar:/usr/hdp/current/druid-middlemanager/lib/bytebuffer-collections-0.10.1.2.6.4.0-91.jar:/usr/hdp/current/druid-middlemanager/lib/protobuf-java-3.1.0.jar:/usr/hdp/current/druid-middlemanager/lib/aether-spi-0.9.0.M2.jar:/usr/hdp/current/druid-middlemanager/lib/derbynet-10.11.1.1.jar:/usr/hdp/current/druid-middlemanager/lib/compress-lzf-1.0.3.jar:/usr/hdp/current/druid-middlemanager/lib/disruptor-3.3.6.jar:/usr/hdp/current/druid-middlemanager/lib/jetty-http-9.3.19.v20170502.jar:/usr/hdp/current/druid-middlemanager/lib/maven-settings-3.1.1.jar:/usr/hdp/current/druid-middlemanager/lib/bcprov-jdk15on-1.52.jar:/usr/hdp/current/druid-middlemanager/lib/classmate-1.0.0.jar:/usr/hdp/current/druid-middlemanager/lib/opencsv-2.3.jar:/usr/hdp/current/druid-middlemanager/lib/druid-aws-common-0.10.1.2.6.4.0-91.jar:/usr/hdp/current/druid-middlemanager/lib/aws-java-sdk-ec2-1.10.77.jar:/usr/hdp/current/druid-middlemanager/lib/druid-services-0.10.1.2.6.4.0-91.jar:/usr/hdp/current/druid-middlemanager/lib/javax.inject-1.jar:/usr/hdp/current/druid-middlemanager/lib/jackson-datatype-joda-2.4.6.jar:/usr/hdp/current/druid-middlemanager/lib/commons-dbcp-1.4.jar:/usr/hdp/current/druid-middlemanager/lib/jackson-datatype-guava-2.4.6.jar:/usr/hdp/current/druid-middlemanager/lib/jackson-core-2.4.6.jar:/usr/hdp/current/druid-middlemanager/lib/jline-0.9.94.jar:/usr/hdp/current/druid-middlemanager/lib/base64-2.3.8.jar:/usr/hdp/current/druid-middlemanager/lib/calcite-core-1.12.0.jar:/usr/hdp/current/druid-middlemanager/lib/mapdb-1.0.8.jar:/usr/hdp/current/druid-middlemanager/lib/curator-framework-2.11.0.jar:/usr/hdp/current/druid-middlemanager/lib/log4j-core-2.5.jar:/usr/hdp/current/druid-middlemanager/lib/jetty-servlets-9.3.19.v20170502.jar:/usr/hdp/current/druid-middlemanager/lib/druid-console-0.0.2.jar:/usr/hdp/current/druid-middlemanager/lib/maven-model-builder-3.1.1.jar:/usr/hdp/current/druid-middlemanager/lib/config-magic-0.9.jar:/usr/hdp/current/druid-middlemanager/lib/emitter-0.4.5.jar:/usr/hdp/current/druid-middlemanager/lib/slf4j-api-1.6.4.jar:/usr/hdp/current/druid-middlemanager/lib/jboss-logging-3.1.3.GA.jar:/usr/hdp/current/druid-middlemanager/lib/commons-cli-1.2.jar:/usr/hdp/current/druid-middlemanager/lib/guice-4.1.0.jar:/usr/hdp/current/druid-middlemanager/lib/druid-common-0.10.1.2.6.4.0-91.jar:/usr/hdp/current/druid-middlemanager/lib/jackson-dataformat-cbor-2.5.3.jar:/usr/hdp/current/druid-middlemanager/lib/derby-10.11.1.1.jar:/usr/hdp/current/druid-middlemanager/lib/aether-api-0.9.0.M2.jar:/usr/hdp/current/druid-middlemanager/lib/commons-io-2.5.jar:/usr/hdp/current/druid-middlemanager/lib/netty-3.10.6.Final.jar:/usr/hdp/current/druid-middlemanager/lib/javax.el-3.0.0.jar:/usr/hdp/current/druid-middlemanager/lib/janino-2.7.6.jar:/usr/hdp/current/druid-middlemanager/lib/plexus-utils-3.0.15.jar:/usr/hdp/current/druid-middlemanager/lib/jackson-jaxrs-base-2.4.6.jar:/usr/hdp/current/druid-middlemanager/lib/aws-java-sdk-core-1.10.77.jar:/usr/hdp/current/druid-middlemanager/lib/javax.servlet-api-3.1.0.jar:/usr/hdp/current/druid-middlemanager/lib/log4j-1.2-api-2.5.jar:/usr/hdp/current/druid-middlemanager/lib/avatica-core-1.9.0.jar:/usr/hdp/current/druid-middlemanager/lib/aether-impl-0.9.0.M2.jar:/usr/hdp/current/druid-middlemanager/lib/jetty-security-9.3.19.v20170502.jar:/usr/hdp/current/druid-middlemanager/lib/spymemcached-2.12.3.jar:/usr/hdp/current/druid-middlemanager/lib/maven-aether-provider-3.1.1.jar:/usr/hdp/current/druid-middlemanager/lib/aggdesigner-algorithm-6.0.jar:/usr/hdp/current/druid-middlemanager/lib/aether-connector-okhttp-0.0.9.jar:/usr/hdp/current/druid-middlemanager/lib/commons-pool2-2.2.jar:/usr/hdp/current/druid-middlemanager/lib/jackson-databind-2.4.6.jar:/usr/hdp/current/druid-middlemanager/lib/asm-tree-5.2.jar:/usr/hdp/current/druid-middlemanager/lib/jetty-continuation-9.3.19.v20170502.jar:/usr/hdp/current/druid-middlemanager/lib/java-util-0.28.2.jar:/usr/hdp/current/druid-middlemanager/lib/maven-repository-metadata-3.1.1.jar:/usr/hdp/current/druid-middlemanager/lib/jackson-annotations-2.4.6.jar:/usr/hdp/current/druid-middlemanager/lib/calcite-linq4j-1.12.0.jar:/usr/hdp/current/druid-middlemanager/lib/jersey-servlet-1.19.3.jar:/usr/hdp/current/druid-middlemanager/lib/jackson-mapper-asl-1.9.13.jar:/usr/hdp/current/druid-middlemanager/lib/asm-5.2.jar:/usr/hdp/current/druid-middlemanager/lib/asm-commons-5.2.jar:/usr/hdp/current/druid-middlemanager/lib/httpclient-4.5.1.jar:/usr/hdp/current/druid-middlemanager/lib/lz4-1.3.0.jar:/usr/hdp/current/druid-middlemanager/lib/avatica-metrics-1.9.0.jar:/usr/hdp/current/druid-middlemanager/lib/jetty-servlet-9.3.19.v20170502.jar:/usr/hdp/current/druid-middlemanager/lib/jackson-core-asl-1.9.13.jar:/usr/hdp/current/druid-middlemanager/lib/log4j-slf4j-impl-2.5.jar:/usr/hdp/current/druid-middlemanager/lib/javax.el-api-3.0.0.jar:/usr/hdp/current/druid-middlemanager/lib/irc-api-1.0-0014.jar:/usr/hdp/current/druid-middlemanager/lib/java-util-0.10.1.2.6.4.0-91.jar:/usr/hdp/current/druid-middlemanager/lib/jersey-guice-1.19.3.jar:/usr/hdp/current/druid-middlemanager/lib/activation-1.1.1.jar:/usr/hdp/current/druid-middlemanager/lib/druid-indexing-hadoop-0.10.1.2.6.4.0-91.jar:/usr/hdp/current/druid-middlemanager/lib/commons-lang-2.6.jar:/usr/hdp/current/druid-middlemanager/lib/jersey-server-1.19.3.jar:/usr/hdp/current/druid-middlemanager/lib/aether-connector-file-0.9.0.M2.jar:/usr/hdp/current/druid-middlemanager/lib/geoip2-0.4.0.jar:/usr/hdp/current/druid-middlemanager/lib/joda-time-2.9.9.jar:/usr/hdp/current/druid-middlemanager/lib/jetty-server-9.3.19.v20170502.jar:/usr/hdp/current/druid-middlemanager/lib/druid-hll-0.10.1.2.6.4.0-91.jar:/usr/hdp/current/druid-middlemanager/lib/guice-multibindings-4.1.0.jar:/usr/hdp/current/druid-middlemanager/lib/curator-x-discovery-2.11.0.jar:/usr/hdp/current/druid-middlemanager/lib/druid-api-0.10.1.2.6.4.0-91.jar:/usr/hdp/current/druid-middlemanager/lib/commons-lang3-3.2.jar:/usr/hdp/current/druid-middlemanager/lib/maxminddb-0.2.0.jar:/usr/hdp/current/druid-middlemanager/lib/jetty-io-9.3.19.v20170502.jar:/usr/hdp/current/druid-middlemanager/lib/maven-settings-builder-3.1.1.jar:/usr/hdp/current/druid-middlemanager/lib/jdbi-2.63.1.jar:/usr/hdp/current/druid-middlemanager/lib/aether-util-0.9.0.M2.jar:/usr/hdp/current/druid-middlemanager/lib/icu4j-54.1.1.jar:/usr/hdp/current/druid-middlemanager/lib/validation-api-1.1.0.Final.jar:/usr/hdp/current/druid-middlemanager/lib/jets3t-0.9.4.jar:/usr/hdp/current/druid-middlemanager/lib/jackson-module-jaxb-annotations-2.4.6.jar:/usr/hdp/current/druid-middlemanager/lib/server-metrics-0.2.8.jar:/usr/hdp/current/druid-middlemanager/lib/jetty-client-9.3.19.v20170502.jar:/usr/hdp/current/druid-middlemanager/lib/avatica-server-1.9.0.jar:/usr/hdp/current/druid-middlemanager/lib/httpcore-4.4.3.jar:/usr/hdp/current/druid-middlemanager/lib/airline-0.7.jar:/usr/hdp/current/druid-middlemanager/lib/jackson-jaxrs-json-provider-2.4.6.jar:/usr/hdp/current/druid-middlemanager/lib/druid-processing-0.10.1.2.6.4.0-91.jar:/usr/hdp/current/druid-middlemanager/lib/zookeeper-3.4.6.2.6.4.0-91.jar:/usr/hdp/current/druid-middlemanager/lib/guava-16.0.1.jar:/usr/hdp/current/druid-middlemanager/lib/commons-codec-1.7.jar:/usr/hdp/current/druid-middlemanager/lib/jersey-core-1.19.3.jar:/usr/hdp/current/druid-middlemanager/lib/commons-dbcp2-2.0.1.jar:/usr/hdp/current/druid-middlemanager/lib/hibernate-validator-5.1.3.Final.jar:/usr/hdp/current/druid-middlemanager/lib/java-xmlbuilder-1.1.jar:/usr/hdp/current/druid-middlemanager/lib/commons-math3-3.6.1.jar:/usr/hdp/current/druid-middlemanager/lib/rhino-1.7R5.jar:/usr/hdp/current/druid-middlemanager/lib/log4j-jul-2.5.jar:/usr/hdp/current/druid-middlemanager/lib/fastutil-7.2.0.jar:/usr/hdp/2.6.4.0-91/hadoop/conf 2018-03-29T14:09:31,413 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib 2018-03-29T14:09:31,413 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:java.io.tmpdir=/tmp 2018-03-29T14:09:31,413 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:java.compiler=<NA> 2018-03-29T14:09:31,413 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:os.name=Linux 2018-03-29T14:09:31,413 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:os.arch=amd64 2018-03-29T14:09:31,413 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:os.version=4.4.114-1.el7.elrepo.x86_64 2018-03-29T14:09:31,413 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:user.name=druid 2018-03-29T14:09:31,413 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:user.home=/home/druid 2018-03-29T14:09:31,413 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:user.dir=/home/druid 2018-03-29T14:09:31,414 INFO [main] org.apache.zookeeper.ZooKeeper - Initiating client connection, connectString=sandbox-hdp.hortonworks.com:2181 sessionTimeout=30000 watcher=org.apache.curator.ConnectionState@29013ef2 2018-03-29T14:09:31,434 INFO [main] io.druid.java.util.common.lifecycle.Lifecycle$AnnotationBasedHandler - Invoking start method[public void io.druid.curator.discovery.ServerDiscoverySelector.start() throws java.lang.Exception] on object[io.druid.curator.discovery.ServerDiscoverySelector@13cc0b90]. 2018-03-29T14:09:31,474 INFO [main-SendThread(sandbox-hdp.hortonworks.com:2181)] org.apache.zookeeper.ClientCnxn - Opening socket connection to server sandbox-hdp.hortonworks.com/172.17.0.2:2181. Will not attempt to authenticate using SASL (unknown error) 2018-03-29T14:09:31,481 INFO [main-SendThread(sandbox-hdp.hortonworks.com:2181)] org.apache.zookeeper.ClientCnxn - Socket connection established, initiating session, client: /172.17.0.2:42090, server: sandbox-hdp.hortonworks.com/172.17.0.2:2181 2018-03-29T14:09:31,491 INFO [main-SendThread(sandbox-hdp.hortonworks.com:2181)] org.apache.zookeeper.ClientCnxn - Session establishment complete on server sandbox-hdp.hortonworks.com/172.17.0.2:2181, sessionid = 0x16271e552860012, negotiated timeout = 30000 2018-03-29T14:09:31,494 INFO [main-EventThread] org.apache.curator.framework.state.ConnectionStateManager - State change: CONNECTED 2018-03-29T14:09:31,590 INFO [main] io.druid.java.util.common.lifecycle.Lifecycle$AnnotationBasedHandler - Invoking start method[public void io.druid.curator.announcement.Announcer.start()] on object[io.druid.curator.announcement.Announcer@1f129467]. 2018-03-29T14:09:31,591 INFO [main] io.druid.java.util.common.lifecycle.Lifecycle$AnnotationBasedHandler - Invoking start method[public void io.druid.curator.discovery.ServerDiscoverySelector.start() throws java.lang.Exception] on object[io.druid.curator.discovery.ServerDiscoverySelector@646cd766]. 2018-03-29T14:09:31,612 INFO [main] io.druid.java.util.common.lifecycle.Lifecycle$AnnotationBasedHandler - Invoking start method[public void io.druid.indexing.worker.executor.ExecutorLifecycle.start() throws java.lang.InterruptedException] on object[io.druid.indexing.worker.executor.ExecutorLifecycle@65eb3597]. 2018-03-29T14:09:31,662 INFO [main] io.druid.indexing.worker.executor.ExecutorLifecycle - Running with task: { "type" : "index", "id" : "index_usgs_2018-03-29T14:09:27.159Z", "resource" : { "availabilityGroup" : "index_usgs_2018-03-29T14:09:27.159Z", "requiredCapacity" : 1 }, "spec" : { "dataSchema" : { "dataSource" : "usgs", "parser" : { "type" : "string", "parseSpec" : { "format" : "tsv", "timestampSpec" : { "column" : "dates", "format" : "auto" }, "dimensionsSpec" : { "dimensions" : [ "staid", "val", "dates" ], "dimensionExclusions" : [ ], "spatialDimensions" : [ ] } } }, "metricsSpec" : [ { "type" : "count", "name" : "count" }, { "type" : "doubleSum", "name" : "avgFlowCuFtsec", "fieldName" : "val", "expression" : null } ], "granularitySpec" : { "type" : "uniform", "segmentGranularity" : "MONTH", "queryGranularity" : { "type" : "none" }, "rollup" : true, "intervals" : [ "1963-01-01T00:00:00.000Z/2013-12-31T00:00:00.000Z" ] } }, "ioConfig" : { "type" : "index", "firehose" : { "type" : "local", "baseDir" : "/tmp/usgs", "filter" : "napa-flow.tsv", "parser" : null }, "appendToExisting" : false }, "tuningConfig" : { "type" : "index", "targetPartitionSize" : 10000, "maxRowsInMemory" : 75000, "numShards" : null, "indexSpec" : { "bitmap" : { "type" : "concise" }, "dimensionCompression" : "lz4", "metricCompression" : "lz4", "longEncoding" : "longs" }, "maxPendingPersists" : 0, "buildV9Directly" : true, "forceExtendableShardSpecs" : false, "reportParseExceptions" : false, "publishTimeout" : 0 } }, "context" : null, "groupId" : "index_usgs_2018-03-29T14:09:27.159Z", "dataSource" : "usgs" } 2018-03-29T14:09:31,664 INFO [main] io.druid.indexing.worker.executor.ExecutorLifecycle - Attempting to lock file[/apps/druid/tasks/index_usgs_2018-03-29T14:09:27.159Z/lock]. 2018-03-29T14:09:31,666 INFO [main] io.druid.indexing.worker.executor.ExecutorLifecycle - Acquired lock file[/apps/druid/tasks/index_usgs_2018-03-29T14:09:27.159Z/lock] in 1ms. 2018-03-29T14:09:31,671 INFO [main] io.druid.indexing.common.actions.RemoteTaskActionClient - Performing action for task[index_usgs_2018-03-29T14:09:27.159Z]: LockTryAcquireAction{interval=1963-01-01T00:00:00.000Z/2014-01-01T00:00:00.000Z} 2018-03-29T14:09:31,679 INFO [main] io.druid.indexing.common.actions.RemoteTaskActionClient - Submitting action for task[index_usgs_2018-03-29T14:09:27.159Z] to overlord[http://sandbox-hdp.hortonworks.com:8090/druid/indexer/v1/action]: LockTryAcquireAction{interval=1963-01-01T00:00:00.000Z/2014-01-01T00:00:00.000Z} 2018-03-29T14:09:31,688 INFO [main] com.metamx.http.client.pool.ChannelResourceFactory - Generating: http://sandbox-hdp.hortonworks.com:8090 2018-03-29T14:09:31,728 INFO [main] com.metamx.http.client.pool.ChannelResourceFactory - Generating: http://sandbox-hdp.hortonworks.com:8090 2018-03-29T14:09:31,729 INFO [main] com.metamx.http.client.pool.ChannelResourceFactory - Generating: http://sandbox-hdp.hortonworks.com:8090 2018-03-29T14:09:31,730 INFO [main] com.metamx.http.client.pool.ChannelResourceFactory - Generating: http://sandbox-hdp.hortonworks.com:8090 2018-03-29T14:09:31,730 INFO [main] com.metamx.http.client.pool.ChannelResourceFactory - Generating: http://sandbox-hdp.hortonworks.com:8090 2018-03-29T14:09:31,731 INFO [main] com.metamx.http.client.pool.ChannelResourceFactory - Generating: http://sandbox-hdp.hortonworks.com:8090 2018-03-29T14:09:31,731 INFO [main] com.metamx.http.client.pool.ChannelResourceFactory - Generating: http://sandbox-hdp.hortonworks.com:8090 2018-03-29T14:09:31,732 INFO [main] com.metamx.http.client.pool.ChannelResourceFactory - Generating: http://sandbox-hdp.hortonworks.com:8090 2018-03-29T14:09:31,732 INFO [main] com.metamx.http.client.pool.ChannelResourceFactory - Generating: http://sandbox-hdp.hortonworks.com:8090 2018-03-29T14:09:31,732 INFO [main] com.metamx.http.client.pool.ChannelResourceFactory - Generating: http://sandbox-hdp.hortonworks.com:8090 2018-03-29T14:09:31,732 INFO [main] com.metamx.http.client.pool.ChannelResourceFactory - Generating: http://sandbox-hdp.hortonworks.com:8090 2018-03-29T14:09:31,733 INFO [main] com.metamx.http.client.pool.ChannelResourceFactory - Generating: http://sandbox-hdp.hortonworks.com:8090 2018-03-29T14:09:31,733 INFO [main] com.metamx.http.client.pool.ChannelResourceFactory - Generating: http://sandbox-hdp.hortonworks.com:8090 2018-03-29T14:09:31,733 INFO [main] com.metamx.http.client.pool.ChannelResourceFactory - Generating: http://sandbox-hdp.hortonworks.com:8090 2018-03-29T14:09:31,734 INFO [main] com.metamx.http.client.pool.ChannelResourceFactory - Generating: http://sandbox-hdp.hortonworks.com:8090 2018-03-29T14:09:31,734 INFO [main] com.metamx.http.client.pool.ChannelResourceFactory - Generating: http://sandbox-hdp.hortonworks.com:8090 2018-03-29T14:09:31,734 INFO [main] com.metamx.http.client.pool.ChannelResourceFactory - Generating: http://sandbox-hdp.hortonworks.com:8090 2018-03-29T14:09:31,735 INFO [main] com.metamx.http.client.pool.ChannelResourceFactory - Generating: http://sandbox-hdp.hortonworks.com:8090 2018-03-29T14:09:31,735 INFO [main] com.metamx.http.client.pool.ChannelResourceFactory - Generating: http://sandbox-hdp.hortonworks.com:8090 2018-03-29T14:09:31,735 INFO [main] com.metamx.http.client.pool.ChannelResourceFactory - Generating: http://sandbox-hdp.hortonworks.com:8090 2018-03-29T14:09:31,821 INFO [task-runner-0-priority-0] io.druid.indexing.overlord.ThreadPoolTaskRunner - Running task: index_usgs_2018-03-29T14:09:27.159Z 2018-03-29T14:09:31,824 INFO [task-runner-0-priority-0] io.druid.indexing.overlord.TaskRunnerUtils - Task [index_usgs_2018-03-29T14:09:27.159Z] location changed to [TaskLocation{host='sandbox-hdp.hortonworks.com', port=8100}]. 2018-03-29T14:09:31,824 INFO [task-runner-0-priority-0] io.druid.indexing.overlord.TaskRunnerUtils - Task [index_usgs_2018-03-29T14:09:27.159Z] status changed to [RUNNING]. 2018-03-29T14:09:31,829 INFO [task-runner-0-priority-0] io.druid.indexing.common.task.IndexTask - Determining intervals and shardSpecs 2018-03-29T14:09:31,831 INFO [main] org.eclipse.jetty.server.Server - jetty-9.3.19.v20170502 2018-03-29T14:09:31,880 ERROR [task-runner-0-priority-0] io.druid.indexing.overlord.ThreadPoolTaskRunner - Exception while running task[IndexTask{id=index_usgs_2018-03-29T14:09:27.159Z, type=index, dataSource=usgs}] java.lang.IllegalArgumentException: Instantiation of [simple type, class io.druid.data.input.impl.DelimitedParseSpec] value failed: If columns field is not set, the first row of your data must have your header and hasHeaderRow must be set to true. at com.fasterxml.jackson.databind.ObjectMapper._convert(ObjectMapper.java:2774) ~[jackson-databind-2.4.6.jar:2.4.6] at com.fasterxml.jackson.databind.ObjectMapper.convertValue(ObjectMapper.java:2700) ~[jackson-databind-2.4.6.jar:2.4.6] at io.druid.segment.indexing.DataSchema.getParser(DataSchema.java:111) ~[druid-server-0.10.1.2.6.4.0-91.jar:0.10.1.2.6.4.0-91] at io.druid.indexing.common.task.IndexTask.determineShardSpecs(IndexTask.java:269) ~[druid-indexing-service-0.10.1.2.6.4.0-91.jar:0.10.1.2.6.4.0-91] at io.druid.indexing.common.task.IndexTask.run(IndexTask.java:188) ~[druid-indexing-service-0.10.1.2.6.4.0-91.jar:0.10.1.2.6.4.0-91] at io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:436) [druid-indexing-service-0.10.1.2.6.4.0-91.jar:0.10.1.2.6.4.0-91] at io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:408) [druid-indexing-service-0.10.1.2.6.4.0-91.jar:0.10.1.2.6.4.0-91] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_161] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_161] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_161] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_161] Caused by: com.fasterxml.jackson.databind.JsonMappingException: Instantiation of [simple type, class io.druid.data.input.impl.DelimitedParseSpec] value failed: If columns field is not set, the first row of your data must have your header and hasHeaderRow must be set to true. at com.fasterxml.jackson.databind.deser.std.StdValueInstantiator.wrapException(StdValueInstantiator.java:405) ~[jackson-databind-2.4.6.jar:2.4.6] at com.fasterxml.jackson.databind.deser.std.StdValueInstantiator.createFromObjectWith(StdValueInstantiator.java:234) ~[jackson-databind-2.4.6.jar:2.4.6] at com.fasterxml.jackson.databind.deser.impl.PropertyBasedCreator.build(PropertyBasedCreator.java:167) ~[jackson-databind-2.4.6.jar:2.4.6] at com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeUsingPropertyBased(BeanDeserializer.java:398) ~[jackson-databind-2.4.6.jar:2.4.6] at com.fasterxml.jackson.databind.deser.BeanDeserializerBase.deserializeFromObjectUsingNonDefault(BeanDeserializerBase.java:1064) ~[jackson-databind-2.4.6.jar:2.4.6] at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserializeFromObject(BeanDeserializer.java:264) ~[jackson-databind-2.4.6.jar:2.4.6] at com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeOther(BeanDeserializer.java:156) ~[jackson-databind-2.4.6.jar:2.4.6] at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:126) ~[jackson-databind-2.4.6.jar:2.4.6] at com.fasterxml.jackson.databind.jsontype.impl.AsPropertyTypeDeserializer._deserializeTypedForId(AsPropertyTypeDeserializer.java:113) ~[jackson-databind-2.4.6.jar:2.4.6] at com.fasterxml.jackson.databind.jsontype.impl.AsPropertyTypeDeserializer.deserializeTypedFromObject(AsPropertyTypeDeserializer.java:84) ~[jackson-databind-2.4.6.jar:2.4.6] at com.fasterxml.jackson.databind.deser.AbstractDeserializer.deserializeWithType(AbstractDeserializer.java:132) ~[jackson-databind-2.4.6.jar:2.4.6] at com.fasterxml.jackson.databind.deser.SettableBeanProperty.deserialize(SettableBeanProperty.java:536) ~[jackson-databind-2.4.6.jar:2.4.6] at com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeUsingPropertyBased(BeanDeserializer.java:344) ~[jackson-databind-2.4.6.jar:2.4.6] at com.fasterxml.jackson.databind.deser.BeanDeserializerBase.deserializeFromObjectUsingNonDefault(BeanDeserializerBase.java:1064) ~[jackson-databind-2.4.6.jar:2.4.6] at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserializeFromObject(BeanDeserializer.java:264) ~[jackson-databind-2.4.6.jar:2.4.6] at com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeOther(BeanDeserializer.java:156) ~[jackson-databind-2.4.6.jar:2.4.6] at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:126) ~[jackson-databind-2.4.6.jar:2.4.6] at com.fasterxml.jackson.databind.jsontype.impl.AsPropertyTypeDeserializer._deserializeTypedForId(AsPropertyTypeDeserializer.java:113) ~[jackson-databind-2.4.6.jar:2.4.6] at com.fasterxml.jackson.databind.jsontype.impl.AsPropertyTypeDeserializer.deserializeTypedFromObject(AsPropertyTypeDeserializer.java:84) ~[jackson-databind-2.4.6.jar:2.4.6] at com.fasterxml.jackson.databind.deser.AbstractDeserializer.deserializeWithType(AbstractDeserializer.java:132) ~[jackson-databind-2.4.6.jar:2.4.6] at com.fasterxml.jackson.databind.deser.impl.TypeWrappedDeserializer.deserialize(TypeWrappedDeserializer.java:41) ~[jackson-databind-2.4.6.jar:2.4.6] at com.fasterxml.jackson.databind.ObjectMapper._convert(ObjectMapper.java:2769) ~[jackson-databind-2.4.6.jar:2.4.6] ... 10 more Caused by: java.lang.IllegalArgumentException: If columns field is not set, the first row of your data must have your header and hasHeaderRow must be set to true. at com.google.common.base.Preconditions.checkArgument(Preconditions.java:125) ~[guava-16.0.1.jar:?] at io.druid.data.input.impl.DelimitedParseSpec.<init>(DelimitedParseSpec.java:65) ~[druid-api-0.10.1.2.6.4.0-91.jar:0.10.1.2.6.4.0-91] at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:1.8.0_161] at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:1.8.0_161] at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:1.8.0_161] at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_161] at com.fasterxml.jackson.databind.introspect.AnnotatedConstructor.call(AnnotatedConstructor.java:125) ~[jackson-databind-2.4.6.jar:2.4.6] at com.fasterxml.jackson.databind.deser.std.StdValueInstantiator.createFromObjectWith(StdValueInstantiator.java:230) ~[jackson-databind-2.4.6.jar:2.4.6] at com.fasterxml.jackson.databind.deser.impl.PropertyBasedCreator.build(PropertyBasedCreator.java:167) ~[jackson-databind-2.4.6.jar:2.4.6] at com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeUsingPropertyBased(BeanDeserializer.java:398) ~[jackson-databind-2.4.6.jar:2.4.6] at com.fasterxml.jackson.databind.deser.BeanDeserializerBase.deserializeFromObjectUsingNonDefault(BeanDeserializerBase.java:1064) ~[jackson-databind-2.4.6.jar:2.4.6] at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserializeFromObject(BeanDeserializer.java:264) ~[jackson-databind-2.4.6.jar:2.4.6] at com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeOther(BeanDeserializer.java:156) ~[jackson-databind-2.4.6.jar:2.4.6] at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:126) ~[jackson-databind-2.4.6.jar:2.4.6] at com.fasterxml.jackson.databind.jsontype.impl.AsPropertyTypeDeserializer._deserializeTypedForId(AsPropertyTypeDeserializer.java:113) ~[jackson-databind-2.4.6.jar:2.4.6] at com.fasterxml.jackson.databind.jsontype.impl.AsPropertyTypeDeserializer.deserializeTypedFromObject(AsPropertyTypeDeserializer.java:84) ~[jackson-databind-2.4.6.jar:2.4.6] at com.fasterxml.jackson.databind.deser.AbstractDeserializer.deserializeWithType(AbstractDeserializer.java:132) ~[jackson-databind-2.4.6.jar:2.4.6] at com.fasterxml.jackson.databind.deser.SettableBeanProperty.deserialize(SettableBeanProperty.java:536) ~[jackson-databind-2.4.6.jar:2.4.6] at com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeUsingPropertyBased(BeanDeserializer.java:344) ~[jackson-databind-2.4.6.jar:2.4.6] at com.fasterxml.jackson.databind.deser.BeanDeserializerBase.deserializeFromObjectUsingNonDefault(BeanDeserializerBase.java:1064) ~[jackson-databind-2.4.6.jar:2.4.6] at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserializeFromObject(BeanDeserializer.java:264) ~[jackson-databind-2.4.6.jar:2.4.6] at com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeOther(BeanDeserializer.java:156) ~[jackson-databind-2.4.6.jar:2.4.6] at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:126) ~[jackson-databind-2.4.6.jar:2.4.6] at com.fasterxml.jackson.databind.jsontype.impl.AsPropertyTypeDeserializer._deserializeTypedForId(AsPropertyTypeDeserializer.java:113) ~[jackson-databind-2.4.6.jar:2.4.6] at com.fasterxml.jackson.databind.jsontype.impl.AsPropertyTypeDeserializer.deserializeTypedFromObject(AsPropertyTypeDeserializer.java:84) ~[jackson-databind-2.4.6.jar:2.4.6] at com.fasterxml.jackson.databind.deser.AbstractDeserializer.deserializeWithType(AbstractDeserializer.java:132) ~[jackson-databind-2.4.6.jar:2.4.6] at com.fasterxml.jackson.databind.deser.impl.TypeWrappedDeserializer.deserialize(TypeWrappedDeserializer.java:41) ~[jackson-databind-2.4.6.jar:2.4.6] at com.fasterxml.jackson.databind.ObjectMapper._convert(ObjectMapper.java:2769) ~[jackson-databind-2.4.6.jar:2.4.6] ... 10 more 2018-03-29T14:09:31,887 INFO [task-runner-0-priority-0] io.druid.indexing.overlord.TaskRunnerUtils - Task [index_usgs_2018-03-29T14:09:27.159Z] status changed to [FAILED]. 2018-03-29T14:09:31,898 INFO [task-runner-0-priority-0] io.druid.indexing.worker.executor.ExecutorLifecycle - Task completed with status: { "id" : "index_usgs_2018-03-29T14:09:27.159Z", "status" : "FAILED", "duration" : 66 } 2018-03-29T14:09:32,015 INFO [main] com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory - Registering com.fasterxml.jackson.jaxrs.json.JacksonJsonProvider as a provider class 2018-03-29T14:09:32,015 INFO [main] com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory - Registering com.fasterxml.jackson.jaxrs.smile.JacksonSmileProvider as a provider class 2018-03-29T14:09:32,016 INFO [main] com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory - Registering io.druid.server.initialization.jetty.CustomExceptionMapper as a provider class 2018-03-29T14:09:32,016 INFO [main] com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory - Registering io.druid.server.StatusResource as a root resource class 2018-03-29T14:09:32,019 INFO [main] com.sun.jersey.server.impl.application.WebApplicationImpl - Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:43 PM' 2018-03-29T14:09:32,086 INFO [main] com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory - Binding io.druid.server.initialization.jetty.CustomExceptionMapper to GuiceManagedComponentProvider with the scope "Singleton" 2018-03-29T14:09:32,089 INFO [main] com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory - Binding com.fasterxml.jackson.jaxrs.json.JacksonJsonProvider to GuiceManagedComponentProvider with the scope "Singleton" 2018-03-29T14:09:32,096 INFO [main] com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory - Binding com.fasterxml.jackson.jaxrs.smile.JacksonSmileProvider to GuiceManagedComponentProvider with the scope "Singleton" 2018-03-29T14:09:32,368 INFO [main] com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory - Binding io.druid.server.http.security.StateResourceFilter to GuiceInstantiatedComponentProvider 2018-03-29T14:09:32,387 INFO [main] com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory - Binding io.druid.server.http.SegmentListerResource to GuiceInstantiatedComponentProvider 2018-03-29T14:09:32,398 INFO [main] com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory - Binding io.druid.server.QueryResource to GuiceInstantiatedComponentProvider 2018-03-29T14:09:32,400 INFO [main] com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory - Binding io.druid.segment.realtime.firehose.ChatHandlerResource to GuiceInstantiatedComponentProvider 2018-03-29T14:09:32,403 INFO [main] com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory - Binding io.druid.query.lookup.LookupListeningResource to GuiceInstantiatedComponentProvider 2018-03-29T14:09:32,405 INFO [main] com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory - Binding io.druid.query.lookup.LookupIntrospectionResource to GuiceInstantiatedComponentProvider 2018-03-29T14:09:32,406 INFO [main] com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory - Binding io.druid.server.StatusResource to GuiceManagedComponentProvider with the scope "Undefined" 2018-03-29T14:09:32,420 WARN [main] com.sun.jersey.spi.inject.Errors - The following warnings have been detected with resource and/or provider classes: WARNING: A HTTP GET method, public void io.druid.server.http.SegmentListerResource.getSegments(long,long,long,javax.servlet.http.HttpServletRequest) throws java.io.IOException, MUST return a non-void type. 2018-03-29T14:09:32,428 INFO [main] org.eclipse.jetty.server.handler.ContextHandler - Started o.e.j.s.ServletContextHandler@63de4fa{/,null,AVAILABLE} 2018-03-29T14:09:32,435 INFO [main] org.eclipse.jetty.server.AbstractConnector - Started ServerConnector@2fba0dac{HTTP/1.1,[http/1.1]}{0.0.0.0:8100} 2018-03-29T14:09:32,435 INFO [main] org.eclipse.jetty.server.Server - Started @5163ms 2018-03-29T14:09:32,436 INFO [main] io.druid.java.util.common.lifecycle.Lifecycle$AnnotationBasedHandler - Invoking start method[public void io.druid.server.listener.announcer.ListenerResourceAnnouncer.start()] on object[io.druid.query.lookup.LookupResourceListenerAnnouncer@7be38eba]. 2018-03-29T14:09:32,450 INFO [main] io.druid.server.listener.announcer.ListenerResourceAnnouncer - Announcing start time on [/druid/listeners/lookups/__default/sandbox-hdp.hortonworks.com:8100] 2018-03-29T14:09:32,455 INFO [main] io.druid.java.util.common.lifecycle.Lifecycle$AnnotationBasedHandler - Invoking stop method[public void io.druid.server.listener.announcer.ListenerResourceAnnouncer.stop()] on object[io.druid.query.lookup.LookupResourceListenerAnnouncer@7be38eba]. 2018-03-29T14:09:32,455 INFO [main] io.druid.curator.announcement.Announcer - unannouncing [/druid/listeners/lookups/__default/sandbox-hdp.hortonworks.com:8100] 2018-03-29T14:09:32,484 INFO [main] io.druid.server.listener.announcer.ListenerResourceAnnouncer - Unannouncing start time on [/druid/listeners/lookups/__default/sandbox-hdp.hortonworks.com:8100] 2018-03-29T14:09:32,488 INFO [main] org.eclipse.jetty.server.AbstractConnector - Stopped ServerConnector@2fba0dac{HTTP/1.1,[http/1.1]}{0.0.0.0:8100} 2018-03-29T14:09:32,489 INFO [main] org.eclipse.jetty.server.handler.ContextHandler - Stopped o.e.j.s.ServletContextHandler@63de4fa{/,null,UNAVAILABLE} 2018-03-29T14:09:32,496 INFO [main] io.druid.java.util.common.lifecycle.Lifecycle$AnnotationBasedHandler - Invoking stop method[public void io.druid.indexing.worker.executor.ExecutorLifecycle.stop() throws java.lang.Exception] on object[io.druid.indexing.worker.executor.ExecutorLifecycle@65eb3597]. 2018-03-29T14:09:32,497 INFO [main] io.druid.java.util.common.lifecycle.Lifecycle$AnnotationBasedHandler - Invoking stop method[public void io.druid.indexing.overlord.ThreadPoolTaskRunner.stop()] on object[io.druid.indexing.overlord.ThreadPoolTaskRunner@37b1149b]. 2018-03-29T14:09:32,497 INFO [main] io.druid.java.util.common.lifecycle.Lifecycle$AnnotationBasedHandler - Invoking stop method[public void io.druid.curator.discovery.ServerDiscoverySelector.stop() throws java.io.IOException] on object[io.druid.curator.discovery.ServerDiscoverySelector@646cd766]. 2018-03-29T14:09:32,501 INFO [main] io.druid.java.util.common.lifecycle.Lifecycle$AnnotationBasedHandler - Invoking stop method[public void io.druid.curator.announcement.Announcer.stop()] on object[io.druid.curator.announcement.Announcer@1f129467]. 2018-03-29T14:09:32,501 INFO [main] io.druid.java.util.common.lifecycle.Lifecycle$AnnotationBasedHandler - Invoking stop method[public void io.druid.curator.discovery.ServerDiscoverySelector.stop() throws java.io.IOException] on object[io.druid.curator.discovery.ServerDiscoverySelector@13cc0b90]. 2018-03-29T14:09:32,501 INFO [main] io.druid.curator.CuratorModule - Stopping Curator 2018-03-29T14:09:32,502 INFO [Curator-Framework-0] org.apache.curator.framework.imps.CuratorFrameworkImpl - backgroundOperationsLoop exiting 2018-03-29T14:09:32,506 INFO [main] org.apache.zookeeper.ZooKeeper - Session: 0x16271e552860012 closed 2018-03-29T14:09:32,506 INFO [main-EventThread] org.apache.zookeeper.ClientCnxn - EventThread shut down 2018-03-29T14:09:32,506 INFO [main] com.metamx.common.lifecycle.Lifecycle$AnnotationBasedHandler - Invoking stop method[public void com.metamx.http.client.NettyHttpClient.stop()] on object[com.metamx.http.client.NettyHttpClient@36ad229b]. 2018-03-29T14:09:32,527 INFO [main] io.druid.java.util.common.lifecycle.Lifecycle$AnnotationBasedHandler - Invoking stop method[public void io.druid.storage.hdfs.HdfsStorageAuthentication.stop()] on object[io.druid.storage.hdfs.HdfsStorageAuthentication@2e56b4d]. 2018-03-29T14:09:32,527 INFO [main] io.druid.java.util.common.lifecycle.Lifecycle$AnnotationBasedHandler - Invoking stop method[public void com.metamx.metrics.MonitorScheduler.stop()] on object[com.metamx.metrics.MonitorScheduler@be164d8]. 2018-03-29T14:09:32,528 INFO [main] io.druid.java.util.common.lifecycle.Lifecycle$AnnotationBasedHandler - Invoking stop method[public void io.druid.query.lookup.LookupReferencesManager.stop()] on object[io.druid.query.lookup.LookupReferencesManager@89caf47]. 2018-03-29T14:09:32,528 INFO [main] io.druid.query.lookup.LookupReferencesManager - LookupReferencesManager is stopping. 2018-03-29T14:09:32,528 INFO [LookupReferencesManager-MainThread] io.druid.query.lookup.LookupReferencesManager - Lookup Management loop exited, Lookup notices are not handled anymore. 2018-03-29T14:09:32,529 INFO [main] io.druid.query.lookup.LookupReferencesManager - LookupReferencesManager is stopped. 2018-03-29T14:09:32,529 INFO [main] io.druid.java.util.common.lifecycle.Lifecycle$AnnotationBasedHandler - Invoking stop method[public void com.metamx.emitter.service.ServiceEmitter.close() throws java.io.IOException] on object[com.metamx.emitter.service.ServiceEmitter@5a090f62]. 2018-03-29T14:09:32,529 INFO [main] io.druid.java.util.common.lifecycle.Lifecycle$AnnotationBasedHandler - Invoking stop method[public void io.druid.initialization.Log4jShutterDownerModule$Log4jShutterDowner.stop()] on object[io.druid.initialization.Log4jShutterDownerModule$Log4jShutterDowner@3dbd7107]. 2018-03-29 14:09:32,567 pool-1-thread-1 ERROR Unable to register shutdown hook because JVM is shutting down. java.lang.IllegalStateException: Not started at io.druid.common.config.Log4jShutdown.addShutdownCallback(Log4jShutdown.java:46) at org.apache.logging.log4j.core.impl.Log4jContextFactory.addShutdownCallback(Log4jContextFactory.java:273) at org.apache.logging.log4j.core.LoggerContext.setUpShutdownHook(LoggerContext.java:256) at org.apache.logging.log4j.core.LoggerContext.start(LoggerContext.java:216) at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:145) at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:41) at org.apache.logging.log4j.LogManager.getContext(LogManager.java:182) at org.apache.logging.log4j.spi.AbstractLoggerAdapter.getContext(AbstractLoggerAdapter.java:103) at org.apache.logging.slf4j.Log4jLoggerFactory.getContext(Log4jLoggerFactory.java:43) at org.apache.logging.log4j.spi.AbstractLoggerAdapter.getLogger(AbstractLoggerAdapter.java:42) at org.apache.logging.slf4j.Log4jLoggerFactory.getLogger(Log4jLoggerFactory.java:29) at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:253) at org.apache.commons.logging.impl.SLF4JLogFactory.getInstance(SLF4JLogFactory.java:155) at org.apache.commons.logging.impl.SLF4JLogFactory.getInstance(SLF4JLogFactory.java:132) at org.apache.commons.logging.LogFactory.getLog(LogFactory.java:273) at org.apache.hadoop.hdfs.LeaseRenewer.<clinit>(LeaseRenewer.java:72) at org.apache.hadoop.hdfs.DFSClient.getLeaseRenewer(DFSClient.java:830) at org.apache.hadoop.hdfs.DFSClient.close(DFSClient.java:968) at org.apache.hadoop.hdfs.DistributedFileSystem.close(DistributedFileSystem.java:1214) at org.apache.hadoop.fs.FileSystem$Cache.closeAll(FileSystem.java:2886) at org.apache.hadoop.fs.FileSystem$Cache$ClientFinalizer.run(FileSystem.java:2903) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Any help would be very much appreciated.

jmedel · ‎03-29-2018

I installed Druid with the Derby metadata storage via Ambari, then I follow the steps on the Druid Quickstart page from Druid's documentation to test Druid on HDP 2.6.4. I am using the default Hadoop-Based Ingestion Spec: { "type" : "index_hadoop", "spec" : { "ioConfig" : { "type" : "hadoop", "inputSpec" : { "type" : "static", "paths" : "quickstart/wikiticker-2015-09-12-sampled.json" } }, "dataSchema" : { "dataSource" : "wikiticker", "granularitySpec" : { "type" : "uniform", "segmentGranularity" : "day", "queryGranularity" : "none", "intervals" : ["2015-09-12/2015-09-13"] }, "parser" : { "type" : "hadoopyString", "parseSpec" : { "format" : "json", "dimensionsSpec" : { "dimensions" : [ "channel", "cityName", "comment", "countryIsoCode", "countryName", "isAnonymous", "isMinor", "isNew", "isRobot", "isUnpatrolled", "metroCode", "namespace", "page", "regionIsoCode", "regionName", "user" ] }, "timestampSpec" : { "format" : "auto", "column" : "time" } } }, "metricsSpec" : [ { "name" : "count", "type" : "count" }, { "name" : "added", "type" : "longSum", "fieldName" : "added" }, { "name" : "deleted", "type" : "longSum", "fieldName" : "deleted" }, { "name" : "delta", "type" : "longSum", "fieldName" : "delta" }, { "name" : "user_unique", "type" : "hyperUnique", "fieldName" : "user" } ] }, "tuningConfig" : { "type" : "hadoop", "partitionsSpec" : { "type" : "hashed", "targetPartitionSize" : 5000000 }, "jobProperties" : {} } } } and am unable to run the Wiki Demo because of ERROR [task-runner-0-priority-0] io.druid.indexing.overlord.ThreadPoolTaskRunner - Exception while running task[HadoopIndexTask{id=index_hadoop_wikiticker_2018-03-29T05:17:04.464Z, type=index_hadoop, dataSource=wikiticker}] java.lang.RuntimeException: java.lang.reflect.InvocationTargetException RuntimeException: java.lang.reflect.InvocationTargetException while running Hadoop-based Batch Ingestion Task "wikiticker". Input Path Doesn't exist: Caused by: java.lang.RuntimeException: org.apache.hadoop.mapreduce.lib.input.InvalidInputException: Input path does not exist: hdfs://sandbox-hdp.hortonworks.com:8020/user/druid/quickstart/wikiticker-2015-09-12-sampled.json.gz Here is the full stack trace of the error: 2018-03-29T14:55:15,326 INFO [main] io.druid.guice.PropertiesModule - Loading properties from common.runtime.properties 2018-03-29T14:55:15,327 INFO [main] io.druid.guice.PropertiesModule - Loading properties from runtime.properties 2018-03-29T14:55:15,356 INFO [main] org.hibernate.validator.internal.util.Version - HV000001: Hibernate Validator 5.1.3.Final 2018-03-29T14:55:15,820 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.segment.NullValueHandlingConfig] from props[druid.null.handling.] as [io.druid.segment.NullValueHandlingConfig@6995bf68] 2018-03-29T14:55:15,878 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.guice.ExtensionsConfig] from props[druid.extensions.] as [ExtensionsConfig{searchCurrentClassloader=true, directory='/usr/hdp/current/druid-middlemanager/extensions', hadoopDependenciesDir='/usr/hdp/current/druid-middlemanager/hadoop-dependencies', hadoopContainerDruidClasspath='null', addExtensionsToHadoopContainer=false, loadList=[druid-datasketches, druid-hdfs-storage, druid-kafka-indexing-service]}] 2018-03-29T14:55:15,932 INFO [main] io.druid.initialization.Initialization - Loading extension [druid-datasketches] for class [interface io.druid.cli.CliCommandCreator] 2018-03-29T14:55:15,942 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-datasketches/druid-datasketches-0.10.1.2.6.4.0-91.jar] 2018-03-29T14:55:15,942 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-datasketches/sketches-core-0.8.4.jar] 2018-03-29T14:55:15,943 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-datasketches/memory-0.8.4.jar] 2018-03-29T14:55:15,945 INFO [main] io.druid.initialization.Initialization - Loading extension [druid-hdfs-storage] for class [interface io.druid.cli.CliCommandCreator] 2018-03-29T14:55:15,947 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/hadoop-common-2.7.3.2.6.4.0-91.jar] 2018-03-29T14:55:15,948 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/okio-1.4.0.jar] 2018-03-29T14:55:15,948 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/curator-client-2.11.0.jar] 2018-03-29T14:55:15,948 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/commons-beanutils-1.7.0.jar] 2018-03-29T14:55:15,948 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/jsp-api-2.1.jar] 2018-03-29T14:55:15,948 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/snappy-java-1.0.4.1.jar] 2018-03-29T14:55:15,949 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/stax-api-1.0-2.jar] 2018-03-29T14:55:15,949 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/servlet-api-2.5.jar] 2018-03-29T14:55:15,949 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/hadoop-client-2.7.3.2.6.4.0-91.jar] 2018-03-29T14:55:15,949 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/hadoop-mapreduce-client-jobclient-2.7.3.2.6.4.0-91.jar] 2018-03-29T14:55:15,949 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/okhttp-2.4.0.jar] 2018-03-29T14:55:15,949 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/commons-logging-1.1.1.jar] 2018-03-29T14:55:15,949 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/api-util-1.0.0-M20.jar] 2018-03-29T14:55:15,950 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/apacheds-i18n-2.0.0-M15.jar] 2018-03-29T14:55:15,950 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/curator-recipes-2.11.0.jar] 2018-03-29T14:55:15,950 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/netty-all-4.1.11.Final.jar] 2018-03-29T14:55:15,950 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/aws-java-sdk-kms-1.10.77.jar] 2018-03-29T14:55:15,950 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/druid-hdfs-storage-0.10.1.2.6.4.0-91.jar] 2018-03-29T14:55:15,950 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/nimbus-jose-jwt-3.9.jar] 2018-03-29T14:55:15,951 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/xmlenc-0.52.jar] 2018-03-29T14:55:15,951 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/htrace-core-3.1.0-incubating.jar] 2018-03-29T14:55:15,951 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/xz-1.0.jar] 2018-03-29T14:55:15,951 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/commons-beanutils-core-1.8.0.jar] 2018-03-29T14:55:15,951 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/gson-2.2.4.jar] 2018-03-29T14:55:15,951 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/jackson-core-2.4.6.jar] 2018-03-29T14:55:15,952 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/azure-storage-5.4.0.jar] 2018-03-29T14:55:15,952 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/curator-framework-2.11.0.jar] 2018-03-29T14:55:15,952 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/hadoop-mapreduce-client-core-2.7.3.2.6.4.0-91.jar] 2018-03-29T14:55:15,952 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/jackson-xc-1.9.13.jar] 2018-03-29T14:55:15,952 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/commons-configuration-1.6.jar] 2018-03-29T14:55:15,952 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/jackson-dataformat-cbor-2.5.3.jar] 2018-03-29T14:55:15,952 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/hadoop-yarn-api-2.7.3.2.6.4.0-91.jar] 2018-03-29T14:55:15,953 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/hadoop-yarn-client-2.7.3.2.6.4.0-91.jar] 2018-03-29T14:55:15,953 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/aws-java-sdk-core-1.10.77.jar] 2018-03-29T14:55:15,953 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/hadoop-hdfs-2.7.3.2.6.4.0-91.jar] 2018-03-29T14:55:15,953 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/hadoop-yarn-registry-2.7.3.2.6.4.0-91.jar] 2018-03-29T14:55:15,953 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/azure-keyvault-core-0.8.0.jar] 2018-03-29T14:55:15,953 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/hadoop-yarn-common-2.7.3.2.6.4.0-91.jar] 2018-03-29T14:55:15,954 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/api-asn1-api-1.0.0-M20.jar] 2018-03-29T14:55:15,954 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/hadoop-yarn-server-common-2.7.3.2.6.4.0-91.jar] 2018-03-29T14:55:15,954 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/commons-lang3-3.4.jar] 2018-03-29T14:55:15,954 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/hadoop-auth-2.7.3.2.6.4.0-91.jar] 2018-03-29T14:55:15,954 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/commons-net-3.1.jar] 2018-03-29T14:55:15,954 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/paranamer-2.3.jar] 2018-03-29T14:55:15,954 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/jersey-client-1.9.jar] 2018-03-29T14:55:15,954 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/hadoop-mapreduce-client-common-2.7.3.2.6.4.0-91.jar] 2018-03-29T14:55:15,955 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/avro-1.7.4.jar] 2018-03-29T14:55:15,955 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/jackson-databind-2.4.6.jar] 2018-03-29T14:55:15,955 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/jaxb-api-2.2.2.jar] 2018-03-29T14:55:15,955 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/jackson-annotations-2.4.6.jar] 2018-03-29T14:55:15,955 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/httpclient-4.5.1.jar] 2018-03-29T14:55:15,955 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/jackson-jaxrs-1.9.13.jar] 2018-03-29T14:55:15,956 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/xml-apis-1.3.04.jar] 2018-03-29T14:55:15,956 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/commons-compress-1.4.1.jar] 2018-03-29T14:55:15,956 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/xercesImpl-2.9.1.jar] 2018-03-29T14:55:15,956 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/joda-time-2.9.9.jar] 2018-03-29T14:55:15,956 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/jetty-sslengine-6.1.26.hwx.jar] 2018-03-29T14:55:15,956 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/commons-digester-1.8.jar] 2018-03-29T14:55:15,957 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/hadoop-aws-2.7.3.2.6.4.0-91.jar] 2018-03-29T14:55:15,957 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/json-smart-1.1.1.jar] 2018-03-29T14:55:15,957 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/apacheds-kerberos-codec-2.0.0-M15.jar] 2018-03-29T14:55:15,957 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/netty-3.6.2.Final.jar] 2018-03-29T14:55:15,957 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/aws-java-sdk-s3-1.10.77.jar] 2018-03-29T14:55:15,957 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/httpcore-4.4.3.jar] 2018-03-29T14:55:15,958 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/guava-16.0.1.jar] 2018-03-29T14:55:15,958 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/hadoop-mapreduce-client-shuffle-2.7.3.2.6.4.0-91.jar] 2018-03-29T14:55:15,958 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/commons-collections-3.2.2.jar] 2018-03-29T14:55:15,958 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/hadoop-mapreduce-client-app-2.7.3.2.6.4.0-91.jar] 2018-03-29T14:55:15,958 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/jcip-annotations-1.0.jar] 2018-03-29T14:55:15,958 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/commons-math3-3.6.1.jar] 2018-03-29T14:55:15,958 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/leveldbjni-all-1.8.jar] 2018-03-29T14:55:15,975 INFO [main] io.druid.initialization.Initialization - Loading extension [druid-kafka-indexing-service] for class [interface io.druid.cli.CliCommandCreator] 2018-03-29T14:55:15,976 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-kafka-indexing-service/snappy-java-1.1.2.6.jar] 2018-03-29T14:55:15,976 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-kafka-indexing-service/slf4j-api-1.6.4.jar] 2018-03-29T14:55:15,976 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-kafka-indexing-service/druid-kafka-indexing-service-0.10.1.2.6.4.0-91.jar] 2018-03-29T14:55:15,976 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-kafka-indexing-service/lz4-1.3.0.jar] 2018-03-29T14:55:15,976 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-kafka-indexing-service/kafka-clients-0.10.2.0.jar] 2018-03-29T14:55:16,059 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.guice.ModulesConfig] from props[druid.modules.] as [ModulesConfig{excludeList=[]}] 2018-03-29T14:55:16,204 INFO [main] io.druid.initialization.Initialization - Loading extension [druid-datasketches] for class [interface io.druid.initialization.DruidModule] 2018-03-29T14:55:16,208 INFO [main] io.druid.initialization.Initialization - Adding implementation [io.druid.query.aggregation.datasketches.theta.SketchModule] for class [interface io.druid.initialization.DruidModule] from local file system extension 2018-03-29T14:55:16,209 INFO [main] io.druid.initialization.Initialization - Adding implementation [io.druid.query.aggregation.datasketches.theta.oldapi.OldApiSketchModule] for class [interface io.druid.initialization.DruidModule] from local file system extension 2018-03-29T14:55:16,209 INFO [main] io.druid.initialization.Initialization - Loading extension [druid-hdfs-storage] for class [interface io.druid.initialization.DruidModule] 2018-03-29T14:55:16,211 INFO [main] io.druid.initialization.Initialization - Adding implementation [io.druid.storage.hdfs.HdfsStorageDruidModule] for class [interface io.druid.initialization.DruidModule] from local file system extension 2018-03-29T14:55:16,211 INFO [main] io.druid.initialization.Initialization - Loading extension [druid-kafka-indexing-service] for class [interface io.druid.initialization.DruidModule] 2018-03-29T14:55:16,212 INFO [main] io.druid.initialization.Initialization - Adding implementation [io.druid.indexing.kafka.KafkaIndexTaskModule] for class [interface io.druid.initialization.DruidModule] from local file system extension 2018-03-29T14:55:17,032 WARN [main] org.apache.hadoop.util.NativeCodeLoader - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2018-03-29T14:55:18,026 WARN [main] org.apache.hadoop.hdfs.shortcircuit.DomainSocketFactory - The short-circuit local reads feature cannot be used because libhadoop cannot be loaded. 2018-03-29T14:55:18,637 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.server.metrics.DruidMonitorSchedulerConfig] from props[druid.monitoring.] as [io.druid.server.metrics.DruidMonitorSchedulerConfig@24eeac69] 2018-03-29T14:55:18,647 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.server.metrics.MonitorsConfig] from props[druid.monitoring.] as [MonitorsConfig{monitors=[class com.metamx.metrics.JvmMonitor]}] 2018-03-29T14:55:18,670 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.server.DruidNode] from props[druid.] as [DruidNode{serviceName='druid/middlemanager', host='sandbox-hdp.hortonworks.com', port=8100}] 2018-03-29T14:55:18,781 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.query.lookup.LookupConfig] from props[druid.lookup.] as [LookupConfig{snapshotWorkingDir=''}] 2018-03-29T14:55:18,845 INFO [main] io.druid.guice.PropertiesModule - Loading properties from common.runtime.properties 2018-03-29T14:55:18,846 INFO [main] io.druid.guice.PropertiesModule - Loading properties from runtime.properties 2018-03-29T14:55:18,855 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.segment.NullValueHandlingConfig] from props[druid.null.handling.] as [io.druid.segment.NullValueHandlingConfig@10b4e7f8] 2018-03-29T14:55:18,861 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.guice.ExtensionsConfig] from props[druid.extensions.] as [ExtensionsConfig{searchCurrentClassloader=true, directory='/usr/hdp/current/druid-middlemanager/extensions', hadoopDependenciesDir='/usr/hdp/current/druid-middlemanager/hadoop-dependencies', hadoopContainerDruidClasspath='null', addExtensionsToHadoopContainer=false, loadList=[druid-datasketches, druid-hdfs-storage, druid-kafka-indexing-service]}] 2018-03-29T14:55:18,865 INFO [main] io.druid.server.metrics.MetricsModule - Adding monitor[com.metamx.metrics.JvmMonitor@4e224df5] 2018-03-29T14:55:18,866 INFO [main] io.druid.server.metrics.MetricsModule - Adding monitor[io.druid.query.ExecutorServiceMonitor@4dd90166] 2018-03-29T14:55:18,868 INFO [main] io.druid.server.metrics.MetricsModule - Adding monitor[io.druid.server.initialization.jetty.JettyServerModule$JettyMonitor@a146b11] 2018-03-29T14:55:18,875 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.server.log.StartupLoggingConfig] from props[druid.startup.logging.] as [io.druid.server.log.StartupLoggingConfig@20580d4e] 2018-03-29T14:55:18,875 INFO [main] io.druid.cli.CliPeon - Starting up with processors[4], memory[316,145,664], maxMemory[1,908,932,608]. 2018-03-29T14:55:18,878 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.storage.hdfs.HdfsKerberosConfig] from props[druid.hadoop.security.kerberos.] as [io.druid.storage.hdfs.HdfsKerberosConfig@0] 2018-03-29T14:55:18,885 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.indexing.common.config.TaskConfig] from props[druid.indexer.task.] as [io.druid.indexing.common.config.TaskConfig@76536c53] 2018-03-29T14:55:18,888 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.guice.http.DruidHttpClientConfig] from props[druid.global.http.] as [io.druid.guice.http.DruidHttpClientConfig@75fd65c] 2018-03-29T14:55:18,965 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.client.indexing.IndexingServiceSelectorConfig] from props[druid.selectors.indexing.] as [io.druid.client.indexing.IndexingServiceSelectorConfig@1f43cab7] 2018-03-29T14:55:18,968 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.curator.CuratorConfig] from props[druid.zk.service.] as [io.druid.curator.CuratorConfig@a6e4897] 2018-03-29T14:55:18,972 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.curator.ExhibitorConfig] from props[druid.exhibitor.service.] as [io.druid.curator.ExhibitorConfig@91da29b] 2018-03-29T14:55:18,977 WARN [main] org.apache.curator.retry.ExponentialBackoffRetry - maxRetries too large (30). Pinning to 29 2018-03-29T14:55:19,003 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.server.initialization.CuratorDiscoveryConfig] from props[druid.discovery.curator.] as [io.druid.server.initialization.CuratorDiscoveryConfig@4aa2877c] 2018-03-29T14:55:19,031 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.indexing.common.RetryPolicyConfig] from props[druid.peon.taskActionClient.retry.] as [io.druid.indexing.common.RetryPolicyConfig@545b5ed0] 2018-03-29T14:55:19,034 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.storage.hdfs.HdfsDataSegmentPusherConfig] from props[druid.storage.] as [io.druid.storage.hdfs.HdfsDataSegmentPusherConfig@abad89c] 2018-03-29T14:55:19,036 INFO [main] io.druid.storage.hdfs.HdfsDataSegmentPusher - Configured HDFS as deep storage 2018-03-29T14:55:19,040 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.segment.loading.LocalDataSegmentPusherConfig] from props[druid.storage.] as [io.druid.segment.loading.LocalDataSegmentPusherConfig@3c232051] 2018-03-29T14:55:19,046 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.client.DruidServerConfig] from props[druid.server.] as [io.druid.client.DruidServerConfig@4c3de38e] 2018-03-29T14:55:19,051 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.server.initialization.BatchDataSegmentAnnouncerConfig] from props[druid.announcer.] as [io.druid.server.initialization.BatchDataSegmentAnnouncerConfig@1983b48a] 2018-03-29T14:55:19,055 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.server.initialization.ZkPathsConfig] from props[druid.zk.paths.] as [io.druid.server.initialization.ZkPathsConfig@22e2266d] 2018-03-29T14:55:19,062 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[interface io.druid.server.coordination.DataSegmentAnnouncerProvider] from props[druid.announcer.] as [io.druid.server.coordination.BatchDataSegmentAnnouncerProvider@5d878b25] 2018-03-29T14:55:19,065 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.client.coordinator.CoordinatorSelectorConfig] from props[druid.selectors.coordinator.] as [io.druid.client.coordinator.CoordinatorSelectorConfig@4e245739] 2018-03-29T14:55:19,068 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.segment.realtime.plumber.CoordinatorBasedSegmentHandoffNotifierConfig] from props[druid.segment.handoff.] as [io.druid.segment.realtime.plumber.CoordinatorBasedSegmentHandoffNotifierConfig@691124ee] 2018-03-29T14:55:19,070 INFO [main] org.skife.config.ConfigurationObjectFactory - Assigning value [50000000] for [druid.processing.buffer.sizeBytes] on [io.druid.query.DruidProcessingConfig#intermediateComputeSizeBytes()] 2018-03-29T14:55:19,073 INFO [main] org.skife.config.ConfigurationObjectFactory - Using method itself for [druid.computation.buffer.poolCacheMaxCount, ${base_path}.buffer.poolCacheMaxCount] on [io.druid.query.DruidProcessingConfig#poolCacheMaxCount()] 2018-03-29T14:55:19,074 INFO [main] org.skife.config.ConfigurationObjectFactory - Using method itself for [${base_path}.tmpDir] on [io.druid.query.DruidProcessingConfig#getTmpDir()] 2018-03-29T14:55:19,074 INFO [main] org.skife.config.ConfigurationObjectFactory - Using method itself for [${base_path}.fifo] on [io.druid.query.DruidProcessingConfig#isFifo()] 2018-03-29T14:55:19,075 INFO [main] org.skife.config.ConfigurationObjectFactory - Assigning value [2] for [druid.processing.numThreads] on [io.druid.query.DruidProcessingConfig#getNumThreadsConfigured()] 2018-03-29T14:55:19,075 INFO [main] org.skife.config.ConfigurationObjectFactory - Using method itself for [${base_path}.numMergeBuffers] on [io.druid.query.DruidProcessingConfig#getNumMergeBuffersConfigured()] 2018-03-29T14:55:19,075 INFO [main] org.skife.config.ConfigurationObjectFactory - Using method itself for [${base_path}.columnCache.sizeBytes] on [io.druid.query.DruidProcessingConfig#columnCacheSizeBytes()] 2018-03-29T14:55:19,075 INFO [main] org.skife.config.ConfigurationObjectFactory - Assigning default value [processing-%s] for [${base_path}.formatString] on [io.druid.java.util.common.concurrent.ExecutorServiceConfig#getFormatString()] 2018-03-29T14:55:19,156 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[interface io.druid.client.cache.CacheProvider] from props[druid.cache.] as [io.druid.client.cache.LocalCacheProvider@5c059a68] 2018-03-29T14:55:19,167 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.client.cache.CacheConfig] from props[druid.realtime.cache.] as [io.druid.client.cache.CacheConfig@63a9c661] 2018-03-29T14:55:19,178 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.server.initialization.ServerConfig] from props[druid.server.http.] as [ServerConfig{numThreads=50, maxIdleTime=PT5M, defaultQueryTimeout=300000, maxScatterGatherBytes=9223372036854775807}] 2018-03-29T14:55:19,187 INFO [main] org.eclipse.jetty.util.log - Logging initialized @4760ms 2018-03-29T14:55:19,321 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.query.search.search.SearchQueryConfig] from props[druid.query.search.] as [io.druid.query.search.search.SearchQueryConfig@601eb4af] 2018-03-29T14:55:19,328 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.query.metadata.SegmentMetadataQueryConfig] from props[druid.query.segmentMetadata.] as [io.druid.query.metadata.SegmentMetadataQueryConfig@35cec305] 2018-03-29T14:55:19,335 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.query.groupby.GroupByQueryConfig] from props[druid.query.groupBy.] as [GroupByQueryConfig{defaultStrategy='v2', singleThreaded=false, maxIntermediateRows=50000, maxResults=500000, bufferGrouperMaxSize=2147483647, bufferGrouperMaxLoadFactor=0.0, bufferGrouperInitialBuckets=0, maxMergingDictionarySize=100000000, maxOnDiskStorage=0, forcePushDownLimit=false, forceHashAggregation=false}] 2018-03-29T14:55:19,381 INFO [main] io.druid.offheap.OffheapBufferGenerator - Allocating new intermediate processing buffer[0] of size[50,000,000] 2018-03-29T14:55:19,421 INFO [main] io.druid.offheap.OffheapBufferGenerator - Allocating new intermediate processing buffer[1] of size[50,000,000] 2018-03-29T14:55:19,462 INFO [main] io.druid.offheap.OffheapBufferGenerator - Allocating new result merging buffer[0] of size[50,000,000] 2018-03-29T14:55:19,503 INFO [main] io.druid.offheap.OffheapBufferGenerator - Allocating new result merging buffer[1] of size[50,000,000] 2018-03-29T14:55:19,550 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.query.scan.ScanQueryConfig] from props[druid.query.scan.] as [ScanQueryConfig{legacy=false}] 2018-03-29T14:55:19,552 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.query.select.SelectQueryConfig] from props[druid.query.select.] as [io.druid.query.select.SelectQueryConfig@380ce5a3] 2018-03-29T14:55:19,554 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.query.topn.TopNQueryConfig] from props[druid.query.topN.] as [io.druid.query.topn.TopNQueryConfig@7ca492d9] 2018-03-29T14:55:19,556 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[interface io.druid.server.log.RequestLoggerProvider] from props[druid.request.logging.] as [io.druid.server.log.NoopRequestLoggerProvider@7f2c57fe] 2018-03-29T14:55:19,559 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.server.security.AuthConfig] from props[druid.auth.] as [AuthConfig{enabled=false}] 2018-03-29T14:55:19,572 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.query.lookup.LookupListeningAnnouncerConfig] from props[druid.lookup.] as [ListeningAnnouncerConfig{listenersPath='/druid/listeners'}] 2018-03-29T14:55:19,574 INFO [main] io.druid.java.util.common.lifecycle.Lifecycle$AnnotationBasedHandler - Invoking start method[public void io.druid.initialization.Log4jShutterDownerModule$Log4jShutterDowner.start()] on object[io.druid.initialization.Log4jShutterDownerModule$Log4jShutterDowner@4f081b5d]. 2018-03-29T14:55:19,575 INFO [main] io.druid.java.util.common.lifecycle.Lifecycle$AnnotationBasedHandler - Invoking start method[public void com.metamx.emitter.service.ServiceEmitter.start()] on object[com.metamx.emitter.service.ServiceEmitter@5b2b8d86]. 2018-03-29T14:55:19,576 INFO [main] io.druid.java.util.common.lifecycle.Lifecycle$AnnotationBasedHandler - Invoking start method[public void io.druid.query.lookup.LookupReferencesManager.start()] on object[io.druid.query.lookup.LookupReferencesManager@19827608]. 2018-03-29T14:55:19,577 INFO [main] io.druid.query.lookup.LookupReferencesManager - LookupReferencesManager is starting. 2018-03-29T14:55:19,577 INFO [main] io.druid.query.lookup.LookupReferencesManager - LookupReferencesManager is started. 2018-03-29T14:55:19,578 INFO [main] io.druid.java.util.common.lifecycle.Lifecycle$AnnotationBasedHandler - Invoking start method[public void com.metamx.metrics.MonitorScheduler.start()] on object[com.metamx.metrics.MonitorScheduler@450f0235]. 2018-03-29T14:55:19,594 INFO [main] io.druid.java.util.common.lifecycle.Lifecycle$AnnotationBasedHandler - Invoking start method[public void io.druid.storage.hdfs.HdfsStorageAuthentication.authenticate()] on object[io.druid.storage.hdfs.HdfsStorageAuthentication@7aea704c]. 2018-03-29T14:55:19,595 INFO [main] com.metamx.common.lifecycle.Lifecycle$AnnotationBasedHandler - Invoking start method[public void com.metamx.http.client.NettyHttpClient.start()] on object[com.metamx.http.client.NettyHttpClient@339f3a55]. 2018-03-29T14:55:19,595 INFO [main] io.druid.curator.CuratorModule - Starting Curator 2018-03-29T14:55:19,595 INFO [main] org.apache.curator.framework.imps.CuratorFrameworkImpl - Starting 2018-03-29T14:55:19,617 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:zookeeper.version=3.4.6-91--1, built on 01/04/2018 09:27 GMT 2018-03-29T14:55:19,617 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:host.name=sandbox-hdp.hortonworks.com 2018-03-29T14:55:19,617 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:java.version=1.8.0_161 2018-03-29T14:55:19,617 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:java.vendor=Oracle Corporation 2018-03-29T14:55:19,617 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:java.home=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.161-3.b14.el6_9.x86_64/jre 2018-03-29T14:55:19,617 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:java.class.path=/usr/hdp/current/druid-middlemanager/conf/_common:/usr/hdp/current/druid-middlemanager/conf/middleManager:/usr/hdp/current/druid-middlemanager/lib/antlr4-runtime-4.5.1.jar:/usr/hdp/current/druid-middlemanager/lib/druid-sql-0.10.1.2.6.4.0-91.jar:/usr/hdp/current/druid-middlemanager/lib/plexus-interpolation-1.19.jar:/usr/hdp/current/druid-middlemanager/lib/guice-servlet-4.1.0.jar:/usr/hdp/current/druid-middlemanager/lib/jackson-jaxrs-smile-provider-2.4.6.jar:/usr/hdp/current/druid-middlemanager/lib/curator-client-2.11.0.jar:/usr/hdp/current/druid-middlemanager/lib/tesla-aether-0.0.5.jar:/usr/hdp/current/druid-middlemanager/lib/wagon-provider-api-2.4.jar:/usr/hdp/current/druid-middlemanager/lib/extendedset-0.10.1.2.6.4.0-91.jar:/usr/hdp/current/druid-middlemanager/lib/jackson-dataformat-smile-2.4.6.jar:/usr/hdp/current/druid-middlemanager/lib/maven-model-3.1.1.jar:/usr/hdp/current/druid-middlemanager/lib/json-path-2.1.0.jar:/usr/hdp/current/druid-middlemanager/lib/jetty-util-9.3.19.v20170502.jar:/usr/hdp/current/druid-middlemanager/lib/commons-compiler-2.7.6.jar:/usr/hdp/current/druid-middlemanager/lib/log4j-api-2.5.jar:/usr/hdp/current/druid-middlemanager/lib/jcl-over-slf4j-1.7.12.jar:/usr/hdp/current/druid-middlemanager/lib/druid-server-0.10.1.2.6.4.0-91.jar:/usr/hdp/current/druid-middlemanager/lib/jetty-proxy-9.3.19.v20170502.jar:/usr/hdp/current/druid-middlemanager/lib/commons-pool-1.6.jar:/usr/hdp/current/druid-middlemanager/lib/jsr311-api-1.1.1.jar:/usr/hdp/current/druid-middlemanager/lib/jsr305-2.0.1.jar:/usr/hdp/current/druid-middlemanager/lib/druid-indexing-service-0.10.1.2.6.4.0-91.jar:/usr/hdp/current/druid-middlemanager/lib/commons-logging-1.1.1.jar:/usr/hdp/current/druid-middlemanager/lib/okhttp-1.0.2.jar:/usr/hdp/current/druid-middlemanager/lib/RoaringBitmap-0.5.18.jar:/usr/hdp/current/druid-middlemanager/lib/http-client-1.1.0.jar:/usr/hdp/current/druid-middlemanager/lib/curator-recipes-2.11.0.jar:/usr/hdp/current/druid-middlemanager/lib/aopalliance-1.0.jar:/usr/hdp/current/druid-middlemanager/lib/netty-all-4.1.11.Final.jar:/usr/hdp/current/druid-middlemanager/lib/derbyclient-10.11.1.1.jar:/usr/hdp/current/druid-middlemanager/lib/bytebuffer-collections-0.10.1.2.6.4.0-91.jar:/usr/hdp/current/druid-middlemanager/lib/protobuf-java-3.1.0.jar:/usr/hdp/current/druid-middlemanager/lib/aether-spi-0.9.0.M2.jar:/usr/hdp/current/druid-middlemanager/lib/derbynet-10.11.1.1.jar:/usr/hdp/current/druid-middlemanager/lib/compress-lzf-1.0.3.jar:/usr/hdp/current/druid-middlemanager/lib/disruptor-3.3.6.jar:/usr/hdp/current/druid-middlemanager/lib/jetty-http-9.3.19.v20170502.jar:/usr/hdp/current/druid-middlemanager/lib/maven-settings-3.1.1.jar:/usr/hdp/current/druid-middlemanager/lib/bcprov-jdk15on-1.52.jar:/usr/hdp/current/druid-middlemanager/lib/classmate-1.0.0.jar:/usr/hdp/current/druid-middlemanager/lib/opencsv-2.3.jar:/usr/hdp/current/druid-middlemanager/lib/druid-aws-common-0.10.1.2.6.4.0-91.jar:/usr/hdp/current/druid-middlemanager/lib/aws-java-sdk-ec2-1.10.77.jar:/usr/hdp/current/druid-middlemanager/lib/druid-services-0.10.1.2.6.4.0-91.jar:/usr/hdp/current/druid-middlemanager/lib/javax.inject-1.jar:/usr/hdp/current/druid-middlemanager/lib/jackson-datatype-joda-2.4.6.jar:/usr/hdp/current/druid-middlemanager/lib/commons-dbcp-1.4.jar:/usr/hdp/current/druid-middlemanager/lib/jackson-datatype-guava-2.4.6.jar:/usr/hdp/current/druid-middlemanager/lib/jackson-core-2.4.6.jar:/usr/hdp/current/druid-middlemanager/lib/jline-0.9.94.jar:/usr/hdp/current/druid-middlemanager/lib/base64-2.3.8.jar:/usr/hdp/current/druid-middlemanager/lib/calcite-core-1.12.0.jar:/usr/hdp/current/druid-middlemanager/lib/mapdb-1.0.8.jar:/usr/hdp/current/druid-middlemanager/lib/curator-framework-2.11.0.jar:/usr/hdp/current/druid-middlemanager/lib/log4j-core-2.5.jar:/usr/hdp/current/druid-middlemanager/lib/jetty-servlets-9.3.19.v20170502.jar:/usr/hdp/current/druid-middlemanager/lib/druid-console-0.0.2.jar:/usr/hdp/current/druid-middlemanager/lib/maven-model-builder-3.1.1.jar:/usr/hdp/current/druid-middlemanager/lib/config-magic-0.9.jar:/usr/hdp/current/druid-middlemanager/lib/emitter-0.4.5.jar:/usr/hdp/current/druid-middlemanager/lib/slf4j-api-1.6.4.jar:/usr/hdp/current/druid-middlemanager/lib/jboss-logging-3.1.3.GA.jar:/usr/hdp/current/druid-middlemanager/lib/commons-cli-1.2.jar:/usr/hdp/current/druid-middlemanager/lib/guice-4.1.0.jar:/usr/hdp/current/druid-middlemanager/lib/druid-common-0.10.1.2.6.4.0-91.jar:/usr/hdp/current/druid-middlemanager/lib/jackson-dataformat-cbor-2.5.3.jar:/usr/hdp/current/druid-middlemanager/lib/derby-10.11.1.1.jar:/usr/hdp/current/druid-middlemanager/lib/aether-api-0.9.0.M2.jar:/usr/hdp/current/druid-middlemanager/lib/commons-io-2.5.jar:/usr/hdp/current/druid-middlemanager/lib/netty-3.10.6.Final.jar:/usr/hdp/current/druid-middlemanager/lib/javax.el-3.0.0.jar:/usr/hdp/current/druid-middlemanager/lib/janino-2.7.6.jar:/usr/hdp/current/druid-middlemanager/lib/plexus-utils-3.0.15.jar:/usr/hdp/current/druid-middlemanager/lib/jackson-jaxrs-base-2.4.6.jar:/usr/hdp/current/druid-middlemanager/lib/aws-java-sdk-core-1.10.77.jar:/usr/hdp/current/druid-middlemanager/lib/javax.servlet-api-3.1.0.jar:/usr/hdp/current/druid-middlemanager/lib/log4j-1.2-api-2.5.jar:/usr/hdp/current/druid-middlemanager/lib/avatica-core-1.9.0.jar:/usr/hdp/current/druid-middlemanager/lib/aether-impl-0.9.0.M2.jar:/usr/hdp/current/druid-middlemanager/lib/jetty-security-9.3.19.v20170502.jar:/usr/hdp/current/druid-middlemanager/lib/spymemcached-2.12.3.jar:/usr/hdp/current/druid-middlemanager/lib/maven-aether-provider-3.1.1.jar:/usr/hdp/current/druid-middlemanager/lib/aggdesigner-algorithm-6.0.jar:/usr/hdp/current/druid-middlemanager/lib/aether-connector-okhttp-0.0.9.jar:/usr/hdp/current/druid-middlemanager/lib/commons-pool2-2.2.jar:/usr/hdp/current/druid-middlemanager/lib/jackson-databind-2.4.6.jar:/usr/hdp/current/druid-middlemanager/lib/asm-tree-5.2.jar:/usr/hdp/current/druid-middlemanager/lib/jetty-continuation-9.3.19.v20170502.jar:/usr/hdp/current/druid-middlemanager/lib/java-util-0.28.2.jar:/usr/hdp/current/druid-middlemanager/lib/maven-repository-metadata-3.1.1.jar:/usr/hdp/current/druid-middlemanager/lib/jackson-annotations-2.4.6.jar:/usr/hdp/current/druid-middlemanager/lib/calcite-linq4j-1.12.0.jar:/usr/hdp/current/druid-middlemanager/lib/jersey-servlet-1.19.3.jar:/usr/hdp/current/druid-middlemanager/lib/jackson-mapper-asl-1.9.13.jar:/usr/hdp/current/druid-middlemanager/lib/asm-5.2.jar:/usr/hdp/current/druid-middlemanager/lib/asm-commons-5.2.jar:/usr/hdp/current/druid-middlemanager/lib/httpclient-4.5.1.jar:/usr/hdp/current/druid-middlemanager/lib/lz4-1.3.0.jar:/usr/hdp/current/druid-middlemanager/lib/avatica-metrics-1.9.0.jar:/usr/hdp/current/druid-middlemanager/lib/jetty-servlet-9.3.19.v20170502.jar:/usr/hdp/current/druid-middlemanager/lib/jackson-core-asl-1.9.13.jar:/usr/hdp/current/druid-middlemanager/lib/log4j-slf4j-impl-2.5.jar:/usr/hdp/current/druid-middlemanager/lib/javax.el-api-3.0.0.jar:/usr/hdp/current/druid-middlemanager/lib/irc-api-1.0-0014.jar:/usr/hdp/current/druid-middlemanager/lib/java-util-0.10.1.2.6.4.0-91.jar:/usr/hdp/current/druid-middlemanager/lib/jersey-guice-1.19.3.jar:/usr/hdp/current/druid-middlemanager/lib/activation-1.1.1.jar:/usr/hdp/current/druid-middlemanager/lib/druid-indexing-hadoop-0.10.1.2.6.4.0-91.jar:/usr/hdp/current/druid-middlemanager/lib/commons-lang-2.6.jar:/usr/hdp/current/druid-middlemanager/lib/jersey-server-1.19.3.jar:/usr/hdp/current/druid-middlemanager/lib/aether-connector-file-0.9.0.M2.jar:/usr/hdp/current/druid-middlemanager/lib/geoip2-0.4.0.jar:/usr/hdp/current/druid-middlemanager/lib/joda-time-2.9.9.jar:/usr/hdp/current/druid-middlemanager/lib/jetty-server-9.3.19.v20170502.jar:/usr/hdp/current/druid-middlemanager/lib/druid-hll-0.10.1.2.6.4.0-91.jar:/usr/hdp/current/druid-middlemanager/lib/guice-multibindings-4.1.0.jar:/usr/hdp/current/druid-middlemanager/lib/curator-x-discovery-2.11.0.jar:/usr/hdp/current/druid-middlemanager/lib/druid-api-0.10.1.2.6.4.0-91.jar:/usr/hdp/current/druid-middlemanager/lib/commons-lang3-3.2.jar:/usr/hdp/current/druid-middlemanager/lib/maxminddb-0.2.0.jar:/usr/hdp/current/druid-middlemanager/lib/jetty-io-9.3.19.v20170502.jar:/usr/hdp/current/druid-middlemanager/lib/maven-settings-builder-3.1.1.jar:/usr/hdp/current/druid-middlemanager/lib/jdbi-2.63.1.jar:/usr/hdp/current/druid-middlemanager/lib/aether-util-0.9.0.M2.jar:/usr/hdp/current/druid-middlemanager/lib/icu4j-54.1.1.jar:/usr/hdp/current/druid-middlemanager/lib/validation-api-1.1.0.Final.jar:/usr/hdp/current/druid-middlemanager/lib/jets3t-0.9.4.jar:/usr/hdp/current/druid-middlemanager/lib/jackson-module-jaxb-annotations-2.4.6.jar:/usr/hdp/current/druid-middlemanager/lib/server-metrics-0.2.8.jar:/usr/hdp/current/druid-middlemanager/lib/jetty-client-9.3.19.v20170502.jar:/usr/hdp/current/druid-middlemanager/lib/avatica-server-1.9.0.jar:/usr/hdp/current/druid-middlemanager/lib/httpcore-4.4.3.jar:/usr/hdp/current/druid-middlemanager/lib/airline-0.7.jar:/usr/hdp/current/druid-middlemanager/lib/jackson-jaxrs-json-provider-2.4.6.jar:/usr/hdp/current/druid-middlemanager/lib/druid-processing-0.10.1.2.6.4.0-91.jar:/usr/hdp/current/druid-middlemanager/lib/zookeeper-3.4.6.2.6.4.0-91.jar:/usr/hdp/current/druid-middlemanager/lib/guava-16.0.1.jar:/usr/hdp/current/druid-middlemanager/lib/commons-codec-1.7.jar:/usr/hdp/current/druid-middlemanager/lib/jersey-core-1.19.3.jar:/usr/hdp/current/druid-middlemanager/lib/commons-dbcp2-2.0.1.jar:/usr/hdp/current/druid-middlemanager/lib/hibernate-validator-5.1.3.Final.jar:/usr/hdp/current/druid-middlemanager/lib/java-xmlbuilder-1.1.jar:/usr/hdp/current/druid-middlemanager/lib/commons-math3-3.6.1.jar:/usr/hdp/current/druid-middlemanager/lib/rhino-1.7R5.jar:/usr/hdp/current/druid-middlemanager/lib/log4j-jul-2.5.jar:/usr/hdp/current/druid-middlemanager/lib/fastutil-7.2.0.jar:/usr/hdp/2.6.4.0-91/hadoop/conf 2018-03-29T14:55:19,617 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib 2018-03-29T14:55:19,617 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:java.io.tmpdir=/tmp 2018-03-29T14:55:19,617 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:java.compiler=<NA> 2018-03-29T14:55:19,617 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:os.name=Linux 2018-03-29T14:55:19,617 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:os.arch=amd64 2018-03-29T14:55:19,617 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:os.version=4.4.114-1.el7.elrepo.x86_64 2018-03-29T14:55:19,618 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:user.name=druid 2018-03-29T14:55:19,618 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:user.home=/home/druid 2018-03-29T14:55:19,618 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:user.dir=/home/druid 2018-03-29T14:55:19,619 INFO [main] org.apache.zookeeper.ZooKeeper - Initiating client connection, connectString=sandbox-hdp.hortonworks.com:2181 sessionTimeout=30000 watcher=org.apache.curator.ConnectionState@280fafd 2018-03-29T14:55:19,638 INFO [main] io.druid.java.util.common.lifecycle.Lifecycle$AnnotationBasedHandler - Invoking start method[public void io.druid.curator.discovery.ServerDiscoverySelector.start() throws java.lang.Exception] on object[io.druid.curator.discovery.ServerDiscoverySelector@44bc2449]. 2018-03-29T14:55:19,641 INFO [main-SendThread(sandbox-hdp.hortonworks.com:2181)] org.apache.zookeeper.ClientCnxn - Opening socket connection to server sandbox-hdp.hortonworks.com/172.17.0.2:2181. Will not attempt to authenticate using SASL (unknown error) 2018-03-29T14:55:19,649 INFO [main-SendThread(sandbox-hdp.hortonworks.com:2181)] org.apache.zookeeper.ClientCnxn - Socket connection established, initiating session, client: /172.17.0.2:46682, server: sandbox-hdp.hortonworks.com/172.17.0.2:2181 2018-03-29T14:55:19,659 INFO [main-SendThread(sandbox-hdp.hortonworks.com:2181)] org.apache.zookeeper.ClientCnxn - Session establishment complete on server sandbox-hdp.hortonworks.com/172.17.0.2:2181, sessionid = 0x16271e552860016, negotiated timeout = 30000 2018-03-29T14:55:19,663 INFO [main-EventThread] org.apache.curator.framework.state.ConnectionStateManager - State change: CONNECTED 2018-03-29T14:55:19,789 INFO [main] io.druid.java.util.common.lifecycle.Lifecycle$AnnotationBasedHandler - Invoking start method[public void io.druid.curator.announcement.Announcer.start()] on object[io.druid.curator.announcement.Announcer@69796bd0]. 2018-03-29T14:55:19,789 INFO [main] io.druid.java.util.common.lifecycle.Lifecycle$AnnotationBasedHandler - Invoking start method[public void io.druid.curator.discovery.ServerDiscoverySelector.start() throws java.lang.Exception] on object[io.druid.curator.discovery.ServerDiscoverySelector@40df6090]. 2018-03-29T14:55:19,806 INFO [main] io.druid.java.util.common.lifecycle.Lifecycle$AnnotationBasedHandler - Invoking start method[public void io.druid.indexing.worker.executor.ExecutorLifecycle.start() throws java.lang.InterruptedException] on object[io.druid.indexing.worker.executor.ExecutorLifecycle@1b37fbec]. 2018-03-29T14:55:19,854 INFO [main] io.druid.indexing.worker.executor.ExecutorLifecycle - Running with task: { "type" : "index_hadoop", "id" : "index_hadoop_wikiticker_2018-03-29T14:55:14.270Z", "spec" : { "dataSchema" : { "dataSource" : "wikiticker", "parser" : { "type" : "hadoopyString", "parseSpec" : { "format" : "json", "dimensionsSpec" : { "dimensions" : [ "channel", "cityName", "comment", "countryIsoCode", "countryName", "isAnonymous", "isMinor", "isNew", "isRobot", "isUnpatrolled", "metroCode", "namespace", "page", "regionIsoCode", "regionName", "user" ] }, "timestampSpec" : { "format" : "auto", "column" : "time" } } }, "metricsSpec" : [ { "type" : "count", "name" : "count" }, { "type" : "longSum", "name" : "added", "fieldName" : "added", "expression" : null }, { "type" : "longSum", "name" : "deleted", "fieldName" : "deleted", "expression" : null }, { "type" : "longSum", "name" : "delta", "fieldName" : "delta", "expression" : null }, { "type" : "hyperUnique", "name" : "user_unique", "fieldName" : "user", "isInputHyperUnique" : false, "round" : false } ], "granularitySpec" : { "type" : "uniform", "segmentGranularity" : "DAY", "queryGranularity" : { "type" : "none" }, "rollup" : true, "intervals" : [ "2015-09-12T00:00:00.000Z/2015-09-13T00:00:00.000Z" ] } }, "ioConfig" : { "type" : "hadoop", "inputSpec" : { "type" : "static", "paths" : "quickstart/wikiticker-2015-09-12-sampled.json.gz" }, "metadataUpdateSpec" : null, "segmentOutputPath" : null }, "tuningConfig" : { "type" : "hadoop", "workingPath" : null, "version" : "2018-03-29T14:55:14.250Z", "partitionsSpec" : { "type" : "hashed", "targetPartitionSize" : 5000000, "maxPartitionSize" : 7500000, "assumeGrouped" : false, "numShards" : -1, "partitionDimensions" : [ ] }, "shardSpecs" : { }, "indexSpec" : { "bitmap" : { "type" : "concise" }, "dimensionCompression" : "lz4", "metricCompression" : "lz4", "longEncoding" : "longs" }, "maxRowsInMemory" : 75000, "leaveIntermediate" : false, "cleanupOnFailure" : true, "overwriteFiles" : false, "ignoreInvalidRows" : false, "jobProperties" : { }, "combineText" : false, "useCombiner" : false, "buildV9Directly" : true, "numBackgroundPersistThreads" : 0, "forceExtendableShardSpecs" : false, "useExplicitVersion" : false, "allowedHadoopPrefix" : [ ] }, "uniqueId" : "ce51d0826af8424f874c1ad389c14f9b" }, "hadoopDependencyCoordinates" : null, "classpathPrefix" : null, "context" : null, "groupId" : "index_hadoop_wikiticker_2018-03-29T14:55:14.270Z", "dataSource" : "wikiticker", "resource" : { "availabilityGroup" : "index_hadoop_wikiticker_2018-03-29T14:55:14.270Z", "requiredCapacity" : 1 } } 2018-03-29T14:55:19,856 INFO [main] io.druid.indexing.worker.executor.ExecutorLifecycle - Attempting to lock file[/apps/druid/tasks/index_hadoop_wikiticker_2018-03-29T14:55:14.270Z/lock]. 2018-03-29T14:55:19,857 INFO [main] io.druid.indexing.worker.executor.ExecutorLifecycle - Acquired lock file[/apps/druid/tasks/index_hadoop_wikiticker_2018-03-29T14:55:14.270Z/lock] in 1ms. 2018-03-29T14:55:19,869 INFO [main] io.druid.indexing.common.actions.RemoteTaskActionClient - Performing action for task[index_hadoop_wikiticker_2018-03-29T14:55:14.270Z]: LockTryAcquireAction{interval=2015-09-12T00:00:00.000Z/2015-09-13T00:00:00.000Z} 2018-03-29T14:55:19,878 INFO [main] io.druid.indexing.common.actions.RemoteTaskActionClient - Submitting action for task[index_hadoop_wikiticker_2018-03-29T14:55:14.270Z] to overlord[http://sandbox-hdp.hortonworks.com:8090/druid/indexer/v1/action]: LockTryAcquireAction{interval=2015-09-12T00:00:00.000Z/2015-09-13T00:00:00.000Z} 2018-03-29T14:55:19,886 INFO [main] com.metamx.http.client.pool.ChannelResourceFactory - Generating: http://sandbox-hdp.hortonworks.com:8090 2018-03-29T14:55:19,912 INFO [main] com.metamx.http.client.pool.ChannelResourceFactory - Generating: http://sandbox-hdp.hortonworks.com:8090 2018-03-29T14:55:19,913 INFO [main] com.metamx.http.client.pool.ChannelResourceFactory - Generating: http://sandbox-hdp.hortonworks.com:8090 2018-03-29T14:55:19,913 INFO [main] com.metamx.http.client.pool.ChannelResourceFactory - Generating: http://sandbox-hdp.hortonworks.com:8090 2018-03-29T14:55:19,913 INFO [main] com.metamx.http.client.pool.ChannelResourceFactory - Generating: http://sandbox-hdp.hortonworks.com:8090 2018-03-29T14:55:19,915 INFO [main] com.metamx.http.client.pool.ChannelResourceFactory - Generating: http://sandbox-hdp.hortonworks.com:8090 2018-03-29T14:55:19,915 INFO [main] com.metamx.http.client.pool.ChannelResourceFactory - Generating: http://sandbox-hdp.hortonworks.com:8090 2018-03-29T14:55:19,918 INFO [main] com.metamx.http.client.pool.ChannelResourceFactory - Generating: http://sandbox-hdp.hortonworks.com:8090 2018-03-29T14:55:19,918 INFO [main] com.metamx.http.client.pool.ChannelResourceFactory - Generating: http://sandbox-hdp.hortonworks.com:8090 2018-03-29T14:55:19,922 INFO [main] com.metamx.http.client.pool.ChannelResourceFactory - Generating: http://sandbox-hdp.hortonworks.com:8090 2018-03-29T14:55:19,922 INFO [main] com.metamx.http.client.pool.ChannelResourceFactory - Generating: http://sandbox-hdp.hortonworks.com:8090 2018-03-29T14:55:19,923 INFO [main] com.metamx.http.client.pool.ChannelResourceFactory - Generating: http://sandbox-hdp.hortonworks.com:8090 2018-03-29T14:55:19,923 INFO [main] com.metamx.http.client.pool.ChannelResourceFactory - Generating: http://sandbox-hdp.hortonworks.com:8090 2018-03-29T14:55:19,924 INFO [main] com.metamx.http.client.pool.ChannelResourceFactory - Generating: http://sandbox-hdp.hortonworks.com:8090 2018-03-29T14:55:19,924 INFO [main] com.metamx.http.client.pool.ChannelResourceFactory - Generating: http://sandbox-hdp.hortonworks.com:8090 2018-03-29T14:55:19,924 INFO [main] com.metamx.http.client.pool.ChannelResourceFactory - Generating: http://sandbox-hdp.hortonworks.com:8090 2018-03-29T14:55:19,924 INFO [main] com.metamx.http.client.pool.ChannelResourceFactory - Generating: http://sandbox-hdp.hortonworks.com:8090 2018-03-29T14:55:19,925 INFO [main] com.metamx.http.client.pool.ChannelResourceFactory - Generating: http://sandbox-hdp.hortonworks.com:8090 2018-03-29T14:55:19,925 INFO [main] com.metamx.http.client.pool.ChannelResourceFactory - Generating: http://sandbox-hdp.hortonworks.com:8090 2018-03-29T14:55:19,925 INFO [main] com.metamx.http.client.pool.ChannelResourceFactory - Generating: http://sandbox-hdp.hortonworks.com:8090 2018-03-29T14:55:20,024 INFO [main] org.eclipse.jetty.server.Server - jetty-9.3.19.v20170502 2018-03-29T14:55:20,025 INFO [task-runner-0-priority-0] io.druid.indexing.overlord.ThreadPoolTaskRunner - Running task: index_hadoop_wikiticker_2018-03-29T14:55:14.270Z 2018-03-29T14:55:20,031 INFO [task-runner-0-priority-0] io.druid.indexing.overlord.TaskRunnerUtils - Task [index_hadoop_wikiticker_2018-03-29T14:55:14.270Z] location changed to [TaskLocation{host='sandbox-hdp.hortonworks.com', port=8100}]. 2018-03-29T14:55:20,032 INFO [task-runner-0-priority-0] io.druid.indexing.overlord.TaskRunnerUtils - Task [index_hadoop_wikiticker_2018-03-29T14:55:14.270Z] status changed to [RUNNING]. 2018-03-29T14:55:20,059 INFO [task-runner-0-priority-0] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/hadoop-dependencies/hadoop-client/2.7.3.2.6.4.0-91/jersey-core-1.9.jar] 2018-03-29T14:55:20,059 INFO [task-runner-0-priority-0] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/hadoop-dependencies/hadoop-client/2.7.3.2.6.4.0-91/hadoop-common-2.7.3.2.6.4.0-91.jar] 2018-03-29T14:55:20,084 INFO [task-runner-0-priority-0] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/hadoop-dependencies/hadoop-client/2.7.3.2.6.4.0-91/okio-1.4.0.jar] 2018-03-29T14:55:20,084 INFO [task-runner-0-priority-0] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/hadoop-dependencies/hadoop-client/2.7.3.2.6.4.0-91/curator-client-2.7.1.jar] 2018-03-29T14:55:20,084 INFO [task-runner-0-priority-0] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/hadoop-dependencies/hadoop-client/2.7.3.2.6.4.0-91/commons-beanutils-1.7.0.jar] 2018-03-29T14:55:20,084 INFO [task-runner-0-priority-0] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/hadoop-dependencies/hadoop-client/2.7.3.2.6.4.0-91/hadoop-annotations-2.7.3.2.6.4.0-91.jar] 2018-03-29T14:55:20,084 INFO [task-runner-0-priority-0] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/hadoop-dependencies/hadoop-client/2.7.3.2.6.4.0-91/jsp-api-2.1.jar] 2018-03-29T14:55:20,084 INFO [task-runner-0-priority-0] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/hadoop-dependencies/hadoop-client/2.7.3.2.6.4.0-91/snappy-java-1.0.4.1.jar] 2018-03-29T14:55:20,085 INFO [task-runner-0-priority-0] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/hadoop-dependencies/hadoop-client/2.7.3.2.6.4.0-91/curator-recipes-2.7.1.jar] 2018-03-29T14:55:20,085 INFO [task-runner-0-priority-0] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/hadoop-dependencies/hadoop-client/2.7.3.2.6.4.0-91/stax-api-1.0-2.jar] 2018-03-29T14:55:20,085 INFO [task-runner-0-priority-0] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/hadoop-dependencies/hadoop-client/2.7.3.2.6.4.0-91/servlet-api-2.5.jar] 2018-03-29T14:55:20,085 INFO [task-runner-0-priority-0] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/hadoop-dependencies/hadoop-client/2.7.3.2.6.4.0-91/slf4j-api-1.7.10.jar] 2018-03-29T14:55:20,085 INFO [task-runner-0-priority-0] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/hadoop-dependencies/hadoop-client/2.7.3.2.6.4.0-91/hadoop-client-2.7.3.2.6.4.0-91.jar] 2018-03-29T14:55:20,085 INFO [task-runner-0-priority-0] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/hadoop-dependencies/hadoop-client/2.7.3.2.6.4.0-91/netty-all-4.0.52.Final.jar] 2018-03-29T14:55:20,085 INFO [task-runner-0-priority-0] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/hadoop-dependencies/hadoop-client/2.7.3.2.6.4.0-91/hadoop-mapreduce-client-jobclient-2.7.3.2.6.4.0-91.jar] 2018-03-29T14:55:20,085 INFO [task-runner-0-priority-0] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/hadoop-dependencies/hadoop-client/2.7.3.2.6.4.0-91/commons-io-2.4.jar] 2018-03-29T14:55:20,085 INFO [task-runner-0-priority-0] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/hadoop-dependencies/hadoop-client/2.7.3.2.6.4.0-91/okhttp-2.4.0.jar] 2018-03-29T14:55:20,085 INFO [task-runner-0-priority-0] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/hadoop-dependencies/hadoop-client/2.7.3.2.6.4.0-91/api-util-1.0.0-M20.jar] 2018-03-29T14:55:20,086 INFO [task-runner-0-priority-0] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/hadoop-dependencies/hadoop-client/2.7.3.2.6.4.0-91/httpcore-4.4.4.jar] 2018-03-29T14:55:20,086 INFO [task-runner-0-priority-0] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/hadoop-dependencies/hadoop-client/2.7.3.2.6.4.0-91/commons-logging-1.1.3.jar] 2018-03-29T14:55:20,086 INFO [task-runner-0-priority-0] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/hadoop-dependencies/hadoop-client/2.7.3.2.6.4.0-91/jsr305-3.0.0.jar] 2018-03-29T14:55:20,086 INFO [task-runner-0-priority-0] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/hadoop-dependencies/hadoop-client/2.7.3.2.6.4.0-91/apacheds-i18n-2.0.0-M15.jar] 2018-03-29T14:55:20,087 INFO [task-runner-0-priority-0] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/hadoop-dependencies/hadoop-client/2.7.3.2.6.4.0-91/activation-1.1.jar] 2018-03-29T14:55:20,087 INFO [task-runner-0-priority-0] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/hadoop-dependencies/hadoop-client/2.7.3.2.6.4.0-91/nimbus-jose-jwt-3.9.jar] 2018-03-29T14:55:20,087 INFO [task-runner-0-priority-0] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/hadoop-dependencies/hadoop-client/2.7.3.2.6.4.0-91/xmlenc-0.52.jar] 2018-03-29T14:55:20,087 INFO [task-runner-0-priority-0] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/hadoop-dependencies/hadoop-client/2.7.3.2.6.4.0-91/htrace-core-3.1.0-incubating.jar] 2018-03-29T14:55:20,087 INFO [task-runner-0-priority-0] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/hadoop-dependencies/hadoop-client/2.7.3.2.6.4.0-91/xz-1.0.jar] 2018-03-29T14:55:20,087 INFO [task-runner-0-priority-0] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/hadoop-dependencies/hadoop-client/2.7.3.2.6.4.0-91/commons-math3-3.1.1.jar] 2018-03-29T14:55:20,087 INFO [task-runner-0-priority-0] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/hadoop-dependencies/hadoop-client/2.7.3.2.6.4.0-91/commons-beanutils-core-1.8.0.jar] 2018-03-29T14:55:20,087 INFO [task-runner-0-priority-0] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/hadoop-dependencies/hadoop-client/2.7.3.2.6.4.0-91/gson-2.2.4.jar] 2018-03-29T14:55:20,087 INFO [task-runner-0-priority-0] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/hadoop-dependencies/hadoop-client/2.7.3.2.6.4.0-91/azure-storage-5.4.0.jar] 2018-03-29T14:55:20,088 INFO [task-runner-0-priority-0] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/hadoop-dependencies/hadoop-client/2.7.3.2.6.4.0-91/protobuf-java-2.5.0.jar] 2018-03-29T14:55:20,088 INFO [task-runner-0-priority-0] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/hadoop-dependencies/hadoop-client/2.7.3.2.6.4.0-91/curator-framework-2.7.1.jar] 2018-03-29T14:55:20,088 INFO [task-runner-0-priority-0] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/hadoop-dependencies/hadoop-client/2.7.3.2.6.4.0-91/hadoop-mapreduce-client-core-2.7.3.2.6.4.0-91.jar] 2018-03-29T14:55:20,088 INFO [task-runner-0-priority-0] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/hadoop-dependencies/hadoop-client/2.7.3.2.6.4.0-91/slf4j-log4j12-1.7.10.jar] 2018-03-29T14:55:20,088 INFO [task-runner-0-priority-0] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/hadoop-dependencies/hadoop-client/2.7.3.2.6.4.0-91/jackson-annotations-2.2.3.jar] 2018-03-29T14:55:20,088 INFO [task-runner-0-priority-0] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/hadoop-dependencies/hadoop-client/2.7.3.2.6.4.0-91/jackson-xc-1.9.13.jar] 2018-03-29T14:55:20,088 INFO [task-runner-0-priority-0] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/hadoop-dependencies/hadoop-client/2.7.3.2.6.4.0-91/commons-cli-1.2.jar] 2018-03-29T14:55:20,088 INFO [task-runner-0-priority-0] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/hadoop-dependencies/hadoop-client/2.7.3.2.6.4.0-91/commons-configuration-1.6.jar] 2018-03-29T14:55:20,088 INFO [task-runner-0-priority-0] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/hadoop-dependencies/hadoop-client/2.7.3.2.6.4.0-91/hadoop-yarn-api-2.7.3.2.6.4.0-91.jar] 2018-03-29T14:55:20,089 INFO [task-runner-0-priority-0] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/hadoop-dependencies/hadoop-client/2.7.3.2.6.4.0-91/hadoop-yarn-client-2.7.3.2.6.4.0-91.jar] 2018-03-29T14:55:20,089 INFO [task-runner-0-priority-0] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/hadoop-dependencies/hadoop-client/2.7.3.2.6.4.0-91/hadoop-hdfs-2.7.3.2.6.4.0-91.jar] 2018-03-29T14:55:20,089 INFO [task-runner-0-priority-0] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/hadoop-dependencies/hadoop-client/2.7.3.2.6.4.0-91/log4j-1.2.17.jar] 2018-03-29T14:55:20,089 INFO [task-runner-0-priority-0] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/hadoop-dependencies/hadoop-client/2.7.3.2.6.4.0-91/hadoop-yarn-registry-2.7.3.2.6.4.0-91.jar] 2018-03-29T14:55:20,089 INFO [task-runner-0-priority-0] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/hadoop-dependencies/hadoop-client/2.7.3.2.6.4.0-91/azure-keyvault-core-0.8.0.jar] 2018-03-29T14:55:20,089 INFO [task-runner-0-priority-0] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/hadoop-dependencies/hadoop-client/2.7.3.2.6.4.0-91/jetty-util-6.1.26.hwx.jar] 2018-03-29T14:55:20,089 INFO [task-runner-0-priority-0] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/hadoop-dependencies/hadoop-client/2.7.3.2.6.4.0-91/hadoop-yarn-common-2.7.3.2.6.4.0-91.jar] 2018-03-29T14:55:20,089 INFO [task-runner-0-priority-0] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/hadoop-dependencies/hadoop-client/2.7.3.2.6.4.0-91/api-asn1-api-1.0.0-M20.jar] 2018-03-29T14:55:20,089 INFO [task-runner-0-priority-0] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/hadoop-dependencies/hadoop-client/2.7.3.2.6.4.0-91/hadoop-yarn-server-common-2.7.3.2.6.4.0-91.jar] 2018-03-29T14:55:20,090 INFO [task-runner-0-priority-0] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/hadoop-dependencies/hadoop-client/2.7.3.2.6.4.0-91/commons-lang3-3.4.jar] 2018-03-29T14:55:20,090 INFO [task-runner-0-priority-0] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/hadoop-dependencies/hadoop-client/2.7.3.2.6.4.0-91/hadoop-auth-2.7.3.2.6.4.0-91.jar] 2018-03-29T14:55:20,090 INFO [task-runner-0-priority-0] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/hadoop-dependencies/hadoop-client/2.7.3.2.6.4.0-91/commons-net-3.1.jar] 2018-03-29T14:55:20,090 INFO [task-runner-0-priority-0] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/hadoop-dependencies/hadoop-client/2.7.3.2.6.4.0-91/paranamer-2.3.jar] 2018-03-29T14:55:20,090 INFO [task-runner-0-priority-0] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/hadoop-dependencies/hadoop-client/2.7.3.2.6.4.0-91/jersey-client-1.9.jar] 2018-03-29T14:55:20,090 INFO [task-runner-0-priority-0] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/hadoop-dependencies/hadoop-client/2.7.3.2.6.4.0-91/hadoop-mapreduce-client-common-2.7.3.2.6.4.0-91.jar] 2018-03-29T14:55:20,090 INFO [task-runner-0-priority-0] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/hadoop-dependencies/hadoop-client/2.7.3.2.6.4.0-91/avro-1.7.4.jar] 2018-03-29T14:55:20,090 INFO [task-runner-0-priority-0] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/hadoop-dependencies/hadoop-client/2.7.3.2.6.4.0-91/httpclient-4.5.2.jar] 2018-03-29T14:55:20,090 INFO [task-runner-0-priority-0] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/hadoop-dependencies/hadoop-client/2.7.3.2.6.4.0-91/jaxb-api-2.2.2.jar] 2018-03-29T14:55:20,090 INFO [task-runner-0-priority-0] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/hadoop-dependencies/hadoop-client/2.7.3.2.6.4.0-91/guava-11.0.2.jar] 2018-03-29T14:55:20,091 INFO [task-runner-0-priority-0] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/hadoop-dependencies/hadoop-client/2.7.3.2.6.4.0-91/jackson-core-2.2.3.jar] 2018-03-29T14:55:20,091 INFO [task-runner-0-priority-0] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/hadoop-dependencies/hadoop-client/2.7.3.2.6.4.0-91/jackson-mapper-asl-1.9.13.jar] 2018-03-29T14:55:20,091 INFO [task-runner-0-priority-0] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/hadoop-dependencies/hadoop-client/2.7.3.2.6.4.0-91/jackson-jaxrs-1.9.13.jar] 2018-03-29T14:55:20,091 INFO [task-runner-0-priority-0] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/hadoop-dependencies/hadoop-client/2.7.3.2.6.4.0-91/jackson-core-asl-1.9.13.jar] 2018-03-29T14:55:20,091 INFO [task-runner-0-priority-0] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/hadoop-dependencies/hadoop-client/2.7.3.2.6.4.0-91/xml-apis-1.3.04.jar] 2018-03-29T14:55:20,091 INFO [task-runner-0-priority-0] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/hadoop-dependencies/hadoop-client/2.7.3.2.6.4.0-91/commons-lang-2.6.jar] 2018-03-29T14:55:20,092 INFO [task-runner-0-priority-0] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/hadoop-dependencies/hadoop-client/2.7.3.2.6.4.0-91/commons-compress-1.4.1.jar] 2018-03-29T14:55:20,092 INFO [task-runner-0-priority-0] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/hadoop-dependencies/hadoop-client/2.7.3.2.6.4.0-91/xercesImpl-2.9.1.jar] 2018-03-29T14:55:20,092 INFO [task-runner-0-priority-0] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/hadoop-dependencies/hadoop-client/2.7.3.2.6.4.0-91/jetty-sslengine-6.1.26.hwx.jar] 2018-03-29T14:55:20,092 INFO [task-runner-0-priority-0] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/hadoop-dependencies/hadoop-client/2.7.3.2.6.4.0-91/commons-digester-1.8.jar] 2018-03-29T14:55:20,092 INFO [task-runner-0-priority-0] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/hadoop-dependencies/hadoop-client/2.7.3.2.6.4.0-91/commons-codec-1.4.jar] 2018-03-29T14:55:20,092 INFO [task-runner-0-priority-0] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/hadoop-dependencies/hadoop-client/2.7.3.2.6.4.0-91/json-smart-1.1.1.jar] 2018-03-29T14:55:20,092 INFO [task-runner-0-priority-0] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/hadoop-dependencies/hadoop-client/2.7.3.2.6.4.0-91/apacheds-kerberos-codec-2.0.0-M15.jar] 2018-03-29T14:55:20,092 INFO [task-runner-0-priority-0] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/hadoop-dependencies/hadoop-client/2.7.3.2.6.4.0-91/netty-3.6.2.Final.jar] 2018-03-29T14:55:20,092 INFO [task-runner-0-priority-0] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/hadoop-dependencies/hadoop-client/2.7.3.2.6.4.0-91/jackson-databind-2.2.3.jar] 2018-03-29T14:55:20,092 INFO [task-runner-0-priority-0] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/hadoop-dependencies/hadoop-client/2.7.3.2.6.4.0-91/zookeeper-3.4.6.2.6.4.0-91.jar] 2018-03-29T14:55:20,093 INFO [task-runner-0-priority-0] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/hadoop-dependencies/hadoop-client/2.7.3.2.6.4.0-91/hadoop-mapreduce-client-shuffle-2.7.3.2.6.4.0-91.jar] 2018-03-29T14:55:20,093 INFO [task-runner-0-priority-0] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/hadoop-dependencies/hadoop-client/2.7.3.2.6.4.0-91/commons-collections-3.2.2.jar] 2018-03-29T14:55:20,093 INFO [task-runner-0-priority-0] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/hadoop-dependencies/hadoop-client/2.7.3.2.6.4.0-91/hadoop-mapreduce-client-app-2.7.3.2.6.4.0-91.jar] 2018-03-29T14:55:20,093 INFO [task-runner-0-priority-0] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/hadoop-dependencies/hadoop-client/2.7.3.2.6.4.0-91/jcip-annotations-1.0.jar] 2018-03-29T14:55:20,093 INFO [task-runner-0-priority-0] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/hadoop-dependencies/hadoop-client/2.7.3.2.6.4.0-91/leveldbjni-all-1.8.jar] 2018-03-29T14:55:20,093 INFO [task-runner-0-priority-0] io.druid.indexing.common.task.HadoopTask - Hadoop Container Druid Classpath is set to [file:/etc/druid/2.6.4.0-91/0/_common/:file:/etc/druid/2.6.4.0-91/0/middleManager/:file:/usr/hdp/2.6.4.0-91/druid/lib/antlr4-runtime-4.5.1.jar:file:/usr/hdp/2.6.4.0-91/druid/lib/druid-sql-0.10.1.2.6.4.0-91.jar:file:/usr/hdp/2.6.4.0-91/druid/lib/plexus-interpolation-1.19.jar:file:/usr/hdp/2.6.4.0-91/druid/lib/guice-servlet-4.1.0.jar:file:/usr/hdp/2.6.4.0-91/druid/lib/jackson-jaxrs-smile-provider-2.4.6.jar:file:/usr/hdp/2.6.4.0-91/druid/lib/curator-client-2.11.0.jar:file:/usr/hdp/2.6.4.0-91/druid/lib/tesla-aether-0.0.5.jar:file:/usr/hdp/2.6.4.0-91/druid/lib/wagon-provider-api-2.4.jar:file:/usr/hdp/2.6.4.0-91/druid/lib/extendedset-0.10.1.2.6.4.0-91.jar:file:/usr/hdp/2.6.4.0-91/druid/lib/jackson-dataformat-smile-2.4.6.jar:file:/usr/hdp/2.6.4.0-91/druid/lib/maven-model-3.1.1.jar:file:/usr/hdp/2.6.4.0-91/druid/lib/json-path-2.1.0.jar:file:/usr/hdp/2.6.4.0-91/druid/lib/jetty-util-9.3.19.v20170502.jar:file:/usr/hdp/2.6.4.0-91/druid/lib/commons-compiler-2.7.6.jar:file:/usr/hdp/2.6.4.0-91/druid/lib/log4j-api-2.5.jar:file:/usr/hdp/2.6.4.0-91/druid/lib/jcl-over-slf4j-1.7.12.jar:file:/usr/hdp/2.6.4.0-91/druid/lib/druid-server-0.10.1.2.6.4.0-91.jar:file:/usr/hdp/2.6.4.0-91/druid/lib/jetty-proxy-9.3.19.v20170502.jar:file:/usr/hdp/2.6.4.0-91/druid/lib/commons-pool-1.6.jar:file:/usr/hdp/2.6.4.0-91/druid/lib/jsr311-api-1.1.1.jar:file:/usr/hdp/2.6.4.0-91/druid/lib/jsr305-2.0.1.jar:file:/usr/hdp/2.6.4.0-91/druid/lib/druid-indexing-service-0.10.1.2.6.4.0-91.jar:file:/usr/hdp/2.6.4.0-91/druid/lib/commons-logging-1.1.1.jar:file:/usr/hdp/2.6.4.0-91/druid/lib/okhttp-1.0.2.jar:file:/usr/hdp/2.6.4.0-91/druid/lib/RoaringBitmap-0.5.18.jar:file:/usr/hdp/2.6.4.0-91/druid/lib/http-client-1.1.0.jar:file:/usr/hdp/2.6.4.0-91/druid/lib/curator-recipes-2.11.0.jar:file:/usr/hdp/2.6.4.0-91/druid/lib/aopalliance-1.0.jar:file:/usr/hdp/2.6.4.0-91/druid/lib/netty-all-4.1.11.Final.jar:file:/usr/hdp/2.6.4.0-91/druid/lib/derbyclient-10.11.1.1.jar:file:/usr/hdp/2.6.4.0-91/druid/lib/bytebuffer-collections-0.10.1.2.6.4.0-91.jar:file:/usr/hdp/2.6.4.0-91/druid/lib/protobuf-java-3.1.0.jar:file:/usr/hdp/2.6.4.0-91/druid/lib/aether-spi-0.9.0.M2.jar:file:/usr/hdp/2.6.4.0-91/druid/lib/derbynet-10.11.1.1.jar:file:/usr/hdp/2.6.4.0-91/druid/lib/compress-lzf-1.0.3.jar:file:/usr/hdp/2.6.4.0-91/druid/lib/disruptor-3.3.6.jar:file:/usr/hdp/2.6.4.0-91/druid/lib/jetty-http-9.3.19.v20170502.jar:file:/usr/hdp/2.6.4.0-91/druid/lib/maven-settings-3.1.1.jar:file:/usr/hdp/2.6.4.0-91/druid/lib/bcprov-jdk15on-1.52.jar:file:/usr/hdp/2.6.4.0-91/druid/lib/classmate-1.0.0.jar:file:/usr/hdp/2.6.4.0-91/druid/lib/opencsv-2.3.jar:file:/usr/hdp/2.6.4.0-91/druid/lib/druid-aws-common-0.10.1.2.6.4.0-91.jar:file:/usr/hdp/2.6.4.0-91/druid/lib/aws-java-sdk-ec2-1.10.77.jar:file:/usr/hdp/2.6.4.0-91/druid/lib/druid-services-0.10.1.2.6.4.0-91.jar:file:/usr/hdp/2.6.4.0-91/druid/lib/javax.inject-1.jar:file:/usr/hdp/2.6.4.0-91/druid/lib/jackson-datatype-joda-2.4.6.jar:file:/usr/hdp/2.6.4.0-91/druid/lib/commons-dbcp-1.4.jar:file:/usr/hdp/2.6.4.0-91/druid/lib/jackson-datatype-guava-2.4.6.jar:file:/usr/hdp/2.6.4.0-91/druid/lib/jackson-core-2.4.6.jar:file:/usr/hdp/2.6.4.0-91/druid/lib/jline-0.9.94.jar:file:/usr/hdp/2.6.4.0-91/druid/lib/base64-2.3.8.jar:file:/usr/hdp/2.6.4.0-91/druid/lib/calcite-core-1.12.0.jar:file:/usr/hdp/2.6.4.0-91/druid/lib/mapdb-1.0.8.jar:file:/usr/hdp/2.6.4.0-91/druid/lib/curator-framework-2.11.0.jar:file:/usr/hdp/2.6.4.0-91/druid/lib/log4j-core-2.5.jar:file:/usr/hdp/2.6.4.0-91/druid/lib/jetty-servlets-9.3.19.v20170502.jar:file:/usr/hdp/2.6.4.0-91/druid/lib/druid-console-0.0.2.jar:file:/usr/hdp/2.6.4.0-91/druid/lib/maven-model-builder-3.1.1.jar:file:/usr/hdp/2.6.4.0-91/druid/lib/config-magic-0.9.jar:file:/usr/hdp/2.6.4.0-91/druid/lib/emitter-0.4.5.jar:file:/usr/hdp/2.6.4.0-91/druid/lib/slf4j-api-1.6.4.jar:file:/usr/hdp/2.6.4.0-91/druid/lib/jboss-logging-3.1.3.GA.jar:file:/usr/hdp/2.6.4.0-91/druid/lib/commons-cli-1.2.jar:file:/usr/hdp/2.6.4.0-91/druid/lib/guice-4.1.0.jar:file:/usr/hdp/2.6.4.0-91/druid/lib/druid-common-0.10.1.2.6.4.0-91.jar:file:/usr/hdp/2.6.4.0-91/druid/lib/jackson-dataformat-cbor-2.5.3.jar:file:/usr/hdp/2.6.4.0-91/druid/lib/derby-10.11.1.1.jar:file:/usr/hdp/2.6.4.0-91/druid/lib/aether-api-0.9.0.M2.jar:file:/usr/hdp/2.6.4.0-91/druid/lib/commons-io-2.5.jar:file:/usr/hdp/2.6.4.0-91/druid/lib/netty-3.10.6.Final.jar:file:/usr/hdp/2.6.4.0-91/druid/lib/javax.el-3.0.0.jar:file:/usr/hdp/2.6.4.0-91/druid/lib/janino-2.7.6.jar:file:/usr/hdp/2.6.4.0-91/druid/lib/plexus-utils-3.0.15.jar:file:/usr/hdp/2.6.4.0-91/druid/lib/jackson-jaxrs-base-2.4.6.jar:file:/usr/hdp/2.6.4.0-91/druid/lib/aws-java-sdk-core-1.10.77.jar:file:/usr/hdp/2.6.4.0-91/druid/lib/javax.servlet-api-3.1.0.jar:file:/usr/hdp/2.6.4.0-91/druid/lib/log4j-1.2-api-2.5.jar:file:/usr/hdp/2.6.4.0-91/druid/lib/avatica-core-1.9.0.jar:file:/usr/hdp/2.6.4.0-91/druid/lib/aether-impl-0.9.0.M2.jar:file:/usr/hdp/2.6.4.0-91/druid/lib/jetty-security-9.3.19.v20170502.jar:file:/usr/hdp/2.6.4.0-91/druid/lib/spymemcached-2.12.3.jar:file:/usr/hdp/2.6.4.0-91/druid/lib/maven-aether-provider-3.1.1.jar:file:/usr/hdp/2.6.4.0-91/druid/lib/aggdesigner-algorithm-6.0.jar:file:/usr/hdp/2.6.4.0-91/druid/lib/aether-connector-okhttp-0.0.9.jar:file:/usr/hdp/2.6.4.0-91/druid/lib/commons-pool2-2.2.jar:file:/usr/hdp/2.6.4.0-91/druid/lib/jackson-databind-2.4.6.jar:file:/usr/hdp/2.6.4.0-91/druid/lib/asm-tree-5.2.jar:file:/usr/hdp/2.6.4.0-91/druid/lib/jetty-continuation-9.3.19.v20170502.jar:file:/usr/hdp/2.6.4.0-91/druid/lib/java-util-0.28.2.jar:file:/usr/hdp/2.6.4.0-91/druid/lib/maven-repository-metadata-3.1.1.jar:file:/usr/hdp/2.6.4.0-91/druid/lib/jackson-annotations-2.4.6.jar:file:/usr/hdp/2.6.4.0-91/druid/lib/calcite-linq4j-1.12.0.jar:file:/usr/hdp/2.6.4.0-91/druid/lib/jersey-servlet-1.19.3.jar:file:/usr/hdp/2.6.4.0-91/druid/lib/jackson-mapper-asl-1.9.13.jar:file:/usr/hdp/2.6.4.0-91/druid/lib/asm-5.2.jar:file:/usr/hdp/2.6.4.0-91/druid/lib/asm-commons-5.2.jar:file:/usr/hdp/2.6.4.0-91/druid/lib/httpclient-4.5.1.jar:file:/usr/hdp/2.6.4.0-91/druid/lib/lz4-1.3.0.jar:file:/usr/hdp/2.6.4.0-91/druid/lib/avatica-metrics-1.9.0.jar:file:/usr/hdp/2.6.4.0-91/druid/lib/jetty-servlet-9.3.19.v20170502.jar:file:/usr/hdp/2.6.4.0-91/druid/lib/jackson-core-asl-1.9.13.jar:file:/usr/hdp/2.6.4.0-91/druid/lib/log4j-slf4j-impl-2.5.jar:file:/usr/hdp/2.6.4.0-91/druid/lib/javax.el-api-3.0.0.jar:file:/usr/hdp/2.6.4.0-91/druid/lib/irc-api-1.0-0014.jar:file:/usr/hdp/2.6.4.0-91/druid/lib/java-util-0.10.1.2.6.4.0-91.jar:file:/usr/hdp/2.6.4.0-91/druid/lib/jersey-guice-1.19.3.jar:file:/usr/hdp/2.6.4.0-91/druid/lib/activation-1.1.1.jar:file:/usr/hdp/2.6.4.0-91/druid/lib/druid-indexing-hadoop-0.10.1.2.6.4.0-91.jar:file:/usr/hdp/2.6.4.0-91/druid/lib/commons-lang-2.6.jar:file:/usr/hdp/2.6.4.0-91/druid/lib/jersey-server-1.19.3.jar:file:/usr/hdp/2.6.4.0-91/druid/lib/aether-connector-file-0.9.0.M2.jar:file:/usr/hdp/2.6.4.0-91/druid/lib/geoip2-0.4.0.jar:file:/usr/hdp/2.6.4.0-91/druid/lib/joda-time-2.9.9.jar:file:/usr/hdp/2.6.4.0-91/druid/lib/jetty-server-9.3.19.v20170502.jar:file:/usr/hdp/2.6.4.0-91/druid/lib/druid-hll-0.10.1.2.6.4.0-91.jar:file:/usr/hdp/2.6.4.0-91/druid/lib/guice-multibindings-4.1.0.jar:file:/usr/hdp/2.6.4.0-91/druid/lib/curator-x-discovery-2.11.0.jar:file:/usr/hdp/2.6.4.0-91/druid/lib/druid-api-0.10.1.2.6.4.0-91.jar:file:/usr/hdp/2.6.4.0-91/druid/lib/commons-lang3-3.2.jar:file:/usr/hdp/2.6.4.0-91/druid/lib/maxminddb-0.2.0.jar:file:/usr/hdp/2.6.4.0-91/druid/lib/jetty-io-9.3.19.v20170502.jar:file:/usr/hdp/2.6.4.0-91/druid/lib/maven-settings-builder-3.1.1.jar:file:/usr/hdp/2.6.4.0-91/druid/lib/jdbi-2.63.1.jar:file:/usr/hdp/2.6.4.0-91/druid/lib/aether-util-0.9.0.M2.jar:file:/usr/hdp/2.6.4.0-91/druid/lib/icu4j-54.1.1.jar:file:/usr/hdp/2.6.4.0-91/druid/lib/validation-api-1.1.0.Final.jar:file:/usr/hdp/2.6.4.0-91/druid/lib/jets3t-0.9.4.jar:file:/usr/hdp/2.6.4.0-91/druid/lib/jackson-module-jaxb-annotations-2.4.6.jar:file:/usr/hdp/2.6.4.0-91/druid/lib/server-metrics-0.2.8.jar:file:/usr/hdp/2.6.4.0-91/druid/lib/jetty-client-9.3.19.v20170502.jar:file:/usr/hdp/2.6.4.0-91/druid/lib/avatica-server-1.9.0.jar:file:/usr/hdp/2.6.4.0-91/druid/lib/httpcore-4.4.3.jar:file:/usr/hdp/2.6.4.0-91/druid/lib/airline-0.7.jar:file:/usr/hdp/2.6.4.0-91/druid/lib/jackson-jaxrs-json-provider-2.4.6.jar:file:/usr/hdp/2.6.4.0-91/druid/lib/druid-processing-0.10.1.2.6.4.0-91.jar:file:/usr/hdp/2.6.4.0-91/druid/lib/zookeeper-3.4.6.2.6.4.0-91.jar:file:/usr/hdp/2.6.4.0-91/druid/lib/guava-16.0.1.jar:file:/usr/hdp/2.6.4.0-91/druid/lib/commons-codec-1.7.jar:file:/usr/hdp/2.6.4.0-91/druid/lib/jersey-core-1.19.3.jar:file:/usr/hdp/2.6.4.0-91/druid/lib/commons-dbcp2-2.0.1.jar:file:/usr/hdp/2.6.4.0-91/druid/lib/hibernate-validator-5.1.3.Final.jar:file:/usr/hdp/2.6.4.0-91/druid/lib/java-xmlbuilder-1.1.jar:file:/usr/hdp/2.6.4.0-91/druid/lib/commons-math3-3.6.1.jar:file:/usr/hdp/2.6.4.0-91/druid/lib/rhino-1.7R5.jar:file:/usr/hdp/2.6.4.0-91/druid/lib/log4j-jul-2.5.jar:file:/usr/hdp/2.6.4.0-91/druid/lib/fastutil-7.2.0.jar:file:/etc/hadoop/2.6.4.0-91/0/:file:/usr/hdp/current/druid-middlemanager/extensions/druid-datasketches/druid-datasketches-0.10.1.2.6.4.0-91.jar:file:/usr/hdp/current/druid-middlemanager/extensions/druid-datasketches/sketches-core-0.8.4.jar:file:/usr/hdp/current/druid-middlemanager/extensions/druid-datasketches/memory-0.8.4.jar:file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/hadoop-common-2.7.3.2.6.4.0-91.jar:file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/okio-1.4.0.jar:file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/curator-client-2.11.0.jar:file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/commons-beanutils-1.7.0.jar:file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/jsp-api-2.1.jar:file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/snappy-java-1.0.4.1.jar:file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/stax-api-1.0-2.jar:file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/servlet-api-2.5.jar:file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/hadoop-client-2.7.3.2.6.4.0-91.jar:file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/hadoop-mapreduce-client-jobclient-2.7.3.2.6.4.0-91.jar:file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/okhttp-2.4.0.jar:file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/commons-logging-1.1.1.jar:file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/api-util-1.0.0-M20.jar:file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/apacheds-i18n-2.0.0-M15.jar:file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/curator-recipes-2.11.0.jar:file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/netty-all-4.1.11.Final.jar:file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/aws-java-sdk-kms-1.10.77.jar:file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/druid-hdfs-storage-0.10.1.2.6.4.0-91.jar:file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/nimbus-jose-jwt-3.9.jar:file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/xmlenc-0.52.jar:file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/htrace-core-3.1.0-incubating.jar:file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/xz-1.0.jar:file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/commons-beanutils-core-1.8.0.jar:file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/gson-2.2.4.jar:file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/jackson-core-2.4.6.jar:file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/azure-storage-5.4.0.jar:file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/curator-framework-2.11.0.jar:file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/hadoop-mapreduce-client-core-2.7.3.2.6.4.0-91.jar:file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/jackson-xc-1.9.13.jar:file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/commons-configuration-1.6.jar:file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/jackson-dataformat-cbor-2.5.3.jar:file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/hadoop-yarn-api-2.7.3.2.6.4.0-91.jar:file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/hadoop-yarn-client-2.7.3.2.6.4.0-91.jar:file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/aws-java-sdk-core-1.10.77.jar:file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/hadoop-hdfs-2.7.3.2.6.4.0-91.jar:file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/hadoop-yarn-registry-2.7.3.2.6.4.0-91.jar:file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/azure-keyvault-core-0.8.0.jar:file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/hadoop-yarn-common-2.7.3.2.6.4.0-91.jar:file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/api-asn1-api-1.0.0-M20.jar:file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/hadoop-yarn-server-common-2.7.3.2.6.4.0-91.jar:file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/commons-lang3-3.4.jar:file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/hadoop-auth-2.7.3.2.6.4.0-91.jar:file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/commons-net-3.1.jar:file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/paranamer-2.3.jar:file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/jersey-client-1.9.jar:file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/hadoop-mapreduce-client-common-2.7.3.2.6.4.0-91.jar:file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/avro-1.7.4.jar:file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/jackson-databind-2.4.6.jar:file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/jaxb-api-2.2.2.jar:file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/jackson-annotations-2.4.6.jar:file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/httpclient-4.5.1.jar:file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/jackson-jaxrs-1.9.13.jar:file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/xml-apis-1.3.04.jar:file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/commons-compress-1.4.1.jar:file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/xercesImpl-2.9.1.jar:file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/joda-time-2.9.9.jar:file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/jetty-sslengine-6.1.26.hwx.jar:file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/commons-digester-1.8.jar:file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/hadoop-aws-2.7.3.2.6.4.0-91.jar:file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/json-smart-1.1.1.jar:file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/apacheds-kerberos-codec-2.0.0-M15.jar:file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/netty-3.6.2.Final.jar:file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/aws-java-sdk-s3-1.10.77.jar:file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/httpcore-4.4.3.jar:file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/guava-16.0.1.jar:file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/hadoop-mapreduce-client-shuffle-2.7.3.2.6.4.0-91.jar:file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/commons-collections-3.2.2.jar:file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/hadoop-mapreduce-client-app-2.7.3.2.6.4.0-91.jar:file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/jcip-annotations-1.0.jar:file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/commons-math3-3.6.1.jar:file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/leveldbjni-all-1.8.jar:file:/usr/hdp/current/druid-middlemanager/extensions/druid-kafka-indexing-service/snappy-java-1.1.2.6.jar:file:/usr/hdp/current/druid-middlemanager/extensions/druid-kafka-indexing-service/slf4j-api-1.6.4.jar:file:/usr/hdp/current/druid-middlemanager/extensions/druid-kafka-indexing-service/druid-kafka-indexing-service-0.10.1.2.6.4.0-91.jar:file:/usr/hdp/current/druid-middlemanager/extensions/druid-kafka-indexing-service/lz4-1.3.0.jar:file:/usr/hdp/current/druid-middlemanager/extensions/druid-kafka-indexing-service/kafka-clients-0.10.2.0.jar] 2018-03-29T14:55:20,234 INFO [main] com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory - Registering com.fasterxml.jackson.jaxrs.json.JacksonJsonProvider as a provider class 2018-03-29T14:55:20,234 INFO [main] com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory - Registering com.fasterxml.jackson.jaxrs.smile.JacksonSmileProvider as a provider class 2018-03-29T14:55:20,235 INFO [main] com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory - Registering io.druid.server.initialization.jetty.CustomExceptionMapper as a provider class 2018-03-29T14:55:20,235 INFO [main] com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory - Registering io.druid.server.StatusResource as a root resource class 2018-03-29T14:55:20,239 INFO [main] com.sun.jersey.server.impl.application.WebApplicationImpl - Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:43 PM' 2018-03-29T14:55:20,318 INFO [main] com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory - Binding io.druid.server.initialization.jetty.CustomExceptionMapper to GuiceManagedComponentProvider with the scope "Singleton" 2018-03-29T14:55:20,321 INFO [main] com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory - Binding com.fasterxml.jackson.jaxrs.json.JacksonJsonProvider to GuiceManagedComponentProvider with the scope "Singleton" 2018-03-29T14:55:20,329 INFO [main] com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory - Binding com.fasterxml.jackson.jaxrs.smile.JacksonSmileProvider to GuiceManagedComponentProvider with the scope "Singleton" 2018-03-29T14:55:20,668 INFO [main] com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory - Binding io.druid.server.http.security.StateResourceFilter to GuiceInstantiatedComponentProvider 2018-03-29T14:55:20,691 INFO [main] com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory - Binding io.druid.server.http.SegmentListerResource to GuiceInstantiatedComponentProvider 2018-03-29T14:55:20,704 INFO [main] com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory - Binding io.druid.server.QueryResource to GuiceInstantiatedComponentProvider 2018-03-29T14:55:20,705 INFO [main] com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory - Binding io.druid.segment.realtime.firehose.ChatHandlerResource to GuiceInstantiatedComponentProvider 2018-03-29T14:55:20,709 INFO [main] com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory - Binding io.druid.query.lookup.LookupListeningResource to GuiceInstantiatedComponentProvider 2018-03-29T14:55:20,711 INFO [main] com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory - Binding io.druid.query.lookup.LookupIntrospectionResource to GuiceInstantiatedComponentProvider 2018-03-29T14:55:20,712 INFO [main] com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory - Binding io.druid.server.StatusResource to GuiceManagedComponentProvider with the scope "Undefined" SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/usr/hdp/2.6.4.0-91/druid/lib/log4j-slf4j-impl-2.5.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/usr/hdp/current/druid-middlemanager/hadoop-dependencies/hadoop-client/2.7.3.2.6.4.0-91/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. 2018-03-29T14:55:20,726 WARN [main] com.sun.jersey.spi.inject.Errors - The following warnings have been detected with resource and/or provider classes: WARNING: A HTTP GET method, public void io.druid.server.http.SegmentListerResource.getSegments(long,long,long,javax.servlet.http.HttpServletRequest) throws java.io.IOException, MUST return a non-void type. 2018-03-29T14:55:20,734 INFO [main] org.eclipse.jetty.server.handler.ContextHandler - Started o.e.j.s.ServletContextHandler@3f3c8b60{/,null,AVAILABLE} 2018-03-29T14:55:20,743 INFO [main] org.eclipse.jetty.server.AbstractConnector - Started ServerConnector@4adf3582{HTTP/1.1,[http/1.1]}{0.0.0.0:8100} 2018-03-29T14:55:20,743 INFO [main] org.eclipse.jetty.server.Server - Started @6317ms 2018-03-29T14:55:20,744 INFO [main] io.druid.java.util.common.lifecycle.Lifecycle$AnnotationBasedHandler - Invoking start method[public void io.druid.server.listener.announcer.ListenerResourceAnnouncer.start()] on object[io.druid.query.lookup.LookupResourceListenerAnnouncer@577bfadb]. 2018-03-29T14:55:20,802 INFO [main] io.druid.server.listener.announcer.ListenerResourceAnnouncer - Announcing start time on [/druid/listeners/lookups/__default/sandbox-hdp.hortonworks.com:8100] 2018-03-29 14:55:21,220 task-runner-0-priority-0 WARN JNDI lookup class is not available because this JRE does not support JNDI. JNDI string lookups will not be available, continuing configuration. java.lang.ClassCastException: Cannot cast org.apache.logging.log4j.core.lookup.JndiLookup to org.apache.logging.log4j.core.lookup.StrLookup at java.lang.Class.cast(Class.java:3369) at org.apache.logging.log4j.util.LoaderUtil.newCheckedInstanceOf(LoaderUtil.java:167) at org.apache.logging.log4j.core.util.Loader.newCheckedInstanceOf(Loader.java:301) at org.apache.logging.log4j.core.lookup.Interpolator.<init>(Interpolator.java:94) at org.apache.logging.log4j.core.config.AbstractConfiguration.<init>(AbstractConfiguration.java:111) at org.apache.logging.log4j.core.config.DefaultConfiguration.<init>(DefaultConfiguration.java:48) at org.apache.logging.log4j.core.LoggerContext.<init>(LoggerContext.java:75) at org.apache.logging.log4j.core.selector.ClassLoaderContextSelector.createContext(ClassLoaderContextSelector.java:171) at org.apache.logging.log4j.core.selector.ClassLoaderContextSelector.locateContext(ClassLoaderContextSelector.java:145) at org.apache.logging.log4j.core.selector.ClassLoaderContextSelector.getContext(ClassLoaderContextSelector.java:70) at org.apache.logging.log4j.core.selector.ClassLoaderContextSelector.getContext(ClassLoaderContextSelector.java:57) at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:140) at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:41) at org.apache.logging.log4j.LogManager.getContext(LogManager.java:182) at org.apache.logging.log4j.spi.AbstractLoggerAdapter.getContext(AbstractLoggerAdapter.java:103) at org.apache.logging.log4j.jul.AbstractLoggerAdapter.getContext(AbstractLoggerAdapter.java:34) at org.apache.logging.log4j.spi.AbstractLoggerAdapter.getLogger(AbstractLoggerAdapter.java:42) at org.apache.logging.log4j.jul.LogManager.getLogger(LogManager.java:89) at java.util.logging.LogManager.demandLogger(LogManager.java:551) at java.util.logging.Logger.demandLogger(Logger.java:455) at java.util.logging.Logger.getLogger(Logger.java:502) at com.google.inject.internal.util.Stopwatch.<clinit>(Stopwatch.java:27) at com.google.inject.internal.InternalInjectorCreator.<init>(InternalInjectorCreator.java:61) at com.google.inject.Guice.createInjector(Guice.java:96) at com.google.inject.Guice.createInjector(Guice.java:73) at io.druid.guice.GuiceInjectors.makeStartupInjector(GuiceInjectors.java:61) at io.druid.indexer.HadoopDruidIndexerConfig.<clinit>(HadoopDruidIndexerConfig.java:104) at io.druid.indexing.common.task.HadoopIndexTask$HadoopDetermineConfigInnerProcessing.runTask(HadoopIndexTask.java:294) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at io.druid.indexing.common.task.HadoopTask.invokeForeignLoader(HadoopTask.java:215) at io.druid.indexing.common.task.HadoopIndexTask.run(HadoopIndexTask.java:178) at io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:436) at io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:408) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2018-03-29 14:55:21,222 task-runner-0-priority-0 WARN JMX runtime input lookup class is not available because this JRE does not support JMX. JMX lookups will not be available, continuing configuration. java.lang.ClassCastException: Cannot cast org.apache.logging.log4j.core.lookup.JmxRuntimeInputArgumentsLookup to org.apache.logging.log4j.core.lookup.StrLookup at java.lang.Class.cast(Class.java:3369) at org.apache.logging.log4j.util.LoaderUtil.newCheckedInstanceOf(LoaderUtil.java:167) at org.apache.logging.log4j.core.util.Loader.newCheckedInstanceOf(Loader.java:301) at org.apache.logging.log4j.core.lookup.Interpolator.<init>(Interpolator.java:105) at org.apache.logging.log4j.core.config.AbstractConfiguration.<init>(AbstractConfiguration.java:111) at org.apache.logging.log4j.core.config.DefaultConfiguration.<init>(DefaultConfiguration.java:48) at org.apache.logging.log4j.core.LoggerContext.<init>(LoggerContext.java:75) at org.apache.logging.log4j.core.selector.ClassLoaderContextSelector.createContext(ClassLoaderContextSelector.java:171) at org.apache.logging.log4j.core.selector.ClassLoaderContextSelector.locateContext(ClassLoaderContextSelector.java:145) at org.apache.logging.log4j.core.selector.ClassLoaderContextSelector.getContext(ClassLoaderContextSelector.java:70) at org.apache.logging.log4j.core.selector.ClassLoaderContextSelector.getContext(ClassLoaderContextSelector.java:57) at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:140) at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:41) at org.apache.logging.log4j.LogManager.getContext(LogManager.java:182) at org.apache.logging.log4j.spi.AbstractLoggerAdapter.getContext(AbstractLoggerAdapter.java:103) at org.apache.logging.log4j.jul.AbstractLoggerAdapter.getContext(AbstractLoggerAdapter.java:34) at org.apache.logging.log4j.spi.AbstractLoggerAdapter.getLogger(AbstractLoggerAdapter.java:42) at org.apache.logging.log4j.jul.LogManager.getLogger(LogManager.java:89) at java.util.logging.LogManager.demandLogger(LogManager.java:551) at java.util.logging.Logger.demandLogger(Logger.java:455) at java.util.logging.Logger.getLogger(Logger.java:502) at com.google.inject.internal.util.Stopwatch.<clinit>(Stopwatch.java:27) at com.google.inject.internal.InternalInjectorCreator.<init>(InternalInjectorCreator.java:61) at com.google.inject.Guice.createInjector(Guice.java:96) at com.google.inject.Guice.createInjector(Guice.java:73) at io.druid.guice.GuiceInjectors.makeStartupInjector(GuiceInjectors.java:61) at io.druid.indexer.HadoopDruidIndexerConfig.<clinit>(HadoopDruidIndexerConfig.java:104) at io.druid.indexing.common.task.HadoopIndexTask$HadoopDetermineConfigInnerProcessing.runTask(HadoopIndexTask.java:294) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at io.druid.indexing.common.task.HadoopTask.invokeForeignLoader(HadoopTask.java:215) at io.druid.indexing.common.task.HadoopIndexTask.run(HadoopIndexTask.java:178) at io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:436) at io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:408) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2018-03-29 14:55:21,240 task-runner-0-priority-0 WARN JNDI lookup class is not available because this JRE does not support JNDI. JNDI string lookups will not be available, continuing configuration. java.lang.ClassCastException: Cannot cast org.apache.logging.log4j.core.lookup.JndiLookup to org.apache.logging.log4j.core.lookup.StrLookup at java.lang.Class.cast(Class.java:3369) at org.apache.logging.log4j.util.LoaderUtil.newCheckedInstanceOf(LoaderUtil.java:167) at org.apache.logging.log4j.core.util.Loader.newCheckedInstanceOf(Loader.java:301) at org.apache.logging.log4j.core.lookup.Interpolator.<init>(Interpolator.java:94) at org.apache.logging.log4j.core.config.AbstractConfiguration.<init>(AbstractConfiguration.java:111) at org.apache.logging.log4j.core.config.xml.XmlConfiguration.<init>(XmlConfiguration.java:81) at org.apache.logging.log4j.core.config.xml.XmlConfigurationFactory.getConfiguration(XmlConfigurationFactory.java:44) at org.apache.logging.log4j.core.config.ConfigurationFactory$Factory.getConfiguration(ConfigurationFactory.java:490) at org.apache.logging.log4j.core.config.ConfigurationFactory$Factory.getConfiguration(ConfigurationFactory.java:460) at org.apache.logging.log4j.core.config.ConfigurationFactory.getConfiguration(ConfigurationFactory.java:256) at org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:561) at org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:578) at org.apache.logging.log4j.core.LoggerContext.start(LoggerContext.java:214) at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:145) at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:41) at org.apache.logging.log4j.LogManager.getContext(LogManager.java:182) at org.apache.logging.log4j.spi.AbstractLoggerAdapter.getContext(AbstractLoggerAdapter.java:103) at org.apache.logging.log4j.jul.AbstractLoggerAdapter.getContext(AbstractLoggerAdapter.java:34) at org.apache.logging.log4j.spi.AbstractLoggerAdapter.getLogger(AbstractLoggerAdapter.java:42) at org.apache.logging.log4j.jul.LogManager.getLogger(LogManager.java:89) at java.util.logging.LogManager.demandLogger(LogManager.java:551) at java.util.logging.Logger.demandLogger(Logger.java:455) at java.util.logging.Logger.getLogger(Logger.java:502) at com.google.inject.internal.util.Stopwatch.<clinit>(Stopwatch.java:27) at com.google.inject.internal.InternalInjectorCreator.<init>(InternalInjectorCreator.java:61) at com.google.inject.Guice.createInjector(Guice.java:96) at com.google.inject.Guice.createInjector(Guice.java:73) at io.druid.guice.GuiceInjectors.makeStartupInjector(GuiceInjectors.java:61) at io.druid.indexer.HadoopDruidIndexerConfig.<clinit>(HadoopDruidIndexerConfig.java:104) at io.druid.indexing.common.task.HadoopIndexTask$HadoopDetermineConfigInnerProcessing.runTask(HadoopIndexTask.java:294) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at io.druid.indexing.common.task.HadoopTask.invokeForeignLoader(HadoopTask.java:215) at io.druid.indexing.common.task.HadoopIndexTask.run(HadoopIndexTask.java:178) at io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:436) at io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:408) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2018-03-29 14:55:21,241 task-runner-0-priority-0 WARN JMX runtime input lookup class is not available because this JRE does not support JMX. JMX lookups will not be available, continuing configuration. java.lang.ClassCastException: Cannot cast org.apache.logging.log4j.core.lookup.JmxRuntimeInputArgumentsLookup to org.apache.logging.log4j.core.lookup.StrLookup at java.lang.Class.cast(Class.java:3369) at org.apache.logging.log4j.util.LoaderUtil.newCheckedInstanceOf(LoaderUtil.java:167) at org.apache.logging.log4j.core.util.Loader.newCheckedInstanceOf(Loader.java:301) at org.apache.logging.log4j.core.lookup.Interpolator.<init>(Interpolator.java:105) at org.apache.logging.log4j.core.config.AbstractConfiguration.<init>(AbstractConfiguration.java:111) at org.apache.logging.log4j.core.config.xml.XmlConfiguration.<init>(XmlConfiguration.java:81) at org.apache.logging.log4j.core.config.xml.XmlConfigurationFactory.getConfiguration(XmlConfigurationFactory.java:44) at org.apache.logging.log4j.core.config.ConfigurationFactory$Factory.getConfiguration(ConfigurationFactory.java:490) at org.apache.logging.log4j.core.config.ConfigurationFactory$Factory.getConfiguration(ConfigurationFactory.java:460) at org.apache.logging.log4j.core.config.ConfigurationFactory.getConfiguration(ConfigurationFactory.java:256) at org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:561) at org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:578) at org.apache.logging.log4j.core.LoggerContext.start(LoggerContext.java:214) at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:145) at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:41) at org.apache.logging.log4j.LogManager.getContext(LogManager.java:182) at org.apache.logging.log4j.spi.AbstractLoggerAdapter.getContext(AbstractLoggerAdapter.java:103) at org.apache.logging.log4j.jul.AbstractLoggerAdapter.getContext(AbstractLoggerAdapter.java:34) at org.apache.logging.log4j.spi.AbstractLoggerAdapter.getLogger(AbstractLoggerAdapter.java:42) at org.apache.logging.log4j.jul.LogManager.getLogger(LogManager.java:89) at java.util.logging.LogManager.demandLogger(LogManager.java:551) at java.util.logging.Logger.demandLogger(Logger.java:455) at java.util.logging.Logger.getLogger(Logger.java:502) at com.google.inject.internal.util.Stopwatch.<clinit>(Stopwatch.java:27) at com.google.inject.internal.InternalInjectorCreator.<init>(InternalInjectorCreator.java:61) at com.google.inject.Guice.createInjector(Guice.java:96) at com.google.inject.Guice.createInjector(Guice.java:73) at io.druid.guice.GuiceInjectors.makeStartupInjector(GuiceInjectors.java:61) at io.druid.indexer.HadoopDruidIndexerConfig.<clinit>(HadoopDruidIndexerConfig.java:104) at io.druid.indexing.common.task.HadoopIndexTask$HadoopDetermineConfigInnerProcessing.runTask(HadoopIndexTask.java:294) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at io.druid.indexing.common.task.HadoopTask.invokeForeignLoader(HadoopTask.java:215) at io.druid.indexing.common.task.HadoopIndexTask.run(HadoopIndexTask.java:178) at io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:436) at io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:408) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2018-03-29 14:55:21,250 task-runner-0-priority-0 WARN JNDI lookup class is not available because this JRE does not support JNDI. JNDI string lookups will not be available, continuing configuration. java.lang.ClassCastException: Cannot cast org.apache.logging.log4j.core.lookup.JndiLookup to org.apache.logging.log4j.core.lookup.StrLookup at java.lang.Class.cast(Class.java:3369) at org.apache.logging.log4j.util.LoaderUtil.newCheckedInstanceOf(LoaderUtil.java:167) at org.apache.logging.log4j.core.util.Loader.newCheckedInstanceOf(Loader.java:301) at org.apache.logging.log4j.core.lookup.Interpolator.<init>(Interpolator.java:94) at org.apache.logging.log4j.core.config.AbstractConfiguration.<init>(AbstractConfiguration.java:111) at org.apache.logging.log4j.core.config.DefaultConfiguration.<init>(DefaultConfiguration.java:48) at org.apache.logging.log4j.core.layout.PatternLayout$Builder.build(PatternLayout.java:444) at org.apache.logging.log4j.core.layout.PatternLayout.createDefaultLayout(PatternLayout.java:327) at org.apache.logging.log4j.core.appender.ConsoleAppender$Builder.<init>(ConsoleAppender.java:157) at org.apache.logging.log4j.core.appender.ConsoleAppender.newBuilder(ConsoleAppender.java:149) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.logging.log4j.core.config.plugins.util.PluginBuilder.createBuilder(PluginBuilder.java:154) at org.apache.logging.log4j.core.config.plugins.util.PluginBuilder.build(PluginBuilder.java:119) at org.apache.logging.log4j.core.config.AbstractConfiguration.createPluginObject(AbstractConfiguration.java:888) at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:828) at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:820) at org.apache.logging.log4j.core.config.AbstractConfiguration.doConfigure(AbstractConfiguration.java:449) at org.apache.logging.log4j.core.config.AbstractConfiguration.initialize(AbstractConfiguration.java:197) at org.apache.logging.log4j.core.config.AbstractConfiguration.start(AbstractConfiguration.java:209) at org.apache.logging.log4j.core.LoggerContext.setConfiguration(LoggerContext.java:492) at org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:562) at org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:578) at org.apache.logging.log4j.core.LoggerContext.start(LoggerContext.java:214) at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:145) at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:41) at org.apache.logging.log4j.LogManager.getContext(LogManager.java:182) at org.apache.logging.log4j.spi.AbstractLoggerAdapter.getContext(AbstractLoggerAdapter.java:103) at org.apache.logging.log4j.jul.AbstractLoggerAdapter.getContext(AbstractLoggerAdapter.java:34) at org.apache.logging.log4j.spi.AbstractLoggerAdapter.getLogger(AbstractLoggerAdapter.java:42) at org.apache.logging.log4j.jul.LogManager.getLogger(LogManager.java:89) at java.util.logging.LogManager.demandLogger(LogManager.java:551) at java.util.logging.Logger.demandLogger(Logger.java:455) at java.util.logging.Logger.getLogger(Logger.java:502) at com.google.inject.internal.util.Stopwatch.<clinit>(Stopwatch.java:27) at com.google.inject.internal.InternalInjectorCreator.<init>(InternalInjectorCreator.java:61) at com.google.inject.Guice.createInjector(Guice.java:96) at com.google.inject.Guice.createInjector(Guice.java:73) at io.druid.guice.GuiceInjectors.makeStartupInjector(GuiceInjectors.java:61) at io.druid.indexer.HadoopDruidIndexerConfig.<clinit>(HadoopDruidIndexerConfig.java:104) at io.druid.indexing.common.task.HadoopIndexTask$HadoopDetermineConfigInnerProcessing.runTask(HadoopIndexTask.java:294) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at io.druid.indexing.common.task.HadoopTask.invokeForeignLoader(HadoopTask.java:215) at io.druid.indexing.common.task.HadoopIndexTask.run(HadoopIndexTask.java:178) at io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:436) at io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:408) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2018-03-29 14:55:21,251 task-runner-0-priority-0 WARN JMX runtime input lookup class is not available because this JRE does not support JMX. JMX lookups will not be available, continuing configuration. java.lang.ClassCastException: Cannot cast org.apache.logging.log4j.core.lookup.JmxRuntimeInputArgumentsLookup to org.apache.logging.log4j.core.lookup.StrLookup at java.lang.Class.cast(Class.java:3369) at org.apache.logging.log4j.util.LoaderUtil.newCheckedInstanceOf(LoaderUtil.java:167) at org.apache.logging.log4j.core.util.Loader.newCheckedInstanceOf(Loader.java:301) at org.apache.logging.log4j.core.lookup.Interpolator.<init>(Interpolator.java:105) at org.apache.logging.log4j.core.config.AbstractConfiguration.<init>(AbstractConfiguration.java:111) at org.apache.logging.log4j.core.config.DefaultConfiguration.<init>(DefaultConfiguration.java:48) at org.apache.logging.log4j.core.layout.PatternLayout$Builder.build(PatternLayout.java:444) at org.apache.logging.log4j.core.layout.PatternLayout.createDefaultLayout(PatternLayout.java:327) at org.apache.logging.log4j.core.appender.ConsoleAppender$Builder.<init>(ConsoleAppender.java:157) at org.apache.logging.log4j.core.appender.ConsoleAppender.newBuilder(ConsoleAppender.java:149) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.logging.log4j.core.config.plugins.util.PluginBuilder.createBuilder(PluginBuilder.java:154) at org.apache.logging.log4j.core.config.plugins.util.PluginBuilder.build(PluginBuilder.java:119) at org.apache.logging.log4j.core.config.AbstractConfiguration.createPluginObject(AbstractConfiguration.java:888) at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:828) at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:820) at org.apache.logging.log4j.core.config.AbstractConfiguration.doConfigure(AbstractConfiguration.java:449) at org.apache.logging.log4j.core.config.AbstractConfiguration.initialize(AbstractConfiguration.java:197) at org.apache.logging.log4j.core.config.AbstractConfiguration.start(AbstractConfiguration.java:209) at org.apache.logging.log4j.core.LoggerContext.setConfiguration(LoggerContext.java:492) at org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:562) at org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:578) at org.apache.logging.log4j.core.LoggerContext.start(LoggerContext.java:214) at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:145) at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:41) at org.apache.logging.log4j.LogManager.getContext(LogManager.java:182) at org.apache.logging.log4j.spi.AbstractLoggerAdapter.getContext(AbstractLoggerAdapter.java:103) at org.apache.logging.log4j.jul.AbstractLoggerAdapter.getContext(AbstractLoggerAdapter.java:34) at org.apache.logging.log4j.spi.AbstractLoggerAdapter.getLogger(AbstractLoggerAdapter.java:42) at org.apache.logging.log4j.jul.LogManager.getLogger(LogManager.java:89) at java.util.logging.LogManager.demandLogger(LogManager.java:551) at java.util.logging.Logger.demandLogger(Logger.java:455) at java.util.logging.Logger.getLogger(Logger.java:502) at com.google.inject.internal.util.Stopwatch.<clinit>(Stopwatch.java:27) at com.google.inject.internal.InternalInjectorCreator.<init>(InternalInjectorCreator.java:61) at com.google.inject.Guice.createInjector(Guice.java:96) at com.google.inject.Guice.createInjector(Guice.java:73) at io.druid.guice.GuiceInjectors.makeStartupInjector(GuiceInjectors.java:61) at io.druid.indexer.HadoopDruidIndexerConfig.<clinit>(HadoopDruidIndexerConfig.java:104) at io.druid.indexing.common.task.HadoopIndexTask$HadoopDetermineConfigInnerProcessing.runTask(HadoopIndexTask.java:294) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at io.druid.indexing.common.task.HadoopTask.invokeForeignLoader(HadoopTask.java:215) at io.druid.indexing.common.task.HadoopIndexTask.run(HadoopIndexTask.java:178) at io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:436) at io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:408) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2018-03-29T14:55:21,499 INFO [task-runner-0-priority-0] io.druid.guice.PropertiesModule - Loading properties from common.runtime.properties 2018-03-29T14:55:21,501 INFO [task-runner-0-priority-0] io.druid.guice.PropertiesModule - Loading properties from runtime.properties 2018-03-29T14:55:21,530 INFO [task-runner-0-priority-0] org.hibernate.validator.internal.util.Version - HV000001: Hibernate Validator 5.1.3.Final 2018-03-29T14:55:22,078 INFO [task-runner-0-priority-0] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.segment.NullValueHandlingConfig] from props[druid.null.handling.] as [io.druid.segment.NullValueHandlingConfig@45f94603] 2018-03-29T14:55:22,092 INFO [task-runner-0-priority-0] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.guice.ModulesConfig] from props[druid.modules.] as [ModulesConfig{excludeList=[]}] 2018-03-29T14:55:22,216 INFO [task-runner-0-priority-0] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.guice.ExtensionsConfig] from props[druid.extensions.] as [ExtensionsConfig{searchCurrentClassloader=true, directory='/usr/hdp/current/druid-middlemanager/extensions', hadoopDependenciesDir='/usr/hdp/current/druid-middlemanager/hadoop-dependencies', hadoopContainerDruidClasspath='null', addExtensionsToHadoopContainer=false, loadList=[druid-datasketches, druid-hdfs-storage, druid-kafka-indexing-service]}] 2018-03-29T14:55:22,219 INFO [task-runner-0-priority-0] io.druid.initialization.Initialization - Adding implementation [io.druid.query.aggregation.datasketches.theta.SketchModule] for class [interface io.druid.initialization.DruidModule] from classpath extension 2018-03-29T14:55:22,220 INFO [task-runner-0-priority-0] io.druid.initialization.Initialization - Adding implementation [io.druid.query.aggregation.datasketches.theta.oldapi.OldApiSketchModule] for class [interface io.druid.initialization.DruidModule] from classpath extension 2018-03-29T14:55:22,221 INFO [task-runner-0-priority-0] io.druid.initialization.Initialization - Adding implementation [io.druid.storage.hdfs.HdfsStorageDruidModule] for class [interface io.druid.initialization.DruidModule] from classpath extension 2018-03-29T14:55:22,222 INFO [task-runner-0-priority-0] io.druid.initialization.Initialization - Adding implementation [io.druid.indexing.kafka.KafkaIndexTaskModule] for class [interface io.druid.initialization.DruidModule] from classpath extension 2018-03-29T14:55:22,223 INFO [task-runner-0-priority-0] io.druid.initialization.Initialization - Loading extension [druid-datasketches] for class [interface io.druid.initialization.DruidModule] 2018-03-29T14:55:22,232 INFO [task-runner-0-priority-0] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-datasketches/druid-datasketches-0.10.1.2.6.4.0-91.jar] 2018-03-29T14:55:22,232 INFO [task-runner-0-priority-0] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-datasketches/sketches-core-0.8.4.jar] 2018-03-29T14:55:22,233 INFO [task-runner-0-priority-0] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-datasketches/memory-0.8.4.jar] 2018-03-29T14:55:22,234 INFO [task-runner-0-priority-0] io.druid.initialization.Initialization - Loading extension [druid-hdfs-storage] for class [interface io.druid.initialization.DruidModule] 2018-03-29T14:55:22,236 INFO [task-runner-0-priority-0] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/hadoop-common-2.7.3.2.6.4.0-91.jar] 2018-03-29T14:55:22,236 INFO [task-runner-0-priority-0] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/okio-1.4.0.jar] 2018-03-29T14:55:22,236 INFO [task-runner-0-priority-0] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/curator-client-2.11.0.jar] 2018-03-29T14:55:22,236 INFO [task-runner-0-priority-0] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/commons-beanutils-1.7.0.jar] 2018-03-29T14:55:22,236 INFO [task-runner-0-priority-0] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/jsp-api-2.1.jar] 2018-03-29T14:55:22,236 INFO [task-runner-0-priority-0] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/snappy-java-1.0.4.1.jar] 2018-03-29T14:55:22,236 INFO [task-runner-0-priority-0] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/stax-api-1.0-2.jar] 2018-03-29T14:55:22,236 INFO [task-runner-0-priority-0] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/servlet-api-2.5.jar] 2018-03-29T14:55:22,237 INFO [task-runner-0-priority-0] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/hadoop-client-2.7.3.2.6.4.0-91.jar] 2018-03-29T14:55:22,237 INFO [task-runner-0-priority-0] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/hadoop-mapreduce-client-jobclient-2.7.3.2.6.4.0-91.jar] 2018-03-29T14:55:22,237 INFO [task-runner-0-priority-0] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/okhttp-2.4.0.jar] 2018-03-29T14:55:22,237 INFO [task-runner-0-priority-0] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/commons-logging-1.1.1.jar] 2018-03-29T14:55:22,237 INFO [task-runner-0-priority-0] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/api-util-1.0.0-M20.jar] 2018-03-29T14:55:22,237 INFO [task-runner-0-priority-0] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/apacheds-i18n-2.0.0-M15.jar] 2018-03-29T14:55:22,237 INFO [task-runner-0-priority-0] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/curator-recipes-2.11.0.jar] 2018-03-29T14:55:22,237 INFO [task-runner-0-priority-0] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/netty-all-4.1.11.Final.jar] 2018-03-29T14:55:22,237 INFO [task-runner-0-priority-0] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/aws-java-sdk-kms-1.10.77.jar] 2018-03-29T14:55:22,237 INFO [task-runner-0-priority-0] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/druid-hdfs-storage-0.10.1.2.6.4.0-91.jar] 2018-03-29T14:55:22,237 INFO [task-runner-0-priority-0] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/nimbus-jose-jwt-3.9.jar] 2018-03-29T14:55:22,237 INFO [task-runner-0-priority-0] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/xmlenc-0.52.jar] 2018-03-29T14:55:22,238 INFO [task-runner-0-priority-0] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/htrace-core-3.1.0-incubating.jar] 2018-03-29T14:55:22,238 INFO [task-runner-0-priority-0] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/xz-1.0.jar] 2018-03-29T14:55:22,238 INFO [task-runner-0-priority-0] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/commons-beanutils-core-1.8.0.jar] 2018-03-29T14:55:22,238 INFO [task-runner-0-priority-0] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/gson-2.2.4.jar] 2018-03-29T14:55:22,238 INFO [task-runner-0-priority-0] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/jackson-core-2.4.6.jar] 2018-03-29T14:55:22,238 INFO [task-runner-0-priority-0] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/azure-storage-5.4.0.jar] 2018-03-29T14:55:22,238 INFO [task-runner-0-priority-0] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/curator-framework-2.11.0.jar] 2018-03-29T14:55:22,238 INFO [task-runner-0-priority-0] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/hadoop-mapreduce-client-core-2.7.3.2.6.4.0-91.jar] 2018-03-29T14:55:22,238 INFO [task-runner-0-priority-0] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/jackson-xc-1.9.13.jar] 2018-03-29T14:55:22,238 INFO [task-runner-0-priority-0] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/commons-configuration-1.6.jar] 2018-03-29T14:55:22,239 INFO [task-runner-0-priority-0] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/jackson-dataformat-cbor-2.5.3.jar] 2018-03-29T14:55:22,239 INFO [task-runner-0-priority-0] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/hadoop-yarn-api-2.7.3.2.6.4.0-91.jar] 2018-03-29T14:55:22,239 INFO [task-runner-0-priority-0] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/hadoop-yarn-client-2.7.3.2.6.4.0-91.jar] 2018-03-29T14:55:22,239 INFO [task-runner-0-priority-0] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/aws-java-sdk-core-1.10.77.jar] 2018-03-29T14:55:22,239 INFO [task-runner-0-priority-0] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/hadoop-hdfs-2.7.3.2.6.4.0-91.jar] 2018-03-29T14:55:22,240 INFO [task-runner-0-priority-0] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/hadoop-yarn-registry-2.7.3.2.6.4.0-91.jar] 2018-03-29T14:55:22,240 INFO [task-runner-0-priority-0] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/azure-keyvault-core-0.8.0.jar] 2018-03-29T14:55:22,240 INFO [task-runner-0-priority-0] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/hadoop-yarn-common-2.7.3.2.6.4.0-91.jar] 2018-03-29T14:55:22,240 INFO [task-runner-0-priority-0] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/api-asn1-api-1.0.0-M20.jar] 2018-03-29T14:55:22,240 INFO [task-runner-0-priority-0] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/hadoop-yarn-server-common-2.7.3.2.6.4.0-91.jar] 2018-03-29T14:55:22,240 INFO [task-runner-0-priority-0] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/commons-lang3-3.4.jar] 2018-03-29T14:55:22,240 INFO [task-runner-0-priority-0] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/hadoop-auth-2.7.3.2.6.4.0-91.jar] 2018-03-29T14:55:22,240 INFO [task-runner-0-priority-0] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/commons-net-3.1.jar] 2018-03-29T14:55:22,241 INFO [task-runner-0-priority-0] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/paranamer-2.3.jar] 2018-03-29T14:55:22,241 INFO [task-runner-0-priority-0] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/jersey-client-1.9.jar] 2018-03-29T14:55:22,241 INFO [task-runner-0-priority-0] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/hadoop-mapreduce-client-common-2.7.3.2.6.4.0-91.jar] 2018-03-29T14:55:22,241 INFO [task-runner-0-priority-0] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/avro-1.7.4.jar] 2018-03-29T14:55:22,241 INFO [task-runner-0-priority-0] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/jackson-databind-2.4.6.jar] 2018-03-29T14:55:22,241 INFO [task-runner-0-priority-0] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/jaxb-api-2.2.2.jar] 2018-03-29T14:55:22,242 INFO [task-runner-0-priority-0] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/jackson-annotations-2.4.6.jar] 2018-03-29T14:55:22,242 INFO [task-runner-0-priority-0] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/httpclient-4.5.1.jar] 2018-03-29T14:55:22,242 INFO [task-runner-0-priority-0] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/jackson-jaxrs-1.9.13.jar] 2018-03-29T14:55:22,242 INFO [task-runner-0-priority-0] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/xml-apis-1.3.04.jar] 2018-03-29T14:55:22,242 INFO [task-runner-0-priority-0] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/commons-compress-1.4.1.jar] 2018-03-29T14:55:22,242 INFO [task-runner-0-priority-0] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/xercesImpl-2.9.1.jar] 2018-03-29T14:55:22,242 INFO [task-runner-0-priority-0] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/joda-time-2.9.9.jar] 2018-03-29T14:55:22,242 INFO [task-runner-0-priority-0] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/jetty-sslengine-6.1.26.hwx.jar] 2018-03-29T14:55:22,243 INFO [task-runner-0-priority-0] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/commons-digester-1.8.jar] 2018-03-29T14:55:22,243 INFO [task-runner-0-priority-0] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/hadoop-aws-2.7.3.2.6.4.0-91.jar] 2018-03-29T14:55:22,243 INFO [task-runner-0-priority-0] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/json-smart-1.1.1.jar] 2018-03-29T14:55:22,243 INFO [task-runner-0-priority-0] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/apacheds-kerberos-codec-2.0.0-M15.jar] 2018-03-29T14:55:22,243 INFO [task-runner-0-priority-0] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/netty-3.6.2.Final.jar] 2018-03-29T14:55:22,243 INFO [task-runner-0-priority-0] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/aws-java-sdk-s3-1.10.77.jar] 2018-03-29T14:55:22,243 INFO [task-runner-0-priority-0] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/httpcore-4.4.3.jar] 2018-03-29T14:55:22,243 INFO [task-runner-0-priority-0] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/guava-16.0.1.jar] 2018-03-29T14:55:22,243 INFO [task-runner-0-priority-0] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/hadoop-mapreduce-client-shuffle-2.7.3.2.6.4.0-91.jar] 2018-03-29T14:55:22,243 INFO [task-runner-0-priority-0] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/commons-collections-3.2.2.jar] 2018-03-29T14:55:22,243 INFO [task-runner-0-priority-0] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/hadoop-mapreduce-client-app-2.7.3.2.6.4.0-91.jar] 2018-03-29T14:55:22,243 INFO [task-runner-0-priority-0] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/jcip-annotations-1.0.jar] 2018-03-29T14:55:22,244 INFO [task-runner-0-priority-0] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/commons-math3-3.6.1.jar] 2018-03-29T14:55:22,244 INFO [task-runner-0-priority-0] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/leveldbjni-all-1.8.jar] 2018-03-29T14:55:22,253 INFO [task-runner-0-priority-0] io.druid.initialization.Initialization - Loading extension [druid-kafka-indexing-service] for class [interface io.druid.initialization.DruidModule] 2018-03-29T14:55:22,253 INFO [task-runner-0-priority-0] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-kafka-indexing-service/snappy-java-1.1.2.6.jar] 2018-03-29T14:55:22,253 INFO [task-runner-0-priority-0] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-kafka-indexing-service/slf4j-api-1.6.4.jar] 2018-03-29T14:55:22,254 INFO [task-runner-0-priority-0] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-kafka-indexing-service/druid-kafka-indexing-service-0.10.1.2.6.4.0-91.jar] 2018-03-29T14:55:22,254 INFO [task-runner-0-priority-0] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-kafka-indexing-service/lz4-1.3.0.jar] 2018-03-29T14:55:22,254 INFO [task-runner-0-priority-0] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-kafka-indexing-service/kafka-clients-0.10.2.0.jar] 2018-03-29T14:55:22,782 WARN [task-runner-0-priority-0] org.apache.hadoop.util.NativeCodeLoader - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2018-03-29T14:55:23,636 WARN [task-runner-0-priority-0] org.apache.hadoop.hdfs.shortcircuit.DomainSocketFactory - The short-circuit local reads feature cannot be used because libhadoop cannot be loaded. 2018-03-29T14:55:23,960 INFO [task-runner-0-priority-0] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.server.metrics.DruidMonitorSchedulerConfig] from props[druid.monitoring.] as [io.druid.server.metrics.DruidMonitorSchedulerConfig@2363f83d] 2018-03-29T14:55:23,998 INFO [task-runner-0-priority-0] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.server.metrics.MonitorsConfig] from props[druid.monitoring.] as [MonitorsConfig{monitors=[class com.metamx.metrics.JvmMonitor]}] 2018-03-29T14:55:24,002 INFO [task-runner-0-priority-0] io.druid.server.metrics.MetricsModule - Adding monitor[com.metamx.metrics.JvmMonitor@76544e65] 2018-03-29T14:55:24,002 INFO [task-runner-0-priority-0] io.druid.server.metrics.MetricsModule - Adding monitor[io.druid.server.initialization.jetty.JettyServerModule$JettyMonitor@f2cbb0e] 2018-03-29T14:55:24,008 INFO [task-runner-0-priority-0] org.skife.config.ConfigurationObjectFactory - Assigning value [50000000] for [druid.processing.buffer.sizeBytes] on [io.druid.query.DruidProcessingConfig#intermediateComputeSizeBytes()] 2018-03-29T14:55:24,010 INFO [task-runner-0-priority-0] org.skife.config.ConfigurationObjectFactory - Using method itself for [druid.computation.buffer.poolCacheMaxCount, ${base_path}.buffer.poolCacheMaxCount] on [io.druid.query.DruidProcessingConfig#poolCacheMaxCount()] 2018-03-29T14:55:24,011 INFO [task-runner-0-priority-0] org.skife.config.ConfigurationObjectFactory - Using method itself for [${base_path}.tmpDir] on [io.druid.query.DruidProcessingConfig#getTmpDir()] 2018-03-29T14:55:24,011 INFO [task-runner-0-priority-0] org.skife.config.ConfigurationObjectFactory - Using method itself for [${base_path}.fifo] on [io.druid.query.DruidProcessingConfig#isFifo()] 2018-03-29T14:55:24,011 INFO [task-runner-0-priority-0] org.skife.config.ConfigurationObjectFactory - Assigning value [2] for [druid.processing.numThreads] on [io.druid.query.DruidProcessingConfig#getNumThreadsConfigured()] 2018-03-29T14:55:24,011 INFO [task-runner-0-priority-0] org.skife.config.ConfigurationObjectFactory - Using method itself for [${base_path}.numMergeBuffers] on [io.druid.query.DruidProcessingConfig#getNumMergeBuffersConfigured()] 2018-03-29T14:55:24,012 INFO [task-runner-0-priority-0] org.skife.config.ConfigurationObjectFactory - Using method itself for [${base_path}.columnCache.sizeBytes] on [io.druid.query.DruidProcessingConfig#columnCacheSizeBytes()] 2018-03-29T14:55:24,012 INFO [task-runner-0-priority-0] org.skife.config.ConfigurationObjectFactory - Assigning default value [processing-%s] for [${base_path}.formatString] on [io.druid.java.util.common.concurrent.ExecutorServiceConfig#getFormatString()] 2018-03-29T14:55:24,115 INFO [task-runner-0-priority-0] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.indexer.HadoopKerberosConfig] from props[druid.hadoop.security.kerberos.] as [io.druid.indexer.HadoopKerberosConfig@0] 2018-03-29T14:55:24,122 INFO [task-runner-0-priority-0] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.storage.hdfs.HdfsDataSegmentPusherConfig] from props[druid.storage.] as [io.druid.storage.hdfs.HdfsDataSegmentPusherConfig@4c55d286] 2018-03-29T14:55:24,124 INFO [task-runner-0-priority-0] io.druid.storage.hdfs.HdfsDataSegmentPusher - Configured HDFS as deep storage 2018-03-29T14:55:24,284 INFO [task-runner-0-priority-0] io.druid.guice.PropertiesModule - Loading properties from common.runtime.properties 2018-03-29T14:55:24,284 INFO [task-runner-0-priority-0] io.druid.guice.PropertiesModule - Loading properties from runtime.properties 2018-03-29T14:55:24,293 INFO [task-runner-0-priority-0] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.segment.NullValueHandlingConfig] from props[druid.null.handling.] as [io.druid.segment.NullValueHandlingConfig@4b4d4e43] 2018-03-29T14:55:24,299 INFO [task-runner-0-priority-0] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.guice.ExtensionsConfig] from props[druid.extensions.] as [ExtensionsConfig{searchCurrentClassloader=true, directory='/usr/hdp/current/druid-middlemanager/extensions', hadoopDependenciesDir='/usr/hdp/current/druid-middlemanager/hadoop-dependencies', hadoopContainerDruidClasspath='null', addExtensionsToHadoopContainer=false, loadList=[druid-datasketches, druid-hdfs-storage, druid-kafka-indexing-service]}] 2018-03-29T14:55:24,300 INFO [task-runner-0-priority-0] io.druid.indexing.common.task.HadoopIndexTask - Starting a hadoop determine configuration job... 2018-03-29T14:55:24,345 INFO [task-runner-0-priority-0] io.druid.indexer.path.StaticPathSpec - Adding paths[quickstart/wikiticker-2015-09-12-sampled.json.gz] 2018-03-29T14:55:25,184 INFO [task-runner-0-priority-0] io.druid.indexer.path.StaticPathSpec - Adding paths[quickstart/wikiticker-2015-09-12-sampled.json.gz] 2018-03-29T14:55:25,513 INFO [task-runner-0-priority-0] org.apache.hadoop.yarn.client.RMProxy - Connecting to ResourceManager at sandbox-hdp.hortonworks.com/172.17.0.2:8032 2018-03-29T14:55:25,645 INFO [task-runner-0-priority-0] org.apache.hadoop.yarn.client.AHSProxy - Connecting to Application History server at sandbox-hdp.hortonworks.com/172.17.0.2:10200 2018-03-29T14:55:25,774 WARN [task-runner-0-priority-0] org.apache.hadoop.mapreduce.JobResourceUploader - Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this. 2018-03-29T14:55:25,785 WARN [task-runner-0-priority-0] org.apache.hadoop.mapreduce.JobResourceUploader - No job jar file set. User classes may not be found. See Job or Job#setJar(String). 2018-03-29T14:55:26,396 INFO [task-runner-0-priority-0] org.apache.hadoop.mapreduce.JobSubmitter - Cleaning up the staging area /user/druid/.staging/job_1522329361898_0001 2018-03-29T14:55:26,406 ERROR [task-runner-0-priority-0] io.druid.indexing.overlord.ThreadPoolTaskRunner - Exception while running task[HadoopIndexTask{id=index_hadoop_wikiticker_2018-03-29T14:55:14.270Z, type=index_hadoop, dataSource=wikiticker}] java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.google.common.base.Throwables.propagate(Throwables.java:160) ~[guava-16.0.1.jar:?] at io.druid.indexing.common.task.HadoopTask.invokeForeignLoader(HadoopTask.java:218) ~[druid-indexing-service-0.10.1.2.6.4.0-91.jar:0.10.1.2.6.4.0-91] at io.druid.indexing.common.task.HadoopIndexTask.run(HadoopIndexTask.java:178) ~[druid-indexing-service-0.10.1.2.6.4.0-91.jar:0.10.1.2.6.4.0-91] at io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:436) [druid-indexing-service-0.10.1.2.6.4.0-91.jar:0.10.1.2.6.4.0-91] at io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:408) [druid-indexing-service-0.10.1.2.6.4.0-91.jar:0.10.1.2.6.4.0-91] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_161] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_161] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_161] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_161] Caused by: java.lang.reflect.InvocationTargetException at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_161] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_161] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_161] at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_161] at io.druid.indexing.common.task.HadoopTask.invokeForeignLoader(HadoopTask.java:215) ~[druid-indexing-service-0.10.1.2.6.4.0-91.jar:0.10.1.2.6.4.0-91] ... 7 more Caused by: java.lang.RuntimeException: org.apache.hadoop.mapreduce.lib.input.InvalidInputException: Input path does not exist: hdfs://sandbox-hdp.hortonworks.com:8020/user/druid/quickstart/wikiticker-2015-09-12-sampled.json.gz at com.google.common.base.Throwables.propagate(Throwables.java:160) ~[guava-16.0.1.jar:?] at io.druid.indexer.DetermineHashedPartitionsJob.run(DetermineHashedPartitionsJob.java:210) ~[druid-indexing-hadoop-0.10.1.2.6.4.0-91.jar:0.10.1.2.6.4.0-91] at io.druid.indexer.JobHelper.runJobs(JobHelper.java:369) ~[druid-indexing-hadoop-0.10.1.2.6.4.0-91.jar:0.10.1.2.6.4.0-91] at io.druid.indexer.HadoopDruidDetermineConfigurationJob.run(HadoopDruidDetermineConfigurationJob.java:91) ~[druid-indexing-hadoop-0.10.1.2.6.4.0-91.jar:0.10.1.2.6.4.0-91] at io.druid.indexing.common.task.HadoopIndexTask$HadoopDetermineConfigInnerProcessing.runTask(HadoopIndexTask.java:308) ~[druid-indexing-service-0.10.1.2.6.4.0-91.jar:0.10.1.2.6.4.0-91] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_161] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_161] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_161] at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_161] at io.druid.indexing.common.task.HadoopTask.invokeForeignLoader(HadoopTask.java:215) ~[druid-indexing-service-0.10.1.2.6.4.0-91.jar:0.10.1.2.6.4.0-91] ... 7 more Caused by: org.apache.hadoop.mapreduce.lib.input.InvalidInputException: Input path does not exist: hdfs://sandbox-hdp.hortonworks.com:8020/user/druid/quickstart/wikiticker-2015-09-12-sampled.json.gz at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.singleThreadedListStatus(FileInputFormat.java:323) ~[?:?] at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.listStatus(FileInputFormat.java:265) ~[?:?] at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.getSplits(FileInputFormat.java:387) ~[?:?] at org.apache.hadoop.mapreduce.lib.input.DelegatingInputFormat.getSplits(DelegatingInputFormat.java:115) ~[?:?] at org.apache.hadoop.mapreduce.JobSubmitter.writeNewSplits(JobSubmitter.java:301) ~[?:?] at org.apache.hadoop.mapreduce.JobSubmitter.writeSplits(JobSubmitter.java:318) ~[?:?] at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:196) ~[?:?] at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1290) ~[?:?] at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1287) ~[?:?] at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_161] at javax.security.auth.Subject.doAs(Subject.java:422) ~[?:1.8.0_161] at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1869) ~[?:?] at org.apache.hadoop.mapreduce.Job.submit(Job.java:1287) ~[?:?] at io.druid.indexer.DetermineHashedPartitionsJob.run(DetermineHashedPartitionsJob.java:118) ~[druid-indexing-hadoop-0.10.1.2.6.4.0-91.jar:0.10.1.2.6.4.0-91] at io.druid.indexer.JobHelper.runJobs(JobHelper.java:369) ~[druid-indexing-hadoop-0.10.1.2.6.4.0-91.jar:0.10.1.2.6.4.0-91] at io.druid.indexer.HadoopDruidDetermineConfigurationJob.run(HadoopDruidDetermineConfigurationJob.java:91) ~[druid-indexing-hadoop-0.10.1.2.6.4.0-91.jar:0.10.1.2.6.4.0-91] at io.druid.indexing.common.task.HadoopIndexTask$HadoopDetermineConfigInnerProcessing.runTask(HadoopIndexTask.java:308) ~[druid-indexing-service-0.10.1.2.6.4.0-91.jar:0.10.1.2.6.4.0-91] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_161] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_161] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_161] at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_161] at io.druid.indexing.common.task.HadoopTask.invokeForeignLoader(HadoopTask.java:215) ~[druid-indexing-service-0.10.1.2.6.4.0-91.jar:0.10.1.2.6.4.0-91] ... 7 more 2018-03-29T14:55:26,421 INFO [task-runner-0-priority-0] io.druid.indexing.overlord.TaskRunnerUtils - Task [index_hadoop_wikiticker_2018-03-29T14:55:14.270Z] status changed to [FAILED]. 2018-03-29T14:55:26,428 INFO [task-runner-0-priority-0] io.druid.indexing.worker.executor.ExecutorLifecycle - Task completed with status: { "id" : "index_hadoop_wikiticker_2018-03-29T14:55:14.270Z", "status" : "FAILED", "duration" : 6396 } 2018-03-29T14:55:26,435 INFO [main] io.druid.java.util.common.lifecycle.Lifecycle$AnnotationBasedHandler - Invoking stop method[public void io.druid.server.listener.announcer.ListenerResourceAnnouncer.stop()] on object[io.druid.query.lookup.LookupResourceListenerAnnouncer@577bfadb]. 2018-03-29T14:55:26,435 INFO [main] io.druid.curator.announcement.Announcer - unannouncing [/druid/listeners/lookups/__default/sandbox-hdp.hortonworks.com:8100] 2018-03-29T14:55:26,452 INFO [main] io.druid.server.listener.announcer.ListenerResourceAnnouncer - Unannouncing start time on [/druid/listeners/lookups/__default/sandbox-hdp.hortonworks.com:8100] 2018-03-29T14:55:26,456 INFO [main] org.eclipse.jetty.server.AbstractConnector - Stopped ServerConnector@4adf3582{HTTP/1.1,[http/1.1]}{0.0.0.0:8100} 2018-03-29T14:55:26,457 INFO [main] org.eclipse.jetty.server.handler.ContextHandler - Stopped o.e.j.s.ServletContextHandler@3f3c8b60{/,null,UNAVAILABLE} 2018-03-29T14:55:26,462 INFO [main] io.druid.java.util.common.lifecycle.Lifecycle$AnnotationBasedHandler - Invoking stop method[public void io.druid.indexing.worker.executor.ExecutorLifecycle.stop() throws java.lang.Exception] on object[io.druid.indexing.worker.executor.ExecutorLifecycle@1b37fbec]. 2018-03-29T14:55:26,463 INFO [main] io.druid.java.util.common.lifecycle.Lifecycle$AnnotationBasedHandler - Invoking stop method[public void io.druid.indexing.overlord.ThreadPoolTaskRunner.stop()] on object[io.druid.indexing.overlord.ThreadPoolTaskRunner@7493d937]. 2018-03-29T14:55:26,463 INFO [main] io.druid.java.util.common.lifecycle.Lifecycle$AnnotationBasedHandler - Invoking stop method[public void io.druid.curator.discovery.ServerDiscoverySelector.stop() throws java.io.IOException] on object[io.druid.curator.discovery.ServerDiscoverySelector@40df6090]. 2018-03-29T14:55:26,466 INFO [main] io.druid.java.util.common.lifecycle.Lifecycle$AnnotationBasedHandler - Invoking stop method[public void io.druid.curator.announcement.Announcer.stop()] on object[io.druid.curator.announcement.Announcer@69796bd0]. 2018-03-29T14:55:26,467 INFO [main] io.druid.java.util.common.lifecycle.Lifecycle$AnnotationBasedHandler - Invoking stop method[public void io.druid.curator.discovery.ServerDiscoverySelector.stop() throws java.io.IOException] on object[io.druid.curator.discovery.ServerDiscoverySelector@44bc2449]. 2018-03-29T14:55:26,467 INFO [main] io.druid.curator.CuratorModule - Stopping Curator 2018-03-29T14:55:26,468 INFO [Curator-Framework-0] org.apache.curator.framework.imps.CuratorFrameworkImpl - backgroundOperationsLoop exiting 2018-03-29T14:55:26,472 INFO [main] org.apache.zookeeper.ZooKeeper - Session: 0x16271e552860016 closed 2018-03-29T14:55:26,472 INFO [main-EventThread] org.apache.zookeeper.ClientCnxn - EventThread shut down 2018-03-29T14:55:26,472 INFO [main] com.metamx.common.lifecycle.Lifecycle$AnnotationBasedHandler - Invoking stop method[public void com.metamx.http.client.NettyHttpClient.stop()] on object[com.metamx.http.client.NettyHttpClient@339f3a55]. 2018-03-29T14:55:26,503 INFO [main] io.druid.java.util.common.lifecycle.Lifecycle$AnnotationBasedHandler - Invoking stop method[public void io.druid.storage.hdfs.HdfsStorageAuthentication.stop()] on object[io.druid.storage.hdfs.HdfsStorageAuthentication@7aea704c]. 2018-03-29T14:55:26,504 INFO [main] io.druid.java.util.common.lifecycle.Lifecycle$AnnotationBasedHandler - Invoking stop method[public void com.metamx.metrics.MonitorScheduler.stop()] on object[com.metamx.metrics.MonitorScheduler@450f0235]. 2018-03-29T14:55:26,504 INFO [main] io.druid.java.util.common.lifecycle.Lifecycle$AnnotationBasedHandler - Invoking stop method[public void io.druid.query.lookup.LookupReferencesManager.stop()] on object[io.druid.query.lookup.LookupReferencesManager@19827608]. 2018-03-29T14:55:26,504 INFO [main] io.druid.query.lookup.LookupReferencesManager - LookupReferencesManager is stopping. 2018-03-29T14:55:26,509 INFO [LookupReferencesManager-MainThread] io.druid.query.lookup.LookupReferencesManager - Lookup Management loop exited, Lookup notices are not handled anymore. 2018-03-29T14:55:26,511 INFO [main] io.druid.query.lookup.LookupReferencesManager - LookupReferencesManager is stopped. 2018-03-29T14:55:26,511 INFO [main] io.druid.java.util.common.lifecycle.Lifecycle$AnnotationBasedHandler - Invoking stop method[public void com.metamx.emitter.service.ServiceEmitter.close() throws java.io.IOException] on object[com.metamx.emitter.service.ServiceEmitter@5b2b8d86]. 2018-03-29T14:55:26,511 INFO [main] io.druid.java.util.common.lifecycle.Lifecycle$AnnotationBasedHandler - Invoking stop method[public void io.druid.initialization.Log4jShutterDownerModule$Log4jShutterDowner.stop()] on object[io.druid.initialization.Log4jShutterDownerModule$Log4jShutterDowner@4f081b5d]. 2018-03-29 14:55:26,579 pool-1-thread-1 ERROR Unable to register shutdown hook because JVM is shutting down. java.lang.IllegalStateException: Not started at io.druid.common.config.Log4jShutdown.addShutdownCallback(Log4jShutdown.java:46) at org.apache.logging.log4j.core.impl.Log4jContextFactory.addShutdownCallback(Log4jContextFactory.java:273) at org.apache.logging.log4j.core.LoggerContext.setUpShutdownHook(LoggerContext.java:256) at org.apache.logging.log4j.core.LoggerContext.start(LoggerContext.java:216) at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:145) at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:41) at org.apache.logging.log4j.LogManager.getContext(LogManager.java:182) at org.apache.logging.log4j.spi.AbstractLoggerAdapter.getContext(AbstractLoggerAdapter.java:103) at org.apache.logging.slf4j.Log4jLoggerFactory.getContext(Log4jLoggerFactory.java:43) at org.apache.logging.log4j.spi.AbstractLoggerAdapter.getLogger(AbstractLoggerAdapter.java:42) at org.apache.logging.slf4j.Log4jLoggerFactory.getLogger(Log4jLoggerFactory.java:29) at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:253) at org.apache.commons.logging.impl.SLF4JLogFactory.getInstance(SLF4JLogFactory.java:155) at org.apache.commons.logging.impl.SLF4JLogFactory.getInstance(SLF4JLogFactory.java:132) at org.apache.commons.logging.LogFactory.getLog(LogFactory.java:273) at org.apache.hadoop.hdfs.LeaseRenewer.<clinit>(LeaseRenewer.java:72) at org.apache.hadoop.hdfs.DFSClient.getLeaseRenewer(DFSClient.java:830) at org.apache.hadoop.hdfs.DFSClient.close(DFSClient.java:968) at org.apache.hadoop.hdfs.DistributedFileSystem.close(DistributedFileSystem.java:1214) at org.apache.hadoop.fs.FileSystem$Cache.closeAll(FileSystem.java:2886) at org.apache.hadoop.fs.FileSystem$Cache$ClientFinalizer.run(FileSystem.java:2903) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) I will try to find the potential solution. I noticed other users using Druid on HDP 2.6 were having issues.

jmedel · ‎07-27-2017

Hello Wendy, VMware should be configured by default, so when you import the sandbox, the sandbox image will load into the VMware manager and you can power the sandbox virtual machine on. Have you had a chance to look at our tutorial on deploying hdp sb on vmware? https://hortonworks.com/tutorial/sandbox-deployment-and-install-guide/section/2/

jmedel · ‎03-11-2017

Hello @Josh Elser, Thank you for your prompt response. I executed instruction: echo ruok | nc sandbox-hdp.hortonworks.com 2181 I received message: "imok" I believe it is not a firewall issue because I tried telnet <hostname-hdp> 2181 against HDP sandbox from HDF sandbox. I also tried telnet <hostname-hdf> 2181 against HDF sandbox from HDP sandbox. The output message I get from console illustrates HDF is connected to zookeeper port running on HDP: [root@sandbox ~]# telnet 172.17.0.3 2181 Trying 172.17.0.3... Connected to 172.17.0.3. Escape character is '^]'. Note: 172.17.0.3 is the ip address of HDP sandbox. What approach would you take to debug a port forwarding issue? Thanks, James

jmedel · ‎03-11-2017

Description of my setup: I have HDF sandbox node-A running NiFi while HDP sandbox node-B runs HBase. Both sandboxes have zookeper activated. In NiFi, I setup HBaseClientService for the PutHbaseJson processor. I added hbase-site.xml to my HBaseClientService under the Hadoop Configuration Files property.hbase-clientservice-error.pnghbase-config-hdp-setup.png Originally I acquired hbase-site.xml from HDP sandbox in path "/var/lib/ambari-server/resources/common-services/HBASE/0.96.0.2.0/configuration/hbase-site.xml". I modified hbase-site.xml: hbase.rootdir, hbase.zookeeper.quorum and added hbase.regionserver.port, so NiFi knows where to find HBase. I have narrowed the problem down to a configuration problem either on NiFi or HBase side and not a networking issue because I have successfully tested that NiFi can dump data into HDFS using the PutHDFS processor. The Problem: A snippet of the stack trace from the nifi-app.log points out that hbase.MasterNotRunningException and can't get connection to Zookeeper. So, I conclude the problem is coming from Zookeeper. I don't understand why this problem is happening because I included both NiFi's and HBase's IP address for the zookeeper.quorum property value in the hbase-site.xml. 2017-03-11 16:18:22,066 ERROR [StandardProcessScheduler Thread-1] o.a.n.c.s.StandardControllerServiceNode HBase_1_1_2_ClientService[id=4d4142fa-0159-1000-a996-9604038fb2af] Failed to invoke @OnEnabled method due to org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after attempts=1, exceptions: Sat Mar 11 16:18:22 UTC 2017, RpcRetryingCaller{globalStartTime=1489249087052, pause=100, retries=1}, org.apache.hadoop.hbase.MasterNotRunningException: org.apache.hadoop.hbase.MasterNotRunningException: Can't get connection to ZooKeeper: KeeperErrorCode = OperationTimeout Update March 11, 2017: I found other interesting insights from HDF and HDP sandbox logs HDF sandbox logs: NiFi logs: The NiFi logs indicate there is a connection problem with Zookeeper. So, I go to check zookeeper logs. o.a.h.h.zookeeper.RecoverableZooKeeper Unable to create ZooKeeper Connectionjava.net.UnknownHostException: sandbox.hortonworks.com: Name or service not knowno.a.h.hbase.zookeeper.ZooKeeperWatcher hconnection-0x30c93240x0, quorum=sandbox-hdp.hortonworks.com:2181, sandbox.hortonworks.com:2181, baseZNode=/hbase-unsecure Received unexpected KeeperException, re-throwing exceptionCaused by: org.apache.hadoop.hbase.MasterNotRunningException: org.apache.hadoop.hbase.MasterNotRunningException: Can't get connection to ZooKeeper: KeeperErrorCode = OperationTimeoutCaused by: org.apache.hadoop.hbase.MasterNotRunningException: Can't get connection to ZooKeeper: KeeperErrorCode = OperationTimeoutCaused by: org.apache.zookeeper.KeeperException$OperationTimeoutException: KeeperErrorCode = OperationTimeout Zookeeper logs: Zookeeper logs show there is no node for Primary Node and Cluster Coordinate. Also I see the client closed the socket, so it can’t read any more data from the client. 2017-03-11 19:55:49,257 - INFO [ProcessThread(sid:0 cport:-1)::PrepRequestProcessor@645] - Got user-level KeeperException when processing sessionid:0x15abb279ca20026 type:delete cxid:0x8 zxid:0x22f5 txntype:-1 reqpath:n/a Error Path:/nifi/leaders/Primary Node/_c_f44978a2-7e01-487d-878b-d99d79fee7a9-lock-0000000005 Error:KeeperErrorCode = NoNode for /nifi/leaders/Primary Node/_c_f44978a2-7e01-487d-878b-d99d79fee7a9-lock-00000000052017-03-11 19:55:49,257 - INFO [ProcessThread(sid:0 cport:-1)::PrepRequestProcessor@645] - Got user-level KeeperException when processing sessionid:0x15abb279ca20026 type:delete cxid:0x9 zxid:0x22f6 txntype:-1 reqpath:n/a Error Path:/nifi/leaders/Cluster Coordinator/_c_17d97f01-625d-459b-bcb9-850d98da42a6-lock-0000000005 Error:KeeperErrorCode = NoNode for /nifi/leaders/Cluster Coordinator/_c_17d97f01-625d-459b-bcb9-850d98da42a6-lock-0000000005EndOfStreamException: Unable to read additional data from client sessionid 0x15abb279ca20035, likely client has closed socket at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:230) at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208) at java.lang.Thread.run(Thread.java:745) Ambari Infra Solr logs: Solr logs indicate a config file of Zookeeper does not exist. Solr also cannot talk to zookeeper.znode.parent. Caused by: org.apache.solr.common.SolrException: Unable to create core [hadoop_logs_shard1_replica1]Caused by: org.apache.solr.common.cloud.ZooKeeperException: Specified config does not exist in ZooKeeper: hadoop_logs2017-03-11 21:35:06,404 [OverseerCollectionConfigSetProcessor-97596120747868221-172.17.0.2:8886_solr-n_0000000000] WARN [ ] org.apache.solr.cloud.OverseerTaskProcessor (OverseerTaskProcessor.java:249) - Overseer cannot talk to ZK2017-03-11 21:35:06,409 [OverseerHdfsCoreFailoverThread-97596120747868221-172.17.0.2:8886_solr-n_0000000000] ERROR [ ] org.apache.solr.common.SolrException (SolrException.java:159) - OverseerAutoReplicaFailoverThread had an error in its thread work loop.:org.apache.solr.common.SolrException: Error reading cluster properties HDP sandbox logs: HBase logs: HBase logs show there is a server refused connection on solr’s side. WARN [timeline] timeline.HadoopTimelineMetricsSink: Unable to send metrics to collector by address:http://sandbox.hortonworks.com:6188/ws/v1/timeline/metricsERROR [org.apache.ranger.audit.queue.AuditBatchQueue0] impl.CloudSolrClient: Request to collection ranger_audits failed due to (0) java.net.ConnectException: Connection refused (Connection refused), retry? 0Caused by: org.apache.solr.client.solrj.SolrServerException: No live SolrServers available to handle this request:[http://172.17.0.2:8886/solr/ranger_audits_shard1_replica1]Caused by: org.apache.solr.client.solrj.SolrServerException: Server refused connection at: http://172.17.0.2:8886/solr/ranger_audits_shard1_replica1Caused by: java.net.ConnectException: Connection refused (Connection refused) Zookeeper logs: Zookeeper logs illustrate there is an invalid configuration and that only one server can be specified, it looks like this issue is related to Zookeeper Quorum. My question is then, when you have multiple zookeeper services running in your cluster, don’t you have to specify their <IP-address> or hostname in the hbase-site.xml on both sandboxes? 2017-03-11 20:07:57,177 - INFO [main:QuorumPeerConfig@103] - Reading configuration from: /usr/hdp/current/zookeeper-server/conf/zoo.cfg2017-03-11 20:07:57,178 - ERROR [main:QuorumPeerConfig@287] - Invalid configuration, only one server specified (ignoring)2017-03-11 21:01:16,208 - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@357] - caught end of stream exceptionEndOfStreamException: Unable to read additional data from client sessionid 0x15abefc1ba0000d, likely client has closed socket at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228) at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208) at java.lang.Thread.run(Thread.java:745)2017-03-11 21:07:27,068 - INFO [ProcessThread(sid:0 cport:-1)::PrepRequestProcessor@645] - Got user-level KeeperException when processing sessionid:0x15abefc1ba00011 type:create cxid:0x1 zxid:0xdbe txntype:-1 reqpath:n/a Error Path:/consumers Error:KeeperErrorCode = NodeExists for /consumers2017-03-11 21:07:27,068 - INFO [ProcessThread(sid:0 cport:-1)::PrepRequestProcessor@645] - Got user-level KeeperException when processing sessionid:0x15abefc1ba00011 type:create cxid:0x2 zxid:0xdbf txntype:-1 reqpath:n/a Error Path:/consumers/ranger_entities_consumer Error:KeeperErrorCode = NodeExists for /consumers/ranger_entities_consumer2017-03-11 21:07:27,068 - INFO [ProcessThread(sid:0 cport:-1)::PrepRequestProcessor@645] - Got user-level KeeperException when processing sessionid:0x15abefc1ba00011 type:create cxid:0x3 zxid:0xdc0 txntype:-1 reqpath:n/a Error Path:/consumers/ranger_entities_consumer/ids Error:KeeperErrorCode = NodeExists for /consumers/ranger_entities_consumer/ids Ambari Infra Solr logs: Solr logs indicate that it could not read the data. I believe this log message refers to solr not being able to read the data coming in from HDF sandbox. [OverseerExitThread] ERROR [ ] org.apache.solr.cloud.Overseer$ClusterStateUpdater (Overseer.java:311) - could not read the dataorg.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader2017-03-11 21:35:06,630 [OverseerHdfsCoreFailoverThread-97600331946393615-172.17.0.3:8886_solr-n_0000000001] ERROR [ ] org.apache.solr.common.SolrException (SolrException.java:159) - OverseerAutoReplicaFailoverThread had an error in its thread work loop.:org.apache.solr.common.SolrException: Error reading cluster propertiesCaused by: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /clusterprops.json Any help for resolving this issue would be greatly appreciated. Thank you. Regards, James hbase-site.xml nifi-app.txt zookeeper-zookeeper-server-sandboxhortonworkscom.txt

jmedel · ‎01-26-2017

My objective: Use NiFi running on an HDF docker container to store data into HBase running on an HDP docker container. Progress: I am running two docker containers, an HDF 2.1.1 container and HDP 2.5 (including HBase) on another container. I have configured the PutHBaseJSON processor to write data to HBase (puthbasejson_configuration.png). I also configured the NiFi's hbase client service on that processor, so NiFi knows which IP address Zookeeper is located at to ask Zookeeper to tell it where HBase Master is (hbaseclientservice_configuration.png). Problem: The problem I am facing is that NiFi is unable to make the connection to HBase Master. I get the following failed to invoke "@OnEnabled method due to ... hbase.client.RetriesExhausted Exception ... hbase.MasterNotRunningException ... java.net.ConnectException: Connection refused." Visual of hbaseclientservice at (hbaseMasterNotRunningException Stack Trace). Configurations I made to troubleshoot the problem: In HDF container, I updated /etc/hosts with 172.25.0.3 -> hdp.hortonworks.com. In HDP container, I updated the hosts file with 172.25.0.2 -> hdf.hortonworks.com. So both containers are aware of eachothers hostnames. I port forwarded the needed ports for NiFi, Zookeeper and HBase when I built the HDF and HDP containers. I checked if all ports on HBase were exposed on the HDP container and the image shows all the ports HDP is listening in on including HBase's ports (ports_hdp_listening_on.png). Here is an image of all ports needed by HBase, I filtered for port keyword in Ambari (hbase_ports_needed.png). 16000 and 16020 ports both looked suspicious since all others had pattern :::port but those two ports had some wording preceding it. So, I checked if I could make the connection to HDP from HDF using telnet 172.25.0.3 16000 and received the output: Trying 172.25.0.3... Connected to 172.25.0.3. Escape character is '^]'. So I was able to connect to HDP container. puthbasejson-configuration.png hbaseclientservice-configuration.png hbase-client-service-log-message.png ports-hdp-listening-on.png hbase-ports-needed.png hbaseMasterNotRunningException Stack Trace: 2017-01-25 22:23:03,342 ERROR [StandardProcessScheduler Thread-7] o.a.n.c.s.StandardControllerServiceNode HBase_1_1_2_ClientService[id=d3eaf393-0159-1000-ffff-ffffa95f1940] Failed to invoke @OnEnabled method due to org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after attempts=1, exceptions: Wed Jan 25 22:23:03 UTC 2017, RpcRetryingCaller{globalStartTime=1485382983338, pause=100, retries=1}, org.apache.hadoop.hbase.MasterNotRunningException: com.google.protobuf.ServiceException: java.net.ConnectException: Connection refused 2017-01-25 22:23:03,348 ERROR [StandardProcessScheduler Thread-7] o.a.n.c.s.StandardControllerServiceNode org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after attempts=1, exceptions: Wed Jan 25 22:23:03 UTC 2017, RpcRetryingCaller{globalStartTime=1485382983338, pause=100, retries=1}, org.apache.hadoop.hbase.MasterNotRunningException: com.google.protobuf.ServiceException: java.net.ConnectException: Connection refused at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:147) ~[na:na] at org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:3917) ~[na:na] at org.apache.hadoop.hbase.client.HBaseAdmin.listTableNames(HBaseAdmin.java:413) ~[na:na] at org.apache.hadoop.hbase.client.HBaseAdmin.listTableNames(HBaseAdmin.java:397) ~[na:na] at org.apache.nifi.hbase.HBase_1_1_2_ClientService.onEnabled(HBase_1_1_2_ClientService.java:187) ~[na:na] at sun.reflect.GeneratedMethodAccessor568.invoke(Unknown Source) ~[na:na] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_111] at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_111] at org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:137) ~[na:na] at org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:125) ~[na:na] at org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:70) ~[na:na] at org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotation(ReflectionUtils.java:47) ~[na:na] at org.apache.nifi.controller.service.StandardControllerServiceNode$2.run(StandardControllerServiceNode.java:345) ~[na:na] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_111] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_111] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [na:1.8.0_111] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [na:1.8.0_111] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_111] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_111] at java.lang.Thread.run(Thread.java:745) [na:1.8.0_111] Caused by: org.apache.hadoop.hbase.MasterNotRunningException: com.google.protobuf.ServiceException: java.net.ConnectException: Connection refused at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$StubMaker.makeStub(ConnectionManager.java:1533) ~[na:na] at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$MasterServiceStubMaker.makeStub(ConnectionManager.java:1553) ~[na:na] at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.getKeepAliveMasterService(ConnectionManager.java:1704) ~[na:na] at org.apache.hadoop.hbase.client.MasterCallable.prepare(MasterCallable.java:38) ~[na:na] at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:124) ~[na:na] ... 19 common frames omitted Caused by: com.google.protobuf.ServiceException: java.net.ConnectException: Connection refused at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:223) ~[na:na] at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:287) ~[na:na] at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$BlockingStub.isMasterRunning(MasterProtos.java:50918) ~[na:na] at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$MasterServiceStubMaker.isMasterRunning(ConnectionManager.java:1564) ~[na:na] at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$StubMaker.makeStubNoRetries(ConnectionManager.java:1502) ~[na:na] at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$StubMaker.makeStub(ConnectionManager.java:1524) ~[na:na] ... 23 common frames omitted Caused by: java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[na:1.8.0_111] at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) ~[na:1.8.0_111] at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206) ~[na:na] at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531) ~[na:na] at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:495) ~[na:na] at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.setupConnection(RpcClientImpl.java:424) ~[na:na] at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.setupIOstreams(RpcClientImpl.java:748) ~[na:na] at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.writeRequest(RpcClientImpl.java:920) ~[na:na] at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.tracedWriteRequest(RpcClientImpl.java:889) ~[na:na] at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1222) ~[na:na] at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:213) ~[na:na] ... 28 common frames omitted 2017-01-25 22:23:03,348 ERROR [StandardProcessScheduler Thread-7] o.a.n.c.s.StandardControllerServiceNode Failed to invoke @OnEnabled method of HBase_1_1_2_ClientService[id=d3eaf393-0159-1000-ffff-ffffa95f1940] due to org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after attempts=1, exceptions: Wed Jan 25 22:23:03 UTC 2017, RpcRetryingCaller{globalStartTime=1485382983338, pause=100, retries=1}, org.apache.hadoop.hbase.MasterNotRunningException: com.google.protobuf.ServiceException: java.net.ConnectException: Connection refused I am currently still dealing with problem: Has anyone setup NiFi HDF docker container to store data into HBase HDP docker container?

jmedel · ‎12-29-2016

The RPi works fine. I unplugged RPi. Yes there are more logs in my "messages" system logs. Stack Trace: Dec 29 19:17:09 raspberrypi kernel: [ 5.677122] IPv6: ADDRCONF(NETDEV_UP): wlan0: link is not ready Dec 29 19:17:09 raspberrypi kernel: [ 5.677131] brcmfmac: power management disabled Dec 29 19:17:09 raspberrypi kernel: [ 5.783063] uart-pl011 3f201000.uart: no DMA platform data Dec 29 19:17:09 raspberrypi kernel: [ 5.908412] random: nonblocking pool is initialized Dec 29 19:17:09 raspberrypi kernel: [ 6.077762] Adding 102396k swap on /var/swap. Priority:-1 extents:2 across:217084k SSFS Dec 29 19:17:09 raspberrypi kernel: [ 6.130497] cfg80211: Regulatory domain changed to country: GB Dec 29 19:17:09 raspberrypi kernel: [ 6.130512] cfg80211: DFS Master region: ETSI Dec 29 19:17:09 raspberrypi kernel: [ 6.130518] cfg80211: (start_freq - end_freq @ bandwidth), (max_antenna_gain, max_eirp), (dfs_cac_time) Dec 29 19:17:09 raspberrypi kernel: [ 6.130526] cfg80211: (2402000 KHz - 2482000 KHz @ 40000 KHz), (N/A, 2000 mBm), (N/A) Dec 29 19:17:09 raspberrypi kernel: [ 6.130534] cfg80211: (5170000 KHz - 5250000 KHz @ 80000 KHz, 160000 KHz AUTO), (N/A, 2000 mBm), (N/A) Dec 29 19:17:09 raspberrypi kernel: [ 6.130541] cfg80211: (5250000 KHz - 5330000 KHz @ 80000 KHz, 160000 KHz AUTO), (N/A, 2000 mBm), (0 s) Dec 29 19:17:09 raspberrypi kernel: [ 6.130548] cfg80211: (5490000 KHz - 5710000 KHz @ 160000 KHz), (N/A, 2700 mBm), (0 s) Dec 29 19:17:09 raspberrypi kernel: [ 6.130554] cfg80211: (57000000 KHz - 66000000 KHz @ 2160000 KHz), (N/A, 4000 mBm), (N/A) Dec 29 19:17:09 raspberrypi kernel: [ 6.419695] smsc95xx 1-1.1:1.0 eth0: hardware isn't capable of remote wakeup Dec 29 19:17:09 raspberrypi kernel: [ 6.419983] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready Dec 29 19:17:13 raspberrypi kernel: [ 10.197345] Bluetooth: Core ver 2.21 Dec 29 19:17:13 raspberrypi kernel: [ 10.197431] NET: Registered protocol family 31 Dec 29 19:17:13 raspberrypi kernel: [ 10.197442] Bluetooth: HCI device and connection manager initialized Dec 29 19:17:13 raspberrypi kernel: [ 10.197464] Bluetooth: HCI socket layer initialized Dec 29 19:17:13 raspberrypi kernel: [ 10.197481] Bluetooth: L2CAP socket layer initialized Dec 29 19:17:13 raspberrypi kernel: [ 10.197538] Bluetooth: SCO socket layer initialized Dec 29 19:17:13 raspberrypi kernel: [ 10.205632] Bluetooth: HCI UART driver ver 2.3 Dec 29 19:17:13 raspberrypi kernel: [ 10.205656] Bluetooth: HCI UART protocol H4 registered Dec 29 19:17:13 raspberrypi kernel: [ 10.205668] Bluetooth: HCI UART protocol Three-wire (H5) registered Dec 29 19:17:13 raspberrypi kernel: [ 10.205845] Bluetooth: HCI UART protocol BCM registered Dec 29 19:17:13 raspberrypi kernel: [ 10.411407] Bluetooth: BNEP (Ethernet Emulation) ver 1.3 Dec 29 19:17:13 raspberrypi kernel: [ 10.411427] Bluetooth: BNEP filters: protocol multicast Dec 29 19:17:13 raspberrypi kernel: [ 10.411452] Bluetooth: BNEP socket layer initialized Dec 29 19:17:14 raspberrypi kernel: [ 10.694791] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready Dec 29 19:17:14 raspberrypi kernel: [ 10.696931] smsc95xx 1-1.1:1.0 eth0: link up, 100Mbps, full-duplex, lpa 0xCDE1

jmedel · ‎12-29-2016

Hello. I am new to MiNiFi. I created MiNiFi flow that executes a python script to ingest raw sensor data using the ExecuteProcess processor and sends that data to my nifi core using a Remote Process Group. The flow runs fine the first time I install MiNiFi on the Raspberry Pi while it is powered and run the flow. I am able to the data transfered to the NiFi core running on my HDP sandbox. The "input port" name on NiFi core matches the particular connection name for the Remote Process Group on MiNiFi. However, when the Raspberry Pi is turned off and then back on, the flow no longer works. I look in the logs of MiNiFi and receive the following message: java.lang.Exception: Unable to load flow due to: java.io.IOException: Expected to read a Sentinel Byte of '1' but got a value of '0' instead at org.apache.nifi.minifi.MiNiFiServer.start(MiNiFiServer.java:125) at org.apache.nifi.minifi.MiNiFi.<init>(MiNiFi.java:134) at org.apache.nifi.minifi.MiNiFi.main(MiNiFi.java:233) Caused by: java.io.IOException: Expected to read a Sentinel Byte of '1' but got a value of '0' instead at org.apache.nifi.repository.schema.SchemaRecordReader.readRecord(SchemaRecordReader.java:65) at org.apache.nifi.controller.repository.SchemaRepositoryRecordSerde.deserializeRecord(SchemaRepositoryRecordSerde.java:111) at org.apache.nifi.controller.repository.SchemaRepositoryRecordSerde.deserializeEdit(SchemaRepositoryRecordSerde.java:105) at org.apache.nifi.controller.repository.SchemaRepositoryRecordSerde.deserializeEdit(SchemaRepositoryRecordSerde.java:46) at org.wali.MinimalLockingWriteAheadLog$Partition.recoverNextTransaction(MinimalLockingWriteAheadLog.java:1072) at org.wali.MinimalLockingWriteAheadLog.recoverFromEdits(MinimalLockingWriteAheadLog.java:442) at org.wali.MinimalLockingWriteAheadLog.recoverRecords(MinimalLockingWriteAheadLog.java:301) at org.apache.nifi.controller.repository.WriteAheadFlowFileRepository.loadFlowFiles(WriteAheadFlowFileRepository.java:346) at org.apache.nifi.controller.FlowController.initializeFlow(FlowController.java:700) at org.apache.nifi.controller.StandardFlowService.loadFromBytes(StandardFlowService.java:701) at org.apache.nifi.controller.StandardFlowService.load(StandardFlowService.java:508) at org.apache.nifi.minifi.MiNiFiServer.start(MiNiFiServer.java:113) ... 2 common frames omitted 2016-12-29 18:00:34,221 INFO [Thread-1] org.apache.nifi.minifi.MiNiFi Initiating shutdown of MiNiFi server... 2016-12-29 18:00:34,224 ERROR [Thread-1] org.apache.nifi.BootstrapListener Failed to communicate with Bootstrap. Bootstrap may be unable to issue or receive commands from MiNiFi 2016-12-29 18:00:34,224 INFO [Thread-1] org.apache.nifi.minifi.MiNiFi MiNiFi server shutdown completed (nicely or otherwise). Any thoughts on why this is happening? Here is my config.yml for the MiNiFi flow.configyml.txt Thank you. Regards, James

jmedel · ‎10-07-2016

Hello Rashmi, Can you let me know the version of HDP sandbox using "sandbox-version" command. I am running: [root@sandbox ~]# sandbox-version Sandbox information: Created on: 30_09_2016_09_43_28 for Hadoop stack version: Hadoop 2.7.3.2.5.0.0-1245 Ambari Version: 2.4.0.0-1225 Ambari Hash: 59175b7aa1ddb74b85551c632e3ce42fed8f0c85 Ambari build: Release : 1225 Java version: 1.8.0_101 OS Version: CentOS release 6.8 (Final) With the sandbox-version above, I am able to successfully run the topology. Do you have the same version or newer build? @Rashmi

jmedel · ‎10-05-2016

Two questions: 1. What version of the sandbox are you running? With the latest sandbox you should not get the kafkaSpout errors. 2. Is your HBase service up and running? HBase Master, RegionServer Usually HBase tends to go down, so I would suggest deactivating services you don't need such as Flume, Oozie, Zeppelin

jmedel · ‎07-23-2016

Hi @Srikanth Gundam, I tested your NiFi template on my local machine and the dataflow works fine. No problems. "connectionTimeout error" occurs usually due to network connectivity issues or firewall could be dropping the API HTTP request. I don't think you have a network connectivity issue because you were able to use SOAP UI. Can you send me the logs after you ran your NiFi Dataflow and the error happened? The logs folder is in HDF-1.2.0.1-1/nifi/logs.

jmedel · ‎07-21-2016

Hi @Srikanth Gundam Can you share your NiFi template, so I can look through your dataflow and help you find the solution?

jmedel · ‎07-21-2016

@Dmitry Kondratiev Can you run the same command with -e option. This option will give us more details of the error. Can you post the details you receive if maven still fails to build the simulator? mvn clean package -e. What is the sandbox and maven version you are using?

jmedel · ‎07-21-2016

Hi @Harish Padmaraju, I updated the install_maven.sh script, so now it downloads the most up to date version of maven from the maven site. Here is a link to lab 0. Let me know if you come across any other issues. http://hortonworks.com/hadoop-tutorial/realtime-event-processing-nifi-kafka-storm/#section_3

jmedel · ‎07-08-2016

@mgregory Thank you for your feedback. The typos have been addressed. View the updated tutorial: http://hortonworks.com/hadoop-tutorial/learning-ropes-apache-nifi#section_4

jmedel · ‎07-08-2016

Tutorial has been updated to solve issue 163. Here is the updated tutorial: https://hortonworks.com/hadoop-tutorial/learning-ropes-apache-nifi/#section_4 @slachterman @dpelaezpri

jmedel · ‎05-17-2016

Thank you Pierre!

jmedel · ‎05-17-2016

I figured out the solution to the issue. SplitXml splits the children of the parent element in XML file into separate flowfiles. When that happens, the flowfiles receive duplicate names cause the children share the same filename as the original XML file. So, I used a updateAttribute processor to give each flowfile a distinct name. Then when the flowfiles were routed to mergeContent processor, I looked at the list queue to check if the flowfiles all had distinct names and they did. I checked the directory the putfile stored the flowfiles into and they were present as expected. Thank you for the quick response! Here is an image of my updated dataflow:

jmedel · ‎05-17-2016

I almost done creating the DataFlow that ingests vehicle location data from traffic stream simulator, does some filtering on that data and lands the data into a new JSON file. The problem occurs with the mergeContent processor. It shows an error of "duplicate entry." I set UnpackContent's and MergeContent's appropriate property value to zip. Why do I receive this error? Here is an image of the NiFi DataFlow with the error message: GetFile -> UnpackContent -> ControlRate -> EvaluateXPath -> SplitXml -> EvaluateXPath -> RouteOnAttribute -> AttributesToJSON -> MergeContent -> PutFile

jmedel · ‎04-12-2016

Yes. It turns out that my stream-simulator.jar file does not contain the TruckEventProcessorKafkaTopology class. Is there a way to add that class to my jar file?

jmedel · ‎04-12-2016

The stream-simulator.jar file is in the target directory. How do I check if the class or the path to the class exists in the stream-simulator.jar file?

jmedel · ‎04-12-2016

The topology I am trying to submit is TruckEventProcessorKafkaTopology and it comes from the stream simulator project on github: https://github.com/ZacBlanco/iot-truck-streaming/tree/master/storm-streaming/src/main/java/com/hortonworks/streaming/impl/topologies. Steps to reproduce: I start and make sure storm has the appropriate configurations. I verified kafka is running. I modified my pom.xml with the latest versions of storm.version and storm.kafka.version. I ran mvn clean package. I execute the following command: storm jar target/stream-simulator.jar com.hortonworks.streaming.impl.topologies.TruckEventProcessorKafkaTopology The problem is the topology fails to submit. I am using the following tutorial, http://hortonworks.com/hadoop-tutorial/simulating-transporting-realtime-events-stream-apache-kafka/#create-storm-topology-lab2, as a reference. I have already gone through lab1 to configure kafka Why do I receive "Error: Could not find or load main class"? Can someone help guide me to the solution? Thank you. Highly appreciate any help!

jmedel · ‎04-09-2016

Thank you! it worked! sandbox.hortonworks.com:6667 putkafka-success-run.png Is the write for PutKafka suppose to be 0 bytes? Thanks.

Online	Offline
Last Visited	‎07-19-2018 02:00 AM

Date Registered	‎03-07-2016 10:06 PM
Last Visited	‎07-19-2018 02:00 AM
Total Messages Posted	37
Total Kudos Received	12

Re: Error while installing maven

Re: Why does MergeContent have a "duplicate entry"...

Re: Problem in Nifi-MiNifi Flow -- /usr/bin/python...

Re: Problem in Nifi-MiNifi Flow -- /usr/bin/python...

Re: Problem in Nifi-MiNifi Flow -- /usr/bin/python...

Re: Convert Json to Avro processor -- Failed to Pa...

Re: Cannot Run Druid Quickstart Wikiticker Demo on...

Re: Failing to Submit Index Task to Druid's Overlo...

Re: Failing to Submit Index Task to Druid's Overlo...

Failing to Submit Index Task to Druid's Overlord v...

Cannot Run Druid Quickstart Wikiticker Demo on HDP...

Re: Network Configuration for HDP 2.6 Sandbox on V...

Re: Can't Connect to HBase through NiFi HBaseClien...

Can't Connect to HBase through NiFi HBaseClientSer...

Can I connect HDF NiFi docker container to a HDP H...

Re: Why does my MiNiFi Flow fail to run when turni...

Why does my MiNiFi Flow fail to run when turning o...

Re: Hi. Am trying to run the hdp25experiment on HD...

Re: Hi. Am trying to run the hdp25experiment on HD...

Re: Unable to connect to Google Places API using N...

Re: Unable to connect to Google Places API using N...

Re: Maven fails: REALTIME EVENT PROCESSING IN HADO...

Re: Error while installing maven

Re: Typo in Lab1

Re: File trafficLocs_data_for_simulator.zip for la...

Re: Why does MergeContent have a "duplicate entry"...

Re: Why does MergeContent have a "duplicate entry"...

Why does MergeContent have a "duplicate entry" err...

Re: Why do I receive "Error: Could not find or loa...

Re: Why do I receive "Error: Could not find or loa...

Why do I receive "Error: Could not find or load ma...

Re: NiFi - PutKafka Error "failed while waiting fo...