Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Failing to Submit Index Task to Druid's Overlord via REST Service on HDP 2.6.4

Solved Go to solution

Failing to Submit Index Task to Druid's Overlord via REST Service on HDP 2.6.4

Contributor

I am developing an HDP sandbox tutorial that teaches users how they can use Druid's Batch Ingestion Method to load local files into Druid. After I store the USGS data into a local file and submit an ingestion spec referring to the local file to Druid Overlord, I get that the task fails meaning the data didn't load into Druid.

Here is my batch ingestion spec:

{
    "type" : "index",
    "spec" : {
      "dataSchema" : {
        "dataSource" : "usgs",
        "parser" : {
          "type" : "string",
          "parseSpec" : {
            "format" : "tsv",
            "timestampSpec" : {
              "column" : "dates",
              "format" : "auto"
            },
            "dimensionsSpec" : {
              "dimensions": ["staid","val","dates"],
              "dimensionExclusions" : [],
              "spatialDimensions" : []
            }
          }
        },
        "metricsSpec" : [
          {
            "type" : "count",
            "name" : "count"
          },
          {
            "type" : "doubleSum",
            "name" : "avgFlowCuFtsec",
            "fieldName" : "val"
          }
        ],
        "granularitySpec" : {
          "type" : "uniform",
          "segmentGranularity" : "MONTH",
          "queryGranularity" : "NONE",
          "intervals" : [ "1963-01-01/2013-12-31" ]
        }
      },
      "ioConfig" : {
        "type" : "index",
        "firehose" : {
          "type" : "local",
          "baseDir" : "/tmp/usgs/",
          "filter" : "napa-flow.tsv"
        }
      },
      "tuningConfig" : {
        "type": "index",
        "targetPartitionSize" : 10000,
        "maxRowsInMemory" : 75000
      }
    }
  }

Here is the stack trace from Druid's Overlord:

2018-03-29T14:09:28,013 INFO [main] io.druid.guice.PropertiesModule - Loading properties from common.runtime.properties
2018-03-29T14:09:28,015 INFO [main] io.druid.guice.PropertiesModule - Loading properties from runtime.properties
2018-03-29T14:09:28,042 INFO [main] org.hibernate.validator.internal.util.Version - HV000001: Hibernate Validator 5.1.3.Final
2018-03-29T14:09:28,411 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.segment.NullValueHandlingConfig] from props[druid.null.handling.] as [io.druid.segment.NullValueHandlingConfig@6995bf68]
2018-03-29T14:09:28,460 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.guice.ExtensionsConfig] from props[druid.extensions.] as [ExtensionsConfig{searchCurrentClassloader=true, directory='/usr/hdp/current/druid-middlemanager/extensions', hadoopDependenciesDir='/usr/hdp/current/druid-middlemanager/hadoop-dependencies', hadoopContainerDruidClasspath='null', addExtensionsToHadoopContainer=false, loadList=[druid-datasketches, druid-hdfs-storage, druid-kafka-indexing-service]}]
2018-03-29T14:09:28,510 INFO [main] io.druid.initialization.Initialization - Loading extension [druid-datasketches] for class [interface io.druid.cli.CliCommandCreator]
2018-03-29T14:09:28,518 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-datasketches/druid-datasketches-0.10.1.2.6.4.0-91.jar]
2018-03-29T14:09:28,518 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-datasketches/sketches-core-0.8.4.jar]
2018-03-29T14:09:28,518 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-datasketches/memory-0.8.4.jar]
2018-03-29T14:09:28,520 INFO [main] io.druid.initialization.Initialization - Loading extension [druid-hdfs-storage] for class [interface io.druid.cli.CliCommandCreator]
2018-03-29T14:09:28,522 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/hadoop-common-2.7.3.2.6.4.0-91.jar]
2018-03-29T14:09:28,522 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/okio-1.4.0.jar]
2018-03-29T14:09:28,522 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/curator-client-2.11.0.jar]
2018-03-29T14:09:28,522 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/commons-beanutils-1.7.0.jar]
2018-03-29T14:09:28,522 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/jsp-api-2.1.jar]
2018-03-29T14:09:28,522 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/snappy-java-1.0.4.1.jar]
2018-03-29T14:09:28,523 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/stax-api-1.0-2.jar]
2018-03-29T14:09:28,523 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/servlet-api-2.5.jar]
2018-03-29T14:09:28,523 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/hadoop-client-2.7.3.2.6.4.0-91.jar]
2018-03-29T14:09:28,523 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/hadoop-mapreduce-client-jobclient-2.7.3.2.6.4.0-91.jar]
2018-03-29T14:09:28,523 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/okhttp-2.4.0.jar]
2018-03-29T14:09:28,523 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/commons-logging-1.1.1.jar]
2018-03-29T14:09:28,523 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/api-util-1.0.0-M20.jar]
2018-03-29T14:09:28,524 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/apacheds-i18n-2.0.0-M15.jar]
2018-03-29T14:09:28,524 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/curator-recipes-2.11.0.jar]
2018-03-29T14:09:28,524 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/netty-all-4.1.11.Final.jar]
2018-03-29T14:09:28,524 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/aws-java-sdk-kms-1.10.77.jar]
2018-03-29T14:09:28,524 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/druid-hdfs-storage-0.10.1.2.6.4.0-91.jar]
2018-03-29T14:09:28,525 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/nimbus-jose-jwt-3.9.jar]
2018-03-29T14:09:28,525 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/xmlenc-0.52.jar]
2018-03-29T14:09:28,525 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/htrace-core-3.1.0-incubating.jar]
2018-03-29T14:09:28,525 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/xz-1.0.jar]
2018-03-29T14:09:28,525 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/commons-beanutils-core-1.8.0.jar]
2018-03-29T14:09:28,525 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/gson-2.2.4.jar]
2018-03-29T14:09:28,526 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/jackson-core-2.4.6.jar]
2018-03-29T14:09:28,526 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/azure-storage-5.4.0.jar]
2018-03-29T14:09:28,526 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/curator-framework-2.11.0.jar]
2018-03-29T14:09:28,526 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/hadoop-mapreduce-client-core-2.7.3.2.6.4.0-91.jar]
2018-03-29T14:09:28,526 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/jackson-xc-1.9.13.jar]
2018-03-29T14:09:28,526 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/commons-configuration-1.6.jar]
2018-03-29T14:09:28,526 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/jackson-dataformat-cbor-2.5.3.jar]
2018-03-29T14:09:28,526 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/hadoop-yarn-api-2.7.3.2.6.4.0-91.jar]
2018-03-29T14:09:28,527 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/hadoop-yarn-client-2.7.3.2.6.4.0-91.jar]
2018-03-29T14:09:28,527 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/aws-java-sdk-core-1.10.77.jar]
2018-03-29T14:09:28,527 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/hadoop-hdfs-2.7.3.2.6.4.0-91.jar]
2018-03-29T14:09:28,527 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/hadoop-yarn-registry-2.7.3.2.6.4.0-91.jar]
2018-03-29T14:09:28,527 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/azure-keyvault-core-0.8.0.jar]
2018-03-29T14:09:28,527 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/hadoop-yarn-common-2.7.3.2.6.4.0-91.jar]
2018-03-29T14:09:28,527 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/api-asn1-api-1.0.0-M20.jar]
2018-03-29T14:09:28,528 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/hadoop-yarn-server-common-2.7.3.2.6.4.0-91.jar]
2018-03-29T14:09:28,528 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/commons-lang3-3.4.jar]
2018-03-29T14:09:28,528 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/hadoop-auth-2.7.3.2.6.4.0-91.jar]
2018-03-29T14:09:28,528 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/commons-net-3.1.jar]
2018-03-29T14:09:28,528 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/paranamer-2.3.jar]
2018-03-29T14:09:28,528 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/jersey-client-1.9.jar]
2018-03-29T14:09:28,528 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/hadoop-mapreduce-client-common-2.7.3.2.6.4.0-91.jar]
2018-03-29T14:09:28,529 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/avro-1.7.4.jar]
2018-03-29T14:09:28,529 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/jackson-databind-2.4.6.jar]
2018-03-29T14:09:28,529 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/jaxb-api-2.2.2.jar]
2018-03-29T14:09:28,529 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/jackson-annotations-2.4.6.jar]
2018-03-29T14:09:28,529 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/httpclient-4.5.1.jar]
2018-03-29T14:09:28,529 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/jackson-jaxrs-1.9.13.jar]
2018-03-29T14:09:28,529 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/xml-apis-1.3.04.jar]
2018-03-29T14:09:28,530 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/commons-compress-1.4.1.jar]
2018-03-29T14:09:28,530 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/xercesImpl-2.9.1.jar]
2018-03-29T14:09:28,530 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/joda-time-2.9.9.jar]
2018-03-29T14:09:28,530 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/jetty-sslengine-6.1.26.hwx.jar]
2018-03-29T14:09:28,530 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/commons-digester-1.8.jar]
2018-03-29T14:09:28,530 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/hadoop-aws-2.7.3.2.6.4.0-91.jar]
2018-03-29T14:09:28,530 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/json-smart-1.1.1.jar]
2018-03-29T14:09:28,530 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/apacheds-kerberos-codec-2.0.0-M15.jar]
2018-03-29T14:09:28,531 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/netty-3.6.2.Final.jar]
2018-03-29T14:09:28,531 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/aws-java-sdk-s3-1.10.77.jar]
2018-03-29T14:09:28,531 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/httpcore-4.4.3.jar]
2018-03-29T14:09:28,531 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/guava-16.0.1.jar]
2018-03-29T14:09:28,531 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/hadoop-mapreduce-client-shuffle-2.7.3.2.6.4.0-91.jar]
2018-03-29T14:09:28,531 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/commons-collections-3.2.2.jar]
2018-03-29T14:09:28,531 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/hadoop-mapreduce-client-app-2.7.3.2.6.4.0-91.jar]
2018-03-29T14:09:28,531 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/jcip-annotations-1.0.jar]
2018-03-29T14:09:28,532 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/commons-math3-3.6.1.jar]
2018-03-29T14:09:28,532 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/leveldbjni-all-1.8.jar]
2018-03-29T14:09:28,547 INFO [main] io.druid.initialization.Initialization - Loading extension [druid-kafka-indexing-service] for class [interface io.druid.cli.CliCommandCreator]
2018-03-29T14:09:28,547 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-kafka-indexing-service/snappy-java-1.1.2.6.jar]
2018-03-29T14:09:28,547 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-kafka-indexing-service/slf4j-api-1.6.4.jar]
2018-03-29T14:09:28,547 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-kafka-indexing-service/druid-kafka-indexing-service-0.10.1.2.6.4.0-91.jar]
2018-03-29T14:09:28,548 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-kafka-indexing-service/lz4-1.3.0.jar]
2018-03-29T14:09:28,548 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-kafka-indexing-service/kafka-clients-0.10.2.0.jar]
2018-03-29T14:09:28,616 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.guice.ModulesConfig] from props[druid.modules.] as [ModulesConfig{excludeList=[]}]
2018-03-29T14:09:28,772 INFO [main] io.druid.initialization.Initialization - Loading extension [druid-datasketches] for class [interface io.druid.initialization.DruidModule]
2018-03-29T14:09:28,775 INFO [main] io.druid.initialization.Initialization - Adding implementation [io.druid.query.aggregation.datasketches.theta.SketchModule] for class [interface io.druid.initialization.DruidModule] from local file system extension
2018-03-29T14:09:28,775 INFO [main] io.druid.initialization.Initialization - Adding implementation [io.druid.query.aggregation.datasketches.theta.oldapi.OldApiSketchModule] for class [interface io.druid.initialization.DruidModule] from local file system extension
2018-03-29T14:09:28,776 INFO [main] io.druid.initialization.Initialization - Loading extension [druid-hdfs-storage] for class [interface io.druid.initialization.DruidModule]
2018-03-29T14:09:28,778 INFO [main] io.druid.initialization.Initialization - Adding implementation [io.druid.storage.hdfs.HdfsStorageDruidModule] for class [interface io.druid.initialization.DruidModule] from local file system extension
2018-03-29T14:09:28,778 INFO [main] io.druid.initialization.Initialization - Loading extension [druid-kafka-indexing-service] for class [interface io.druid.initialization.DruidModule]
2018-03-29T14:09:28,779 INFO [main] io.druid.initialization.Initialization - Adding implementation [io.druid.indexing.kafka.KafkaIndexTaskModule] for class [interface io.druid.initialization.DruidModule] from local file system extension
2018-03-29T14:09:29,499 WARN [main] org.apache.hadoop.util.NativeCodeLoader - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2018-03-29T14:09:30,292 WARN [main] org.apache.hadoop.hdfs.shortcircuit.DomainSocketFactory - The short-circuit local reads feature cannot be used because libhadoop cannot be loaded.
2018-03-29T14:09:30,672 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.server.metrics.DruidMonitorSchedulerConfig] from props[druid.monitoring.] as [io.druid.server.metrics.DruidMonitorSchedulerConfig@24eeac69]
2018-03-29T14:09:30,679 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.server.metrics.MonitorsConfig] from props[druid.monitoring.] as [MonitorsConfig{monitors=[class com.metamx.metrics.JvmMonitor]}]
2018-03-29T14:09:30,698 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.server.DruidNode] from props[druid.] as [DruidNode{serviceName='druid/middlemanager', host='sandbox-hdp.hortonworks.com', port=8100}]
2018-03-29T14:09:30,774 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.query.lookup.LookupConfig] from props[druid.lookup.] as [LookupConfig{snapshotWorkingDir=''}]
2018-03-29T14:09:30,844 INFO [main] io.druid.server.metrics.MetricsModule - Adding monitor[com.metamx.metrics.JvmMonitor@485caa8f]
2018-03-29T14:09:30,847 INFO [main] io.druid.server.metrics.MetricsModule - Adding monitor[io.druid.query.ExecutorServiceMonitor@4d157493]
2018-03-29T14:09:30,848 INFO [main] io.druid.server.metrics.MetricsModule - Adding monitor[io.druid.server.initialization.jetty.JettyServerModule$JettyMonitor@76e3b45b]
2018-03-29T14:09:30,854 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.server.log.StartupLoggingConfig] from props[druid.startup.logging.] as [io.druid.server.log.StartupLoggingConfig@3e28fee1]
2018-03-29T14:09:30,854 INFO [main] io.druid.cli.CliPeon - Starting up with processors[4], memory[308,281,344], maxMemory[1,908,932,608].
2018-03-29T14:09:30,857 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.storage.hdfs.HdfsKerberosConfig] from props[druid.hadoop.security.kerberos.] as [io.druid.storage.hdfs.HdfsKerberosConfig@0]
2018-03-29T14:09:30,864 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.indexing.common.config.TaskConfig] from props[druid.indexer.task.] as [io.druid.indexing.common.config.TaskConfig@2eda072]
2018-03-29T14:09:30,867 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.guice.http.DruidHttpClientConfig] from props[druid.global.http.] as [io.druid.guice.http.DruidHttpClientConfig@4091b9c3]
2018-03-29T14:09:30,928 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.client.indexing.IndexingServiceSelectorConfig] from props[druid.selectors.indexing.] as [io.druid.client.indexing.IndexingServiceSelectorConfig@7e7e962d]
2018-03-29T14:09:30,931 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.curator.CuratorConfig] from props[druid.zk.service.] as [io.druid.curator.CuratorConfig@7499eac7]
2018-03-29T14:09:30,934 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.curator.ExhibitorConfig] from props[druid.exhibitor.service.] as [io.druid.curator.ExhibitorConfig@d88f893]
2018-03-29T14:09:30,938 WARN [main] org.apache.curator.retry.ExponentialBackoffRetry - maxRetries too large (30). Pinning to 29
2018-03-29T14:09:30,959 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.server.initialization.CuratorDiscoveryConfig] from props[druid.discovery.curator.] as [io.druid.server.initialization.CuratorDiscoveryConfig@5b000fe6]
2018-03-29T14:09:30,980 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.indexing.common.RetryPolicyConfig] from props[druid.peon.taskActionClient.retry.] as [io.druid.indexing.common.RetryPolicyConfig@7186b202]
2018-03-29T14:09:30,982 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.storage.hdfs.HdfsDataSegmentPusherConfig] from props[druid.storage.] as [io.druid.storage.hdfs.HdfsDataSegmentPusherConfig@1a538ed8]
2018-03-29T14:09:30,983 INFO [main] io.druid.storage.hdfs.HdfsDataSegmentPusher - Configured HDFS as deep storage
2018-03-29T14:09:30,986 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.segment.loading.LocalDataSegmentPusherConfig] from props[druid.storage.] as [io.druid.segment.loading.LocalDataSegmentPusherConfig@270a620]
2018-03-29T14:09:30,991 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.client.DruidServerConfig] from props[druid.server.] as [io.druid.client.DruidServerConfig@4dfe8b37]
2018-03-29T14:09:30,994 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.server.initialization.BatchDataSegmentAnnouncerConfig] from props[druid.announcer.] as [io.druid.server.initialization.BatchDataSegmentAnnouncerConfig@3c232051]
2018-03-29T14:09:30,996 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.server.initialization.ZkPathsConfig] from props[druid.zk.paths.] as [io.druid.server.initialization.ZkPathsConfig@22e2266d]
2018-03-29T14:09:31,002 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[interface io.druid.server.coordination.DataSegmentAnnouncerProvider] from props[druid.announcer.] as [io.druid.server.coordination.BatchDataSegmentAnnouncerProvider@59fc6d05]
2018-03-29T14:09:31,004 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.client.coordinator.CoordinatorSelectorConfig] from props[druid.selectors.coordinator.] as [io.druid.client.coordinator.CoordinatorSelectorConfig@7e87ef9e]
2018-03-29T14:09:31,007 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.segment.realtime.plumber.CoordinatorBasedSegmentHandoffNotifierConfig] from props[druid.segment.handoff.] as [io.druid.segment.realtime.plumber.CoordinatorBasedSegmentHandoffNotifierConfig@7c241a9f]
2018-03-29T14:09:31,009 INFO [main] org.skife.config.ConfigurationObjectFactory - Assigning value [50000000] for [druid.processing.buffer.sizeBytes] on [io.druid.query.DruidProcessingConfig#intermediateComputeSizeBytes()]
2018-03-29T14:09:31,011 INFO [main] org.skife.config.ConfigurationObjectFactory - Using method itself for [druid.computation.buffer.poolCacheMaxCount, ${base_path}.buffer.poolCacheMaxCount] on [io.druid.query.DruidProcessingConfig#poolCacheMaxCount()]
2018-03-29T14:09:31,012 INFO [main] org.skife.config.ConfigurationObjectFactory - Using method itself for [${base_path}.fifo] on [io.druid.query.DruidProcessingConfig#isFifo()]
2018-03-29T14:09:31,012 INFO [main] org.skife.config.ConfigurationObjectFactory - Assigning value [2] for [druid.processing.numThreads] on [io.druid.query.DruidProcessingConfig#getNumThreadsConfigured()]
2018-03-29T14:09:31,012 INFO [main] org.skife.config.ConfigurationObjectFactory - Using method itself for [${base_path}.numMergeBuffers] on [io.druid.query.DruidProcessingConfig#getNumMergeBuffersConfigured()]
2018-03-29T14:09:31,013 INFO [main] org.skife.config.ConfigurationObjectFactory - Using method itself for [${base_path}.columnCache.sizeBytes] on [io.druid.query.DruidProcessingConfig#columnCacheSizeBytes()]
2018-03-29T14:09:31,013 INFO [main] org.skife.config.ConfigurationObjectFactory - Using method itself for [${base_path}.tmpDir] on [io.druid.query.DruidProcessingConfig#getTmpDir()]
2018-03-29T14:09:31,013 INFO [main] org.skife.config.ConfigurationObjectFactory - Assigning default value [processing-%s] for [${base_path}.formatString] on [io.druid.java.util.common.concurrent.ExecutorServiceConfig#getFormatString()]
2018-03-29T14:09:31,083 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[interface io.druid.client.cache.CacheProvider] from props[druid.cache.] as [io.druid.client.cache.LocalCacheProvider@182e7eda]
2018-03-29T14:09:31,091 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.client.cache.CacheConfig] from props[druid.realtime.cache.] as [io.druid.client.cache.CacheConfig@27dc627a]
2018-03-29T14:09:31,099 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.server.initialization.ServerConfig] from props[druid.server.http.] as [ServerConfig{numThreads=50, maxIdleTime=PT5M, defaultQueryTimeout=300000, maxScatterGatherBytes=9223372036854775807}]
2018-03-29T14:09:31,107 INFO [main] org.eclipse.jetty.util.log - Logging initialized @3834ms
2018-03-29T14:09:31,221 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.query.search.search.SearchQueryConfig] from props[druid.query.search.] as [io.druid.query.search.search.SearchQueryConfig@622d7e4]
2018-03-29T14:09:31,226 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.query.metadata.SegmentMetadataQueryConfig] from props[druid.query.segmentMetadata.] as [io.druid.query.metadata.SegmentMetadataQueryConfig@26582ca]
2018-03-29T14:09:31,229 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.query.groupby.GroupByQueryConfig] from props[druid.query.groupBy.] as [GroupByQueryConfig{defaultStrategy='v2', singleThreaded=false, maxIntermediateRows=50000, maxResults=500000, bufferGrouperMaxSize=2147483647, bufferGrouperMaxLoadFactor=0.0, bufferGrouperInitialBuckets=0, maxMergingDictionarySize=100000000, maxOnDiskStorage=0, forcePushDownLimit=false, forceHashAggregation=false}]
2018-03-29T14:09:31,244 INFO [main] io.druid.offheap.OffheapBufferGenerator - Allocating new intermediate processing buffer[0] of size[50,000,000]
2018-03-29T14:09:31,254 INFO [main] io.druid.offheap.OffheapBufferGenerator - Allocating new intermediate processing buffer[1] of size[50,000,000]
2018-03-29T14:09:31,263 INFO [main] io.druid.offheap.OffheapBufferGenerator - Allocating new result merging buffer[0] of size[50,000,000]
2018-03-29T14:09:31,314 INFO [main] io.druid.offheap.OffheapBufferGenerator - Allocating new result merging buffer[1] of size[50,000,000]
2018-03-29T14:09:31,370 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.query.scan.ScanQueryConfig] from props[druid.query.scan.] as [ScanQueryConfig{legacy=false}]
2018-03-29T14:09:31,373 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.query.select.SelectQueryConfig] from props[druid.query.select.] as [io.druid.query.select.SelectQueryConfig@707865bd]
2018-03-29T14:09:31,375 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.query.topn.TopNQueryConfig] from props[druid.query.topN.] as [io.druid.query.topn.TopNQueryConfig@3ebc955b]
2018-03-29T14:09:31,379 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[interface io.druid.server.log.RequestLoggerProvider] from props[druid.request.logging.] as [io.druid.server.log.NoopRequestLoggerProvider@6ffd4c0d]
2018-03-29T14:09:31,380 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.server.security.AuthConfig] from props[druid.auth.] as [AuthConfig{enabled=false}]
2018-03-29T14:09:31,388 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.query.lookup.LookupListeningAnnouncerConfig] from props[druid.lookup.] as [ListeningAnnouncerConfig{listenersPath='/druid/listeners'}]
2018-03-29T14:09:31,389 INFO [main] io.druid.java.util.common.lifecycle.Lifecycle$AnnotationBasedHandler - Invoking start method[public void io.druid.initialization.Log4jShutterDownerModule$Log4jShutterDowner.start()] on object[io.druid.initialization.Log4jShutterDownerModule$Log4jShutterDowner@3dbd7107].
2018-03-29T14:09:31,391 INFO [main] io.druid.java.util.common.lifecycle.Lifecycle$AnnotationBasedHandler - Invoking start method[public void com.metamx.emitter.service.ServiceEmitter.start()] on object[com.metamx.emitter.service.ServiceEmitter@5a090f62].
2018-03-29T14:09:31,392 INFO [main] io.druid.java.util.common.lifecycle.Lifecycle$AnnotationBasedHandler - Invoking start method[public void io.druid.query.lookup.LookupReferencesManager.start()] on object[io.druid.query.lookup.LookupReferencesManager@89caf47].
2018-03-29T14:09:31,393 INFO [main] io.druid.query.lookup.LookupReferencesManager - LookupReferencesManager is starting.
2018-03-29T14:09:31,393 INFO [main] io.druid.query.lookup.LookupReferencesManager - LookupReferencesManager is started.
2018-03-29T14:09:31,394 INFO [main] io.druid.java.util.common.lifecycle.Lifecycle$AnnotationBasedHandler - Invoking start method[public void com.metamx.metrics.MonitorScheduler.start()] on object[com.metamx.metrics.MonitorScheduler@be164d8].
2018-03-29T14:09:31,403 INFO [main] io.druid.java.util.common.lifecycle.Lifecycle$AnnotationBasedHandler - Invoking start method[public void io.druid.storage.hdfs.HdfsStorageAuthentication.authenticate()] on object[io.druid.storage.hdfs.HdfsStorageAuthentication@2e56b4d].
2018-03-29T14:09:31,405 INFO [main] com.metamx.common.lifecycle.Lifecycle$AnnotationBasedHandler - Invoking start method[public void com.metamx.http.client.NettyHttpClient.start()] on object[com.metamx.http.client.NettyHttpClient@36ad229b].
2018-03-29T14:09:31,405 INFO [main] io.druid.curator.CuratorModule - Starting Curator
2018-03-29T14:09:31,405 INFO [main] org.apache.curator.framework.imps.CuratorFrameworkImpl - Starting
2018-03-29T14:09:31,413 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:zookeeper.version=3.4.6-91--1, built on 01/04/2018 09:27 GMT
2018-03-29T14:09:31,413 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:host.name=sandbox-hdp.hortonworks.com
2018-03-29T14:09:31,413 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:java.version=1.8.0_161
2018-03-29T14:09:31,413 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:java.vendor=Oracle Corporation
2018-03-29T14:09:31,413 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:java.home=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.161-3.b14.el6_9.x86_64/jre
2018-03-29T14:09:31,413 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:java.class.path=/usr/hdp/current/druid-middlemanager/conf/_common:/usr/hdp/current/druid-middlemanager/conf/middleManager:/usr/hdp/current/druid-middlemanager/lib/antlr4-runtime-4.5.1.jar:/usr/hdp/current/druid-middlemanager/lib/druid-sql-0.10.1.2.6.4.0-91.jar:/usr/hdp/current/druid-middlemanager/lib/plexus-interpolation-1.19.jar:/usr/hdp/current/druid-middlemanager/lib/guice-servlet-4.1.0.jar:/usr/hdp/current/druid-middlemanager/lib/jackson-jaxrs-smile-provider-2.4.6.jar:/usr/hdp/current/druid-middlemanager/lib/curator-client-2.11.0.jar:/usr/hdp/current/druid-middlemanager/lib/tesla-aether-0.0.5.jar:/usr/hdp/current/druid-middlemanager/lib/wagon-provider-api-2.4.jar:/usr/hdp/current/druid-middlemanager/lib/extendedset-0.10.1.2.6.4.0-91.jar:/usr/hdp/current/druid-middlemanager/lib/jackson-dataformat-smile-2.4.6.jar:/usr/hdp/current/druid-middlemanager/lib/maven-model-3.1.1.jar:/usr/hdp/current/druid-middlemanager/lib/json-path-2.1.0.jar:/usr/hdp/current/druid-middlemanager/lib/jetty-util-9.3.19.v20170502.jar:/usr/hdp/current/druid-middlemanager/lib/commons-compiler-2.7.6.jar:/usr/hdp/current/druid-middlemanager/lib/log4j-api-2.5.jar:/usr/hdp/current/druid-middlemanager/lib/jcl-over-slf4j-1.7.12.jar:/usr/hdp/current/druid-middlemanager/lib/druid-server-0.10.1.2.6.4.0-91.jar:/usr/hdp/current/druid-middlemanager/lib/jetty-proxy-9.3.19.v20170502.jar:/usr/hdp/current/druid-middlemanager/lib/commons-pool-1.6.jar:/usr/hdp/current/druid-middlemanager/lib/jsr311-api-1.1.1.jar:/usr/hdp/current/druid-middlemanager/lib/jsr305-2.0.1.jar:/usr/hdp/current/druid-middlemanager/lib/druid-indexing-service-0.10.1.2.6.4.0-91.jar:/usr/hdp/current/druid-middlemanager/lib/commons-logging-1.1.1.jar:/usr/hdp/current/druid-middlemanager/lib/okhttp-1.0.2.jar:/usr/hdp/current/druid-middlemanager/lib/RoaringBitmap-0.5.18.jar:/usr/hdp/current/druid-middlemanager/lib/http-client-1.1.0.jar:/usr/hdp/current/druid-middlemanager/lib/curator-recipes-2.11.0.jar:/usr/hdp/current/druid-middlemanager/lib/aopalliance-1.0.jar:/usr/hdp/current/druid-middlemanager/lib/netty-all-4.1.11.Final.jar:/usr/hdp/current/druid-middlemanager/lib/derbyclient-10.11.1.1.jar:/usr/hdp/current/druid-middlemanager/lib/bytebuffer-collections-0.10.1.2.6.4.0-91.jar:/usr/hdp/current/druid-middlemanager/lib/protobuf-java-3.1.0.jar:/usr/hdp/current/druid-middlemanager/lib/aether-spi-0.9.0.M2.jar:/usr/hdp/current/druid-middlemanager/lib/derbynet-10.11.1.1.jar:/usr/hdp/current/druid-middlemanager/lib/compress-lzf-1.0.3.jar:/usr/hdp/current/druid-middlemanager/lib/disruptor-3.3.6.jar:/usr/hdp/current/druid-middlemanager/lib/jetty-http-9.3.19.v20170502.jar:/usr/hdp/current/druid-middlemanager/lib/maven-settings-3.1.1.jar:/usr/hdp/current/druid-middlemanager/lib/bcprov-jdk15on-1.52.jar:/usr/hdp/current/druid-middlemanager/lib/classmate-1.0.0.jar:/usr/hdp/current/druid-middlemanager/lib/opencsv-2.3.jar:/usr/hdp/current/druid-middlemanager/lib/druid-aws-common-0.10.1.2.6.4.0-91.jar:/usr/hdp/current/druid-middlemanager/lib/aws-java-sdk-ec2-1.10.77.jar:/usr/hdp/current/druid-middlemanager/lib/druid-services-0.10.1.2.6.4.0-91.jar:/usr/hdp/current/druid-middlemanager/lib/javax.inject-1.jar:/usr/hdp/current/druid-middlemanager/lib/jackson-datatype-joda-2.4.6.jar:/usr/hdp/current/druid-middlemanager/lib/commons-dbcp-1.4.jar:/usr/hdp/current/druid-middlemanager/lib/jackson-datatype-guava-2.4.6.jar:/usr/hdp/current/druid-middlemanager/lib/jackson-core-2.4.6.jar:/usr/hdp/current/druid-middlemanager/lib/jline-0.9.94.jar:/usr/hdp/current/druid-middlemanager/lib/base64-2.3.8.jar:/usr/hdp/current/druid-middlemanager/lib/calcite-core-1.12.0.jar:/usr/hdp/current/druid-middlemanager/lib/mapdb-1.0.8.jar:/usr/hdp/current/druid-middlemanager/lib/curator-framework-2.11.0.jar:/usr/hdp/current/druid-middlemanager/lib/log4j-core-2.5.jar:/usr/hdp/current/druid-middlemanager/lib/jetty-servlets-9.3.19.v20170502.jar:/usr/hdp/current/druid-middlemanager/lib/druid-console-0.0.2.jar:/usr/hdp/current/druid-middlemanager/lib/maven-model-builder-3.1.1.jar:/usr/hdp/current/druid-middlemanager/lib/config-magic-0.9.jar:/usr/hdp/current/druid-middlemanager/lib/emitter-0.4.5.jar:/usr/hdp/current/druid-middlemanager/lib/slf4j-api-1.6.4.jar:/usr/hdp/current/druid-middlemanager/lib/jboss-logging-3.1.3.GA.jar:/usr/hdp/current/druid-middlemanager/lib/commons-cli-1.2.jar:/usr/hdp/current/druid-middlemanager/lib/guice-4.1.0.jar:/usr/hdp/current/druid-middlemanager/lib/druid-common-0.10.1.2.6.4.0-91.jar:/usr/hdp/current/druid-middlemanager/lib/jackson-dataformat-cbor-2.5.3.jar:/usr/hdp/current/druid-middlemanager/lib/derby-10.11.1.1.jar:/usr/hdp/current/druid-middlemanager/lib/aether-api-0.9.0.M2.jar:/usr/hdp/current/druid-middlemanager/lib/commons-io-2.5.jar:/usr/hdp/current/druid-middlemanager/lib/netty-3.10.6.Final.jar:/usr/hdp/current/druid-middlemanager/lib/javax.el-3.0.0.jar:/usr/hdp/current/druid-middlemanager/lib/janino-2.7.6.jar:/usr/hdp/current/druid-middlemanager/lib/plexus-utils-3.0.15.jar:/usr/hdp/current/druid-middlemanager/lib/jackson-jaxrs-base-2.4.6.jar:/usr/hdp/current/druid-middlemanager/lib/aws-java-sdk-core-1.10.77.jar:/usr/hdp/current/druid-middlemanager/lib/javax.servlet-api-3.1.0.jar:/usr/hdp/current/druid-middlemanager/lib/log4j-1.2-api-2.5.jar:/usr/hdp/current/druid-middlemanager/lib/avatica-core-1.9.0.jar:/usr/hdp/current/druid-middlemanager/lib/aether-impl-0.9.0.M2.jar:/usr/hdp/current/druid-middlemanager/lib/jetty-security-9.3.19.v20170502.jar:/usr/hdp/current/druid-middlemanager/lib/spymemcached-2.12.3.jar:/usr/hdp/current/druid-middlemanager/lib/maven-aether-provider-3.1.1.jar:/usr/hdp/current/druid-middlemanager/lib/aggdesigner-algorithm-6.0.jar:/usr/hdp/current/druid-middlemanager/lib/aether-connector-okhttp-0.0.9.jar:/usr/hdp/current/druid-middlemanager/lib/commons-pool2-2.2.jar:/usr/hdp/current/druid-middlemanager/lib/jackson-databind-2.4.6.jar:/usr/hdp/current/druid-middlemanager/lib/asm-tree-5.2.jar:/usr/hdp/current/druid-middlemanager/lib/jetty-continuation-9.3.19.v20170502.jar:/usr/hdp/current/druid-middlemanager/lib/java-util-0.28.2.jar:/usr/hdp/current/druid-middlemanager/lib/maven-repository-metadata-3.1.1.jar:/usr/hdp/current/druid-middlemanager/lib/jackson-annotations-2.4.6.jar:/usr/hdp/current/druid-middlemanager/lib/calcite-linq4j-1.12.0.jar:/usr/hdp/current/druid-middlemanager/lib/jersey-servlet-1.19.3.jar:/usr/hdp/current/druid-middlemanager/lib/jackson-mapper-asl-1.9.13.jar:/usr/hdp/current/druid-middlemanager/lib/asm-5.2.jar:/usr/hdp/current/druid-middlemanager/lib/asm-commons-5.2.jar:/usr/hdp/current/druid-middlemanager/lib/httpclient-4.5.1.jar:/usr/hdp/current/druid-middlemanager/lib/lz4-1.3.0.jar:/usr/hdp/current/druid-middlemanager/lib/avatica-metrics-1.9.0.jar:/usr/hdp/current/druid-middlemanager/lib/jetty-servlet-9.3.19.v20170502.jar:/usr/hdp/current/druid-middlemanager/lib/jackson-core-asl-1.9.13.jar:/usr/hdp/current/druid-middlemanager/lib/log4j-slf4j-impl-2.5.jar:/usr/hdp/current/druid-middlemanager/lib/javax.el-api-3.0.0.jar:/usr/hdp/current/druid-middlemanager/lib/irc-api-1.0-0014.jar:/usr/hdp/current/druid-middlemanager/lib/java-util-0.10.1.2.6.4.0-91.jar:/usr/hdp/current/druid-middlemanager/lib/jersey-guice-1.19.3.jar:/usr/hdp/current/druid-middlemanager/lib/activation-1.1.1.jar:/usr/hdp/current/druid-middlemanager/lib/druid-indexing-hadoop-0.10.1.2.6.4.0-91.jar:/usr/hdp/current/druid-middlemanager/lib/commons-lang-2.6.jar:/usr/hdp/current/druid-middlemanager/lib/jersey-server-1.19.3.jar:/usr/hdp/current/druid-middlemanager/lib/aether-connector-file-0.9.0.M2.jar:/usr/hdp/current/druid-middlemanager/lib/geoip2-0.4.0.jar:/usr/hdp/current/druid-middlemanager/lib/joda-time-2.9.9.jar:/usr/hdp/current/druid-middlemanager/lib/jetty-server-9.3.19.v20170502.jar:/usr/hdp/current/druid-middlemanager/lib/druid-hll-0.10.1.2.6.4.0-91.jar:/usr/hdp/current/druid-middlemanager/lib/guice-multibindings-4.1.0.jar:/usr/hdp/current/druid-middlemanager/lib/curator-x-discovery-2.11.0.jar:/usr/hdp/current/druid-middlemanager/lib/druid-api-0.10.1.2.6.4.0-91.jar:/usr/hdp/current/druid-middlemanager/lib/commons-lang3-3.2.jar:/usr/hdp/current/druid-middlemanager/lib/maxminddb-0.2.0.jar:/usr/hdp/current/druid-middlemanager/lib/jetty-io-9.3.19.v20170502.jar:/usr/hdp/current/druid-middlemanager/lib/maven-settings-builder-3.1.1.jar:/usr/hdp/current/druid-middlemanager/lib/jdbi-2.63.1.jar:/usr/hdp/current/druid-middlemanager/lib/aether-util-0.9.0.M2.jar:/usr/hdp/current/druid-middlemanager/lib/icu4j-54.1.1.jar:/usr/hdp/current/druid-middlemanager/lib/validation-api-1.1.0.Final.jar:/usr/hdp/current/druid-middlemanager/lib/jets3t-0.9.4.jar:/usr/hdp/current/druid-middlemanager/lib/jackson-module-jaxb-annotations-2.4.6.jar:/usr/hdp/current/druid-middlemanager/lib/server-metrics-0.2.8.jar:/usr/hdp/current/druid-middlemanager/lib/jetty-client-9.3.19.v20170502.jar:/usr/hdp/current/druid-middlemanager/lib/avatica-server-1.9.0.jar:/usr/hdp/current/druid-middlemanager/lib/httpcore-4.4.3.jar:/usr/hdp/current/druid-middlemanager/lib/airline-0.7.jar:/usr/hdp/current/druid-middlemanager/lib/jackson-jaxrs-json-provider-2.4.6.jar:/usr/hdp/current/druid-middlemanager/lib/druid-processing-0.10.1.2.6.4.0-91.jar:/usr/hdp/current/druid-middlemanager/lib/zookeeper-3.4.6.2.6.4.0-91.jar:/usr/hdp/current/druid-middlemanager/lib/guava-16.0.1.jar:/usr/hdp/current/druid-middlemanager/lib/commons-codec-1.7.jar:/usr/hdp/current/druid-middlemanager/lib/jersey-core-1.19.3.jar:/usr/hdp/current/druid-middlemanager/lib/commons-dbcp2-2.0.1.jar:/usr/hdp/current/druid-middlemanager/lib/hibernate-validator-5.1.3.Final.jar:/usr/hdp/current/druid-middlemanager/lib/java-xmlbuilder-1.1.jar:/usr/hdp/current/druid-middlemanager/lib/commons-math3-3.6.1.jar:/usr/hdp/current/druid-middlemanager/lib/rhino-1.7R5.jar:/usr/hdp/current/druid-middlemanager/lib/log4j-jul-2.5.jar:/usr/hdp/current/druid-middlemanager/lib/fastutil-7.2.0.jar:/usr/hdp/2.6.4.0-91/hadoop/conf
2018-03-29T14:09:31,413 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
2018-03-29T14:09:31,413 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:java.io.tmpdir=/tmp
2018-03-29T14:09:31,413 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:java.compiler=<NA>
2018-03-29T14:09:31,413 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:os.name=Linux
2018-03-29T14:09:31,413 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:os.arch=amd64
2018-03-29T14:09:31,413 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:os.version=4.4.114-1.el7.elrepo.x86_64
2018-03-29T14:09:31,413 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:user.name=druid
2018-03-29T14:09:31,413 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:user.home=/home/druid
2018-03-29T14:09:31,413 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:user.dir=/home/druid
2018-03-29T14:09:31,414 INFO [main] org.apache.zookeeper.ZooKeeper - Initiating client connection, connectString=sandbox-hdp.hortonworks.com:2181 sessionTimeout=30000 watcher=org.apache.curator.ConnectionState@29013ef2
2018-03-29T14:09:31,434 INFO [main] io.druid.java.util.common.lifecycle.Lifecycle$AnnotationBasedHandler - Invoking start method[public void io.druid.curator.discovery.ServerDiscoverySelector.start() throws java.lang.Exception] on object[io.druid.curator.discovery.ServerDiscoverySelector@13cc0b90].
2018-03-29T14:09:31,474 INFO [main-SendThread(sandbox-hdp.hortonworks.com:2181)] org.apache.zookeeper.ClientCnxn - Opening socket connection to server sandbox-hdp.hortonworks.com/172.17.0.2:2181. Will not attempt to authenticate using SASL (unknown error)
2018-03-29T14:09:31,481 INFO [main-SendThread(sandbox-hdp.hortonworks.com:2181)] org.apache.zookeeper.ClientCnxn - Socket connection established, initiating session, client: /172.17.0.2:42090, server: sandbox-hdp.hortonworks.com/172.17.0.2:2181
2018-03-29T14:09:31,491 INFO [main-SendThread(sandbox-hdp.hortonworks.com:2181)] org.apache.zookeeper.ClientCnxn - Session establishment complete on server sandbox-hdp.hortonworks.com/172.17.0.2:2181, sessionid = 0x16271e552860012, negotiated timeout = 30000
2018-03-29T14:09:31,494 INFO [main-EventThread] org.apache.curator.framework.state.ConnectionStateManager - State change: CONNECTED
2018-03-29T14:09:31,590 INFO [main] io.druid.java.util.common.lifecycle.Lifecycle$AnnotationBasedHandler - Invoking start method[public void io.druid.curator.announcement.Announcer.start()] on object[io.druid.curator.announcement.Announcer@1f129467].
2018-03-29T14:09:31,591 INFO [main] io.druid.java.util.common.lifecycle.Lifecycle$AnnotationBasedHandler - Invoking start method[public void io.druid.curator.discovery.ServerDiscoverySelector.start() throws java.lang.Exception] on object[io.druid.curator.discovery.ServerDiscoverySelector@646cd766].
2018-03-29T14:09:31,612 INFO [main] io.druid.java.util.common.lifecycle.Lifecycle$AnnotationBasedHandler - Invoking start method[public void io.druid.indexing.worker.executor.ExecutorLifecycle.start() throws java.lang.InterruptedException] on object[io.druid.indexing.worker.executor.ExecutorLifecycle@65eb3597].
2018-03-29T14:09:31,662 INFO [main] io.druid.indexing.worker.executor.ExecutorLifecycle - Running with task: {
  "type" : "index",
  "id" : "index_usgs_2018-03-29T14:09:27.159Z",
  "resource" : {
    "availabilityGroup" : "index_usgs_2018-03-29T14:09:27.159Z",
    "requiredCapacity" : 1
  },
  "spec" : {
    "dataSchema" : {
      "dataSource" : "usgs",
      "parser" : {
        "type" : "string",
        "parseSpec" : {
          "format" : "tsv",
          "timestampSpec" : {
            "column" : "dates",
            "format" : "auto"
          },
          "dimensionsSpec" : {
            "dimensions" : [ "staid", "val", "dates" ],
            "dimensionExclusions" : [ ],
            "spatialDimensions" : [ ]
          }
        }
      },
      "metricsSpec" : [ {
        "type" : "count",
        "name" : "count"
      }, {
        "type" : "doubleSum",
        "name" : "avgFlowCuFtsec",
        "fieldName" : "val",
        "expression" : null
      } ],
      "granularitySpec" : {
        "type" : "uniform",
        "segmentGranularity" : "MONTH",
        "queryGranularity" : {
          "type" : "none"
        },
        "rollup" : true,
        "intervals" : [ "1963-01-01T00:00:00.000Z/2013-12-31T00:00:00.000Z" ]
      }
    },
    "ioConfig" : {
      "type" : "index",
      "firehose" : {
        "type" : "local",
        "baseDir" : "/tmp/usgs",
        "filter" : "napa-flow.tsv",
        "parser" : null
      },
      "appendToExisting" : false
    },
    "tuningConfig" : {
      "type" : "index",
      "targetPartitionSize" : 10000,
      "maxRowsInMemory" : 75000,
      "numShards" : null,
      "indexSpec" : {
        "bitmap" : {
          "type" : "concise"
        },
        "dimensionCompression" : "lz4",
        "metricCompression" : "lz4",
        "longEncoding" : "longs"
      },
      "maxPendingPersists" : 0,
      "buildV9Directly" : true,
      "forceExtendableShardSpecs" : false,
      "reportParseExceptions" : false,
      "publishTimeout" : 0
    }
  },
  "context" : null,
  "groupId" : "index_usgs_2018-03-29T14:09:27.159Z",
  "dataSource" : "usgs"
}
2018-03-29T14:09:31,664 INFO [main] io.druid.indexing.worker.executor.ExecutorLifecycle - Attempting to lock file[/apps/druid/tasks/index_usgs_2018-03-29T14:09:27.159Z/lock].
2018-03-29T14:09:31,666 INFO [main] io.druid.indexing.worker.executor.ExecutorLifecycle - Acquired lock file[/apps/druid/tasks/index_usgs_2018-03-29T14:09:27.159Z/lock] in 1ms.
2018-03-29T14:09:31,671 INFO [main] io.druid.indexing.common.actions.RemoteTaskActionClient - Performing action for task[index_usgs_2018-03-29T14:09:27.159Z]: LockTryAcquireAction{interval=1963-01-01T00:00:00.000Z/2014-01-01T00:00:00.000Z}
2018-03-29T14:09:31,679 INFO [main] io.druid.indexing.common.actions.RemoteTaskActionClient - Submitting action for task[index_usgs_2018-03-29T14:09:27.159Z] to overlord[http://sandbox-hdp.hortonworks.com:8090/druid/indexer/v1/action]: LockTryAcquireAction{interval=1963-01-01T00:00:00.000Z/2014-01-01T00:00:00.000Z}
2018-03-29T14:09:31,688 INFO [main] com.metamx.http.client.pool.ChannelResourceFactory - Generating: http://sandbox-hdp.hortonworks.com:8090
2018-03-29T14:09:31,728 INFO [main] com.metamx.http.client.pool.ChannelResourceFactory - Generating: http://sandbox-hdp.hortonworks.com:8090
2018-03-29T14:09:31,729 INFO [main] com.metamx.http.client.pool.ChannelResourceFactory - Generating: http://sandbox-hdp.hortonworks.com:8090
2018-03-29T14:09:31,730 INFO [main] com.metamx.http.client.pool.ChannelResourceFactory - Generating: http://sandbox-hdp.hortonworks.com:8090
2018-03-29T14:09:31,730 INFO [main] com.metamx.http.client.pool.ChannelResourceFactory - Generating: http://sandbox-hdp.hortonworks.com:8090
2018-03-29T14:09:31,731 INFO [main] com.metamx.http.client.pool.ChannelResourceFactory - Generating: http://sandbox-hdp.hortonworks.com:8090
2018-03-29T14:09:31,731 INFO [main] com.metamx.http.client.pool.ChannelResourceFactory - Generating: http://sandbox-hdp.hortonworks.com:8090
2018-03-29T14:09:31,732 INFO [main] com.metamx.http.client.pool.ChannelResourceFactory - Generating: http://sandbox-hdp.hortonworks.com:8090
2018-03-29T14:09:31,732 INFO [main] com.metamx.http.client.pool.ChannelResourceFactory - Generating: http://sandbox-hdp.hortonworks.com:8090
2018-03-29T14:09:31,732 INFO [main] com.metamx.http.client.pool.ChannelResourceFactory - Generating: http://sandbox-hdp.hortonworks.com:8090
2018-03-29T14:09:31,732 INFO [main] com.metamx.http.client.pool.ChannelResourceFactory - Generating: http://sandbox-hdp.hortonworks.com:8090
2018-03-29T14:09:31,733 INFO [main] com.metamx.http.client.pool.ChannelResourceFactory - Generating: http://sandbox-hdp.hortonworks.com:8090
2018-03-29T14:09:31,733 INFO [main] com.metamx.http.client.pool.ChannelResourceFactory - Generating: http://sandbox-hdp.hortonworks.com:8090
2018-03-29T14:09:31,733 INFO [main] com.metamx.http.client.pool.ChannelResourceFactory - Generating: http://sandbox-hdp.hortonworks.com:8090
2018-03-29T14:09:31,734 INFO [main] com.metamx.http.client.pool.ChannelResourceFactory - Generating: http://sandbox-hdp.hortonworks.com:8090
2018-03-29T14:09:31,734 INFO [main] com.metamx.http.client.pool.ChannelResourceFactory - Generating: http://sandbox-hdp.hortonworks.com:8090
2018-03-29T14:09:31,734 INFO [main] com.metamx.http.client.pool.ChannelResourceFactory - Generating: http://sandbox-hdp.hortonworks.com:8090
2018-03-29T14:09:31,735 INFO [main] com.metamx.http.client.pool.ChannelResourceFactory - Generating: http://sandbox-hdp.hortonworks.com:8090
2018-03-29T14:09:31,735 INFO [main] com.metamx.http.client.pool.ChannelResourceFactory - Generating: http://sandbox-hdp.hortonworks.com:8090
2018-03-29T14:09:31,735 INFO [main] com.metamx.http.client.pool.ChannelResourceFactory - Generating: http://sandbox-hdp.hortonworks.com:8090
2018-03-29T14:09:31,821 INFO [task-runner-0-priority-0] io.druid.indexing.overlord.ThreadPoolTaskRunner - Running task: index_usgs_2018-03-29T14:09:27.159Z
2018-03-29T14:09:31,824 INFO [task-runner-0-priority-0] io.druid.indexing.overlord.TaskRunnerUtils - Task [index_usgs_2018-03-29T14:09:27.159Z] location changed to [TaskLocation{host='sandbox-hdp.hortonworks.com', port=8100}].
2018-03-29T14:09:31,824 INFO [task-runner-0-priority-0] io.druid.indexing.overlord.TaskRunnerUtils - Task [index_usgs_2018-03-29T14:09:27.159Z] status changed to [RUNNING].
2018-03-29T14:09:31,829 INFO [task-runner-0-priority-0] io.druid.indexing.common.task.IndexTask - Determining intervals and shardSpecs
2018-03-29T14:09:31,831 INFO [main] org.eclipse.jetty.server.Server - jetty-9.3.19.v20170502
2018-03-29T14:09:31,880 ERROR [task-runner-0-priority-0] io.druid.indexing.overlord.ThreadPoolTaskRunner - Exception while running task[IndexTask{id=index_usgs_2018-03-29T14:09:27.159Z, type=index, dataSource=usgs}]
java.lang.IllegalArgumentException: Instantiation of [simple type, class io.druid.data.input.impl.DelimitedParseSpec] value failed: If columns field is not set, the first row of your data must have your header and hasHeaderRow must be set to true.
	at com.fasterxml.jackson.databind.ObjectMapper._convert(ObjectMapper.java:2774) ~[jackson-databind-2.4.6.jar:2.4.6]
	at com.fasterxml.jackson.databind.ObjectMapper.convertValue(ObjectMapper.java:2700) ~[jackson-databind-2.4.6.jar:2.4.6]
	at io.druid.segment.indexing.DataSchema.getParser(DataSchema.java:111) ~[druid-server-0.10.1.2.6.4.0-91.jar:0.10.1.2.6.4.0-91]
	at io.druid.indexing.common.task.IndexTask.determineShardSpecs(IndexTask.java:269) ~[druid-indexing-service-0.10.1.2.6.4.0-91.jar:0.10.1.2.6.4.0-91]
	at io.druid.indexing.common.task.IndexTask.run(IndexTask.java:188) ~[druid-indexing-service-0.10.1.2.6.4.0-91.jar:0.10.1.2.6.4.0-91]
	at io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:436) [druid-indexing-service-0.10.1.2.6.4.0-91.jar:0.10.1.2.6.4.0-91]
	at io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:408) [druid-indexing-service-0.10.1.2.6.4.0-91.jar:0.10.1.2.6.4.0-91]
	at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_161]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_161]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_161]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_161]
Caused by: com.fasterxml.jackson.databind.JsonMappingException: Instantiation of [simple type, class io.druid.data.input.impl.DelimitedParseSpec] value failed: If columns field is not set, the first row of your data must have your header and hasHeaderRow must be set to true.
	at com.fasterxml.jackson.databind.deser.std.StdValueInstantiator.wrapException(StdValueInstantiator.java:405) ~[jackson-databind-2.4.6.jar:2.4.6]
	at com.fasterxml.jackson.databind.deser.std.StdValueInstantiator.createFromObjectWith(StdValueInstantiator.java:234) ~[jackson-databind-2.4.6.jar:2.4.6]
	at com.fasterxml.jackson.databind.deser.impl.PropertyBasedCreator.build(PropertyBasedCreator.java:167) ~[jackson-databind-2.4.6.jar:2.4.6]
	at com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeUsingPropertyBased(BeanDeserializer.java:398) ~[jackson-databind-2.4.6.jar:2.4.6]
	at com.fasterxml.jackson.databind.deser.BeanDeserializerBase.deserializeFromObjectUsingNonDefault(BeanDeserializerBase.java:1064) ~[jackson-databind-2.4.6.jar:2.4.6]
	at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserializeFromObject(BeanDeserializer.java:264) ~[jackson-databind-2.4.6.jar:2.4.6]
	at com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeOther(BeanDeserializer.java:156) ~[jackson-databind-2.4.6.jar:2.4.6]
	at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:126) ~[jackson-databind-2.4.6.jar:2.4.6]
	at com.fasterxml.jackson.databind.jsontype.impl.AsPropertyTypeDeserializer._deserializeTypedForId(AsPropertyTypeDeserializer.java:113) ~[jackson-databind-2.4.6.jar:2.4.6]
	at com.fasterxml.jackson.databind.jsontype.impl.AsPropertyTypeDeserializer.deserializeTypedFromObject(AsPropertyTypeDeserializer.java:84) ~[jackson-databind-2.4.6.jar:2.4.6]
	at com.fasterxml.jackson.databind.deser.AbstractDeserializer.deserializeWithType(AbstractDeserializer.java:132) ~[jackson-databind-2.4.6.jar:2.4.6]
	at com.fasterxml.jackson.databind.deser.SettableBeanProperty.deserialize(SettableBeanProperty.java:536) ~[jackson-databind-2.4.6.jar:2.4.6]
	at com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeUsingPropertyBased(BeanDeserializer.java:344) ~[jackson-databind-2.4.6.jar:2.4.6]
	at com.fasterxml.jackson.databind.deser.BeanDeserializerBase.deserializeFromObjectUsingNonDefault(BeanDeserializerBase.java:1064) ~[jackson-databind-2.4.6.jar:2.4.6]
	at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserializeFromObject(BeanDeserializer.java:264) ~[jackson-databind-2.4.6.jar:2.4.6]
	at com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeOther(BeanDeserializer.java:156) ~[jackson-databind-2.4.6.jar:2.4.6]
	at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:126) ~[jackson-databind-2.4.6.jar:2.4.6]
	at com.fasterxml.jackson.databind.jsontype.impl.AsPropertyTypeDeserializer._deserializeTypedForId(AsPropertyTypeDeserializer.java:113) ~[jackson-databind-2.4.6.jar:2.4.6]
	at com.fasterxml.jackson.databind.jsontype.impl.AsPropertyTypeDeserializer.deserializeTypedFromObject(AsPropertyTypeDeserializer.java:84) ~[jackson-databind-2.4.6.jar:2.4.6]
	at com.fasterxml.jackson.databind.deser.AbstractDeserializer.deserializeWithType(AbstractDeserializer.java:132) ~[jackson-databind-2.4.6.jar:2.4.6]
	at com.fasterxml.jackson.databind.deser.impl.TypeWrappedDeserializer.deserialize(TypeWrappedDeserializer.java:41) ~[jackson-databind-2.4.6.jar:2.4.6]
	at com.fasterxml.jackson.databind.ObjectMapper._convert(ObjectMapper.java:2769) ~[jackson-databind-2.4.6.jar:2.4.6]
	... 10 more
Caused by: java.lang.IllegalArgumentException: If columns field is not set, the first row of your data must have your header and hasHeaderRow must be set to true.
	at com.google.common.base.Preconditions.checkArgument(Preconditions.java:125) ~[guava-16.0.1.jar:?]
	at io.druid.data.input.impl.DelimitedParseSpec.<init>(DelimitedParseSpec.java:65) ~[druid-api-0.10.1.2.6.4.0-91.jar:0.10.1.2.6.4.0-91]
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:1.8.0_161]
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:1.8.0_161]
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:1.8.0_161]
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_161]
	at com.fasterxml.jackson.databind.introspect.AnnotatedConstructor.call(AnnotatedConstructor.java:125) ~[jackson-databind-2.4.6.jar:2.4.6]
	at com.fasterxml.jackson.databind.deser.std.StdValueInstantiator.createFromObjectWith(StdValueInstantiator.java:230) ~[jackson-databind-2.4.6.jar:2.4.6]
	at com.fasterxml.jackson.databind.deser.impl.PropertyBasedCreator.build(PropertyBasedCreator.java:167) ~[jackson-databind-2.4.6.jar:2.4.6]
	at com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeUsingPropertyBased(BeanDeserializer.java:398) ~[jackson-databind-2.4.6.jar:2.4.6]
	at com.fasterxml.jackson.databind.deser.BeanDeserializerBase.deserializeFromObjectUsingNonDefault(BeanDeserializerBase.java:1064) ~[jackson-databind-2.4.6.jar:2.4.6]
	at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserializeFromObject(BeanDeserializer.java:264) ~[jackson-databind-2.4.6.jar:2.4.6]
	at com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeOther(BeanDeserializer.java:156) ~[jackson-databind-2.4.6.jar:2.4.6]
	at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:126) ~[jackson-databind-2.4.6.jar:2.4.6]
	at com.fasterxml.jackson.databind.jsontype.impl.AsPropertyTypeDeserializer._deserializeTypedForId(AsPropertyTypeDeserializer.java:113) ~[jackson-databind-2.4.6.jar:2.4.6]
	at com.fasterxml.jackson.databind.jsontype.impl.AsPropertyTypeDeserializer.deserializeTypedFromObject(AsPropertyTypeDeserializer.java:84) ~[jackson-databind-2.4.6.jar:2.4.6]
	at com.fasterxml.jackson.databind.deser.AbstractDeserializer.deserializeWithType(AbstractDeserializer.java:132) ~[jackson-databind-2.4.6.jar:2.4.6]
	at com.fasterxml.jackson.databind.deser.SettableBeanProperty.deserialize(SettableBeanProperty.java:536) ~[jackson-databind-2.4.6.jar:2.4.6]
	at com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeUsingPropertyBased(BeanDeserializer.java:344) ~[jackson-databind-2.4.6.jar:2.4.6]
	at com.fasterxml.jackson.databind.deser.BeanDeserializerBase.deserializeFromObjectUsingNonDefault(BeanDeserializerBase.java:1064) ~[jackson-databind-2.4.6.jar:2.4.6]
	at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserializeFromObject(BeanDeserializer.java:264) ~[jackson-databind-2.4.6.jar:2.4.6]
	at com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeOther(BeanDeserializer.java:156) ~[jackson-databind-2.4.6.jar:2.4.6]
	at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:126) ~[jackson-databind-2.4.6.jar:2.4.6]
	at com.fasterxml.jackson.databind.jsontype.impl.AsPropertyTypeDeserializer._deserializeTypedForId(AsPropertyTypeDeserializer.java:113) ~[jackson-databind-2.4.6.jar:2.4.6]
	at com.fasterxml.jackson.databind.jsontype.impl.AsPropertyTypeDeserializer.deserializeTypedFromObject(AsPropertyTypeDeserializer.java:84) ~[jackson-databind-2.4.6.jar:2.4.6]
	at com.fasterxml.jackson.databind.deser.AbstractDeserializer.deserializeWithType(AbstractDeserializer.java:132) ~[jackson-databind-2.4.6.jar:2.4.6]
	at com.fasterxml.jackson.databind.deser.impl.TypeWrappedDeserializer.deserialize(TypeWrappedDeserializer.java:41) ~[jackson-databind-2.4.6.jar:2.4.6]
	at com.fasterxml.jackson.databind.ObjectMapper._convert(ObjectMapper.java:2769) ~[jackson-databind-2.4.6.jar:2.4.6]
	... 10 more
2018-03-29T14:09:31,887 INFO [task-runner-0-priority-0] io.druid.indexing.overlord.TaskRunnerUtils - Task [index_usgs_2018-03-29T14:09:27.159Z] status changed to [FAILED].
2018-03-29T14:09:31,898 INFO [task-runner-0-priority-0] io.druid.indexing.worker.executor.ExecutorLifecycle - Task completed with status: {
  "id" : "index_usgs_2018-03-29T14:09:27.159Z",
  "status" : "FAILED",
  "duration" : 66
}
2018-03-29T14:09:32,015 INFO [main] com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory - Registering com.fasterxml.jackson.jaxrs.json.JacksonJsonProvider as a provider class
2018-03-29T14:09:32,015 INFO [main] com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory - Registering com.fasterxml.jackson.jaxrs.smile.JacksonSmileProvider as a provider class
2018-03-29T14:09:32,016 INFO [main] com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory - Registering io.druid.server.initialization.jetty.CustomExceptionMapper as a provider class
2018-03-29T14:09:32,016 INFO [main] com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory - Registering io.druid.server.StatusResource as a root resource class
2018-03-29T14:09:32,019 INFO [main] com.sun.jersey.server.impl.application.WebApplicationImpl - Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:43 PM'
2018-03-29T14:09:32,086 INFO [main] com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory - Binding io.druid.server.initialization.jetty.CustomExceptionMapper to GuiceManagedComponentProvider with the scope "Singleton"
2018-03-29T14:09:32,089 INFO [main] com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory - Binding com.fasterxml.jackson.jaxrs.json.JacksonJsonProvider to GuiceManagedComponentProvider with the scope "Singleton"
2018-03-29T14:09:32,096 INFO [main] com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory - Binding com.fasterxml.jackson.jaxrs.smile.JacksonSmileProvider to GuiceManagedComponentProvider with the scope "Singleton"
2018-03-29T14:09:32,368 INFO [main] com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory - Binding io.druid.server.http.security.StateResourceFilter to GuiceInstantiatedComponentProvider
2018-03-29T14:09:32,387 INFO [main] com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory - Binding io.druid.server.http.SegmentListerResource to GuiceInstantiatedComponentProvider
2018-03-29T14:09:32,398 INFO [main] com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory - Binding io.druid.server.QueryResource to GuiceInstantiatedComponentProvider
2018-03-29T14:09:32,400 INFO [main] com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory - Binding io.druid.segment.realtime.firehose.ChatHandlerResource to GuiceInstantiatedComponentProvider
2018-03-29T14:09:32,403 INFO [main] com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory - Binding io.druid.query.lookup.LookupListeningResource to GuiceInstantiatedComponentProvider
2018-03-29T14:09:32,405 INFO [main] com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory - Binding io.druid.query.lookup.LookupIntrospectionResource to GuiceInstantiatedComponentProvider
2018-03-29T14:09:32,406 INFO [main] com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory - Binding io.druid.server.StatusResource to GuiceManagedComponentProvider with the scope "Undefined"
2018-03-29T14:09:32,420 WARN [main] com.sun.jersey.spi.inject.Errors - The following warnings have been detected with resource and/or provider classes:
  WARNING: A HTTP GET method, public void io.druid.server.http.SegmentListerResource.getSegments(long,long,long,javax.servlet.http.HttpServletRequest) throws java.io.IOException, MUST return a non-void type.
2018-03-29T14:09:32,428 INFO [main] org.eclipse.jetty.server.handler.ContextHandler - Started o.e.j.s.ServletContextHandler@63de4fa{/,null,AVAILABLE}
2018-03-29T14:09:32,435 INFO [main] org.eclipse.jetty.server.AbstractConnector - Started ServerConnector@2fba0dac{HTTP/1.1,[http/1.1]}{0.0.0.0:8100}
2018-03-29T14:09:32,435 INFO [main] org.eclipse.jetty.server.Server - Started @5163ms
2018-03-29T14:09:32,436 INFO [main] io.druid.java.util.common.lifecycle.Lifecycle$AnnotationBasedHandler - Invoking start method[public void io.druid.server.listener.announcer.ListenerResourceAnnouncer.start()] on object[io.druid.query.lookup.LookupResourceListenerAnnouncer@7be38eba].
2018-03-29T14:09:32,450 INFO [main] io.druid.server.listener.announcer.ListenerResourceAnnouncer - Announcing start time on [/druid/listeners/lookups/__default/sandbox-hdp.hortonworks.com:8100]
2018-03-29T14:09:32,455 INFO [main] io.druid.java.util.common.lifecycle.Lifecycle$AnnotationBasedHandler - Invoking stop method[public void io.druid.server.listener.announcer.ListenerResourceAnnouncer.stop()] on object[io.druid.query.lookup.LookupResourceListenerAnnouncer@7be38eba].
2018-03-29T14:09:32,455 INFO [main] io.druid.curator.announcement.Announcer - unannouncing [/druid/listeners/lookups/__default/sandbox-hdp.hortonworks.com:8100]
2018-03-29T14:09:32,484 INFO [main] io.druid.server.listener.announcer.ListenerResourceAnnouncer - Unannouncing start time on [/druid/listeners/lookups/__default/sandbox-hdp.hortonworks.com:8100]
2018-03-29T14:09:32,488 INFO [main] org.eclipse.jetty.server.AbstractConnector - Stopped ServerConnector@2fba0dac{HTTP/1.1,[http/1.1]}{0.0.0.0:8100}
2018-03-29T14:09:32,489 INFO [main] org.eclipse.jetty.server.handler.ContextHandler - Stopped o.e.j.s.ServletContextHandler@63de4fa{/,null,UNAVAILABLE}
2018-03-29T14:09:32,496 INFO [main] io.druid.java.util.common.lifecycle.Lifecycle$AnnotationBasedHandler - Invoking stop method[public void io.druid.indexing.worker.executor.ExecutorLifecycle.stop() throws java.lang.Exception] on object[io.druid.indexing.worker.executor.ExecutorLifecycle@65eb3597].
2018-03-29T14:09:32,497 INFO [main] io.druid.java.util.common.lifecycle.Lifecycle$AnnotationBasedHandler - Invoking stop method[public void io.druid.indexing.overlord.ThreadPoolTaskRunner.stop()] on object[io.druid.indexing.overlord.ThreadPoolTaskRunner@37b1149b].
2018-03-29T14:09:32,497 INFO [main] io.druid.java.util.common.lifecycle.Lifecycle$AnnotationBasedHandler - Invoking stop method[public void io.druid.curator.discovery.ServerDiscoverySelector.stop() throws java.io.IOException] on object[io.druid.curator.discovery.ServerDiscoverySelector@646cd766].
2018-03-29T14:09:32,501 INFO [main] io.druid.java.util.common.lifecycle.Lifecycle$AnnotationBasedHandler - Invoking stop method[public void io.druid.curator.announcement.Announcer.stop()] on object[io.druid.curator.announcement.Announcer@1f129467].
2018-03-29T14:09:32,501 INFO [main] io.druid.java.util.common.lifecycle.Lifecycle$AnnotationBasedHandler - Invoking stop method[public void io.druid.curator.discovery.ServerDiscoverySelector.stop() throws java.io.IOException] on object[io.druid.curator.discovery.ServerDiscoverySelector@13cc0b90].
2018-03-29T14:09:32,501 INFO [main] io.druid.curator.CuratorModule - Stopping Curator
2018-03-29T14:09:32,502 INFO [Curator-Framework-0] org.apache.curator.framework.imps.CuratorFrameworkImpl - backgroundOperationsLoop exiting
2018-03-29T14:09:32,506 INFO [main] org.apache.zookeeper.ZooKeeper - Session: 0x16271e552860012 closed
2018-03-29T14:09:32,506 INFO [main-EventThread] org.apache.zookeeper.ClientCnxn - EventThread shut down
2018-03-29T14:09:32,506 INFO [main] com.metamx.common.lifecycle.Lifecycle$AnnotationBasedHandler - Invoking stop method[public void com.metamx.http.client.NettyHttpClient.stop()] on object[com.metamx.http.client.NettyHttpClient@36ad229b].
2018-03-29T14:09:32,527 INFO [main] io.druid.java.util.common.lifecycle.Lifecycle$AnnotationBasedHandler - Invoking stop method[public void io.druid.storage.hdfs.HdfsStorageAuthentication.stop()] on object[io.druid.storage.hdfs.HdfsStorageAuthentication@2e56b4d].
2018-03-29T14:09:32,527 INFO [main] io.druid.java.util.common.lifecycle.Lifecycle$AnnotationBasedHandler - Invoking stop method[public void com.metamx.metrics.MonitorScheduler.stop()] on object[com.metamx.metrics.MonitorScheduler@be164d8].
2018-03-29T14:09:32,528 INFO [main] io.druid.java.util.common.lifecycle.Lifecycle$AnnotationBasedHandler - Invoking stop method[public void io.druid.query.lookup.LookupReferencesManager.stop()] on object[io.druid.query.lookup.LookupReferencesManager@89caf47].
2018-03-29T14:09:32,528 INFO [main] io.druid.query.lookup.LookupReferencesManager - LookupReferencesManager is stopping.
2018-03-29T14:09:32,528 INFO [LookupReferencesManager-MainThread] io.druid.query.lookup.LookupReferencesManager - Lookup Management loop exited, Lookup notices are not handled anymore.
2018-03-29T14:09:32,529 INFO [main] io.druid.query.lookup.LookupReferencesManager - LookupReferencesManager is stopped.
2018-03-29T14:09:32,529 INFO [main] io.druid.java.util.common.lifecycle.Lifecycle$AnnotationBasedHandler - Invoking stop method[public void com.metamx.emitter.service.ServiceEmitter.close() throws java.io.IOException] on object[com.metamx.emitter.service.ServiceEmitter@5a090f62].
2018-03-29T14:09:32,529 INFO [main] io.druid.java.util.common.lifecycle.Lifecycle$AnnotationBasedHandler - Invoking stop method[public void io.druid.initialization.Log4jShutterDownerModule$Log4jShutterDowner.stop()] on object[io.druid.initialization.Log4jShutterDownerModule$Log4jShutterDowner@3dbd7107].
2018-03-29 14:09:32,567 pool-1-thread-1 ERROR Unable to register shutdown hook because JVM is shutting down. java.lang.IllegalStateException: Not started
	at io.druid.common.config.Log4jShutdown.addShutdownCallback(Log4jShutdown.java:46)
	at org.apache.logging.log4j.core.impl.Log4jContextFactory.addShutdownCallback(Log4jContextFactory.java:273)
	at org.apache.logging.log4j.core.LoggerContext.setUpShutdownHook(LoggerContext.java:256)
	at org.apache.logging.log4j.core.LoggerContext.start(LoggerContext.java:216)
	at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:145)
	at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:41)
	at org.apache.logging.log4j.LogManager.getContext(LogManager.java:182)
	at org.apache.logging.log4j.spi.AbstractLoggerAdapter.getContext(AbstractLoggerAdapter.java:103)
	at org.apache.logging.slf4j.Log4jLoggerFactory.getContext(Log4jLoggerFactory.java:43)
	at org.apache.logging.log4j.spi.AbstractLoggerAdapter.getLogger(AbstractLoggerAdapter.java:42)
	at org.apache.logging.slf4j.Log4jLoggerFactory.getLogger(Log4jLoggerFactory.java:29)
	at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:253)
	at org.apache.commons.logging.impl.SLF4JLogFactory.getInstance(SLF4JLogFactory.java:155)
	at org.apache.commons.logging.impl.SLF4JLogFactory.getInstance(SLF4JLogFactory.java:132)
	at org.apache.commons.logging.LogFactory.getLog(LogFactory.java:273)
	at org.apache.hadoop.hdfs.LeaseRenewer.<clinit>(LeaseRenewer.java:72)
	at org.apache.hadoop.hdfs.DFSClient.getLeaseRenewer(DFSClient.java:830)
	at org.apache.hadoop.hdfs.DFSClient.close(DFSClient.java:968)
	at org.apache.hadoop.hdfs.DistributedFileSystem.close(DistributedFileSystem.java:1214)
	at org.apache.hadoop.fs.FileSystem$Cache.closeAll(FileSystem.java:2886)
	at org.apache.hadoop.fs.FileSystem$Cache$ClientFinalizer.run(FileSystem.java:2903)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)


Any help would be very much appreciated.

1 ACCEPTED SOLUTION

Accepted Solutions
Highlighted

Re: Failing to Submit Index Task to Druid's Overlord via REST Service on HDP 2.6.4

New Contributor

@jmedel

The root cause of failure is invalid parse spec

java.lang.IllegalArgumentException: Instantiation of [simple type, class io.druid.data.input.impl.DelimitedParseSpec] value failed: If columns field is not set, the first row of your data must have your header and hasHeaderRow must be set to true.

Check if your input file has a header row or not.

If yes, set hasHeaderRow = true in parseSpec, otherwise you need to specify list of columns in your parseSpec so that druid knows about the columns present in the file.

3 REPLIES 3
Highlighted

Re: Failing to Submit Index Task to Druid's Overlord via REST Service on HDP 2.6.4

New Contributor

@jmedel

The root cause of failure is invalid parse spec

java.lang.IllegalArgumentException: Instantiation of [simple type, class io.druid.data.input.impl.DelimitedParseSpec] value failed: If columns field is not set, the first row of your data must have your header and hasHeaderRow must be set to true.

Check if your input file has a header row or not.

If yes, set hasHeaderRow = true in parseSpec, otherwise you need to specify list of columns in your parseSpec so that druid knows about the columns present in the file.

Re: Failing to Submit Index Task to Druid's Overlord via REST Service on HDP 2.6.4

Contributor

overlord-log.txtHi Nishant,

The input file doesn't have a header row.napa-flowtsv.txt

But, I utilized your suggestion about adding the "hasHeaderRow = true" and that caused the task to complete successfully.

Here is my latest Stack trace with the ERROR I noticed:

2018-03-29 15:54:37,879 pool-1-thread-1 ERROR Unable to register shutdown hook because JVM is shutting down. java.lang.IllegalStateException: Not started
	at io.druid.common.config.Log4jShutdown.addShutdownCallback(Log4jShutdown.java:46)
	at org.apache.logging.log4j.core.impl.Log4jContextFactory.addShutdownCallback(Log4jContextFactory.java:273)
	at org.apache.logging.log4j.core.LoggerContext.setUpShutdownHook(LoggerContext.java:256)
	at org.apache.logging.log4j.core.LoggerContext.start(LoggerContext.java:216)
	at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:145)
	at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:41)
	at org.apache.logging.log4j.LogManager.getContext(LogManager.java:182)
	at org.apache.logging.log4j.spi.AbstractLoggerAdapter.getContext(AbstractLoggerAdapter.java:103)
	at org.apache.logging.slf4j.Log4jLoggerFactory.getContext(Log4jLoggerFactory.java:43)
	at org.apache.logging.log4j.spi.AbstractLoggerAdapter.getLogger(AbstractLoggerAdapter.java:42)
	at org.apache.logging.slf4j.Log4jLoggerFactory.getLogger(Log4jLoggerFactory.java:29)
	at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:253)
	at org.apache.commons.logging.impl.SLF4JLogFactory.getInstance(SLF4JLogFactory.java:155)
	at org.apache.commons.logging.impl.SLF4JLogFactory.getInstance(SLF4JLogFactory.java:132)
	at org.apache.commons.logging.LogFactory.getLog(LogFactory.java:273)
	at org.apache.hadoop.hdfs.LeaseRenewer.<clinit>(LeaseRenewer.java:72)
	at org.apache.hadoop.hdfs.DFSClient.getLeaseRenewer(DFSClient.java:830)
	at org.apache.hadoop.hdfs.DFSClient.close(DFSClient.java:968)
	at org.apache.hadoop.hdfs.DistributedFileSystem.close(DistributedFileSystem.java:1214)
	at org.apache.hadoop.fs.FileSystem$Cache.closeAll(FileSystem.java:2886)
	at org.apache.hadoop.fs.FileSystem$Cache$ClientFinalizer.run(FileSystem.java:2903)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

Re: Failing to Submit Index Task to Druid's Overlord via REST Service on HDP 2.6.4

Contributor

Hi Nishant,

I have an update, so utiliizing your suggestion of adding the "columns": [ ... ]

solved my problem and now I am able to successfully ingest my data into Druid. For users who want to see the ingestion spec, I have included it below:

{
    "type" : "index_hadoop",
    "spec" : {
      "dataSchema" : {
        "dataSource" : "usgs",
        "parser" : {
          "type" : "hadoopyString",
          "parseSpec" : {
            "format" : "tsv",
            "timestampSpec" : {
              "column" : "dates",
              "format" : "auto"
            },
            "dimensionsSpec" : {
              "dimensions": ["staid","val","dates"],
              "dimensionExclusions" : [],
              "spatialDimensions" : []
            },
            "columns" : ["staid","val","dates"]
          }
        },
        "metricsSpec" : [
          {
            "type" : "count",
            "name" : "count"
          },
          {
            "type" : "doubleSum",
            "name" : "avgFlowCuFtsec",
            "fieldName" : "val"
          }
        ],
        "granularitySpec" : {
          "type" : "uniform",
          "segmentGranularity" : "MONTH",
          "queryGranularity" : "NONE",
          "intervals" : [ "1963-01-01/2013-12-31" ]
        }
      },
      "ioConfig" : {
        "type" : "hadoop",
        "inputSpec" : {
          "type" : "static",
          "paths" : "/tmp/druid/napa-flow.tsv.gz"
        }
      },
      "tuningConfig" : {
        "type": "hadoop",
        "targetPartitionSize" : 10000,
        "maxRowsInMemory" : 75000
      }
    }
}


Note: I will add the "index" ingestion spec soon too, so users can ingest data into Druid from their local file system or they can go with hadoop file system.