Support Questions

Find answers, ask questions, and share your expertise

Failing to Submit Index Task to Druid's Overlord via REST Service on HDP 2.6.4

Contributor

I am developing an HDP sandbox tutorial that teaches users how they can use Druid's Batch Ingestion Method to load local files into Druid. After I store the USGS data into a local file and submit an ingestion spec referring to the local file to Druid Overlord, I get that the task fails meaning the data didn't load into Druid.

Here is my batch ingestion spec:

{
    "type" : "index",
    "spec" : {
      "dataSchema" : {
        "dataSource" : "usgs",
        "parser" : {
          "type" : "string",
          "parseSpec" : {
            "format" : "tsv",
            "timestampSpec" : {
              "column" : "dates",
              "format" : "auto"
            },
            "dimensionsSpec" : {
              "dimensions": ["staid","val","dates"],
              "dimensionExclusions" : [],
              "spatialDimensions" : []
            }
          }
        },
        "metricsSpec" : [
          {
            "type" : "count",
            "name" : "count"
          },
          {
            "type" : "doubleSum",
            "name" : "avgFlowCuFtsec",
            "fieldName" : "val"
          }
        ],
        "granularitySpec" : {
          "type" : "uniform",
          "segmentGranularity" : "MONTH",
          "queryGranularity" : "NONE",
          "intervals" : [ "1963-01-01/2013-12-31" ]
        }
      },
      "ioConfig" : {
        "type" : "index",
        "firehose" : {
          "type" : "local",
          "baseDir" : "/tmp/usgs/",
          "filter" : "napa-flow.tsv"
        }
      },
      "tuningConfig" : {
        "type": "index",
        "targetPartitionSize" : 10000,
        "maxRowsInMemory" : 75000
      }
    }
  }

Here is the stack trace from Druid's Overlord:

2018-03-29T14:09:28,013 INFO [main] io.druid.guice.PropertiesModule - Loading properties from common.runtime.properties
2018-03-29T14:09:28,015 INFO [main] io.druid.guice.PropertiesModule - Loading properties from runtime.properties
2018-03-29T14:09:28,042 INFO [main] org.hibernate.validator.internal.util.Version - HV000001: Hibernate Validator 5.1.3.Final
2018-03-29T14:09:28,411 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.segment.NullValueHandlingConfig] from props[druid.null.handling.] as [io.druid.segment.NullValueHandlingConfig@6995bf68]
2018-03-29T14:09:28,460 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.guice.ExtensionsConfig] from props[druid.extensions.] as [ExtensionsConfig{searchCurrentClassloader=true, directory='/usr/hdp/current/druid-middlemanager/extensions', hadoopDependenciesDir='/usr/hdp/current/druid-middlemanager/hadoop-dependencies', hadoopContainerDruidClasspath='null', addExtensionsToHadoopContainer=false, loadList=[druid-datasketches, druid-hdfs-storage, druid-kafka-indexing-service]}]
2018-03-29T14:09:28,510 INFO [main] io.druid.initialization.Initialization - Loading extension [druid-datasketches] for class [interface io.druid.cli.CliCommandCreator]
2018-03-29T14:09:28,518 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-datasketches/druid-datasketches-0.10.1.2.6.4.0-91.jar]
2018-03-29T14:09:28,518 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-datasketches/sketches-core-0.8.4.jar]
2018-03-29T14:09:28,518 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-datasketches/memory-0.8.4.jar]
2018-03-29T14:09:28,520 INFO [main] io.druid.initialization.Initialization - Loading extension [druid-hdfs-storage] for class [interface io.druid.cli.CliCommandCreator]
2018-03-29T14:09:28,522 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/hadoop-common-2.7.3.2.6.4.0-91.jar]
2018-03-29T14:09:28,522 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/okio-1.4.0.jar]
2018-03-29T14:09:28,522 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/curator-client-2.11.0.jar]
2018-03-29T14:09:28,522 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/commons-beanutils-1.7.0.jar]
2018-03-29T14:09:28,522 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/jsp-api-2.1.jar]
2018-03-29T14:09:28,522 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/snappy-java-1.0.4.1.jar]
2018-03-29T14:09:28,523 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/stax-api-1.0-2.jar]
2018-03-29T14:09:28,523 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/servlet-api-2.5.jar]
2018-03-29T14:09:28,523 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/hadoop-client-2.7.3.2.6.4.0-91.jar]
2018-03-29T14:09:28,523 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/hadoop-mapreduce-client-jobclient-2.7.3.2.6.4.0-91.jar]
2018-03-29T14:09:28,523 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/okhttp-2.4.0.jar]
2018-03-29T14:09:28,523 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/commons-logging-1.1.1.jar]
2018-03-29T14:09:28,523 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/api-util-1.0.0-M20.jar]
2018-03-29T14:09:28,524 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/apacheds-i18n-2.0.0-M15.jar]
2018-03-29T14:09:28,524 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/curator-recipes-2.11.0.jar]
2018-03-29T14:09:28,524 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/netty-all-4.1.11.Final.jar]
2018-03-29T14:09:28,524 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/aws-java-sdk-kms-1.10.77.jar]
2018-03-29T14:09:28,524 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/druid-hdfs-storage-0.10.1.2.6.4.0-91.jar]
2018-03-29T14:09:28,525 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/nimbus-jose-jwt-3.9.jar]
2018-03-29T14:09:28,525 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/xmlenc-0.52.jar]
2018-03-29T14:09:28,525 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/htrace-core-3.1.0-incubating.jar]
2018-03-29T14:09:28,525 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/xz-1.0.jar]
2018-03-29T14:09:28,525 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/commons-beanutils-core-1.8.0.jar]
2018-03-29T14:09:28,525 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/gson-2.2.4.jar]
2018-03-29T14:09:28,526 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/jackson-core-2.4.6.jar]
2018-03-29T14:09:28,526 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/azure-storage-5.4.0.jar]
2018-03-29T14:09:28,526 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/curator-framework-2.11.0.jar]
2018-03-29T14:09:28,526 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/hadoop-mapreduce-client-core-2.7.3.2.6.4.0-91.jar]
2018-03-29T14:09:28,526 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/jackson-xc-1.9.13.jar]
2018-03-29T14:09:28,526 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/commons-configuration-1.6.jar]
2018-03-29T14:09:28,526 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/jackson-dataformat-cbor-2.5.3.jar]
2018-03-29T14:09:28,526 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/hadoop-yarn-api-2.7.3.2.6.4.0-91.jar]
2018-03-29T14:09:28,527 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/hadoop-yarn-client-2.7.3.2.6.4.0-91.jar]
2018-03-29T14:09:28,527 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/aws-java-sdk-core-1.10.77.jar]
2018-03-29T14:09:28,527 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/hadoop-hdfs-2.7.3.2.6.4.0-91.jar]
2018-03-29T14:09:28,527 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/hadoop-yarn-registry-2.7.3.2.6.4.0-91.jar]
2018-03-29T14:09:28,527 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/azure-keyvault-core-0.8.0.jar]
2018-03-29T14:09:28,527 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/hadoop-yarn-common-2.7.3.2.6.4.0-91.jar]
2018-03-29T14:09:28,527 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/api-asn1-api-1.0.0-M20.jar]
2018-03-29T14:09:28,528 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/hadoop-yarn-server-common-2.7.3.2.6.4.0-91.jar]
2018-03-29T14:09:28,528 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/commons-lang3-3.4.jar]
2018-03-29T14:09:28,528 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/hadoop-auth-2.7.3.2.6.4.0-91.jar]
2018-03-29T14:09:28,528 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/commons-net-3.1.jar]
2018-03-29T14:09:28,528 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/paranamer-2.3.jar]
2018-03-29T14:09:28,528 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/jersey-client-1.9.jar]
2018-03-29T14:09:28,528 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/hadoop-mapreduce-client-common-2.7.3.2.6.4.0-91.jar]
2018-03-29T14:09:28,529 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/avro-1.7.4.jar]
2018-03-29T14:09:28,529 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/jackson-databind-2.4.6.jar]
2018-03-29T14:09:28,529 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/jaxb-api-2.2.2.jar]
2018-03-29T14:09:28,529 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/jackson-annotations-2.4.6.jar]
2018-03-29T14:09:28,529 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/httpclient-4.5.1.jar]
2018-03-29T14:09:28,529 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/jackson-jaxrs-1.9.13.jar]
2018-03-29T14:09:28,529 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/xml-apis-1.3.04.jar]
2018-03-29T14:09:28,530 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/commons-compress-1.4.1.jar]
2018-03-29T14:09:28,530 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/xercesImpl-2.9.1.jar]
2018-03-29T14:09:28,530 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/joda-time-2.9.9.jar]
2018-03-29T14:09:28,530 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/jetty-sslengine-6.1.26.hwx.jar]
2018-03-29T14:09:28,530 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/commons-digester-1.8.jar]
2018-03-29T14:09:28,530 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/hadoop-aws-2.7.3.2.6.4.0-91.jar]
2018-03-29T14:09:28,530 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/json-smart-1.1.1.jar]
2018-03-29T14:09:28,530 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/apacheds-kerberos-codec-2.0.0-M15.jar]
2018-03-29T14:09:28,531 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/netty-3.6.2.Final.jar]
2018-03-29T14:09:28,531 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/aws-java-sdk-s3-1.10.77.jar]
2018-03-29T14:09:28,531 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/httpcore-4.4.3.jar]
2018-03-29T14:09:28,531 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/guava-16.0.1.jar]
2018-03-29T14:09:28,531 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/hadoop-mapreduce-client-shuffle-2.7.3.2.6.4.0-91.jar]
2018-03-29T14:09:28,531 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/commons-collections-3.2.2.jar]
2018-03-29T14:09:28,531 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/hadoop-mapreduce-client-app-2.7.3.2.6.4.0-91.jar]
2018-03-29T14:09:28,531 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/jcip-annotations-1.0.jar]
2018-03-29T14:09:28,532 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/commons-math3-3.6.1.jar]
2018-03-29T14:09:28,532 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-hdfs-storage/leveldbjni-all-1.8.jar]
2018-03-29T14:09:28,547 INFO [main] io.druid.initialization.Initialization - Loading extension [druid-kafka-indexing-service] for class [interface io.druid.cli.CliCommandCreator]
2018-03-29T14:09:28,547 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-kafka-indexing-service/snappy-java-1.1.2.6.jar]
2018-03-29T14:09:28,547 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-kafka-indexing-service/slf4j-api-1.6.4.jar]
2018-03-29T14:09:28,547 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-kafka-indexing-service/druid-kafka-indexing-service-0.10.1.2.6.4.0-91.jar]
2018-03-29T14:09:28,548 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-kafka-indexing-service/lz4-1.3.0.jar]
2018-03-29T14:09:28,548 INFO [main] io.druid.initialization.Initialization - added URL[file:/usr/hdp/current/druid-middlemanager/extensions/druid-kafka-indexing-service/kafka-clients-0.10.2.0.jar]
2018-03-29T14:09:28,616 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.guice.ModulesConfig] from props[druid.modules.] as [ModulesConfig{excludeList=[]}]
2018-03-29T14:09:28,772 INFO [main] io.druid.initialization.Initialization - Loading extension [druid-datasketches] for class [interface io.druid.initialization.DruidModule]
2018-03-29T14:09:28,775 INFO [main] io.druid.initialization.Initialization - Adding implementation [io.druid.query.aggregation.datasketches.theta.SketchModule] for class [interface io.druid.initialization.DruidModule] from local file system extension
2018-03-29T14:09:28,775 INFO [main] io.druid.initialization.Initialization - Adding implementation [io.druid.query.aggregation.datasketches.theta.oldapi.OldApiSketchModule] for class [interface io.druid.initialization.DruidModule] from local file system extension
2018-03-29T14:09:28,776 INFO [main] io.druid.initialization.Initialization - Loading extension [druid-hdfs-storage] for class [interface io.druid.initialization.DruidModule]
2018-03-29T14:09:28,778 INFO [main] io.druid.initialization.Initialization - Adding implementation [io.druid.storage.hdfs.HdfsStorageDruidModule] for class [interface io.druid.initialization.DruidModule] from local file system extension
2018-03-29T14:09:28,778 INFO [main] io.druid.initialization.Initialization - Loading extension [druid-kafka-indexing-service] for class [interface io.druid.initialization.DruidModule]
2018-03-29T14:09:28,779 INFO [main] io.druid.initialization.Initialization - Adding implementation [io.druid.indexing.kafka.KafkaIndexTaskModule] for class [interface io.druid.initialization.DruidModule] from local file system extension
2018-03-29T14:09:29,499 WARN [main] org.apache.hadoop.util.NativeCodeLoader - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2018-03-29T14:09:30,292 WARN [main] org.apache.hadoop.hdfs.shortcircuit.DomainSocketFactory - The short-circuit local reads feature cannot be used because libhadoop cannot be loaded.
2018-03-29T14:09:30,672 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.server.metrics.DruidMonitorSchedulerConfig] from props[druid.monitoring.] as [io.druid.server.metrics.DruidMonitorSchedulerConfig@24eeac69]
2018-03-29T14:09:30,679 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.server.metrics.MonitorsConfig] from props[druid.monitoring.] as [MonitorsConfig{monitors=[class com.metamx.metrics.JvmMonitor]}]
2018-03-29T14:09:30,698 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.server.DruidNode] from props[druid.] as [DruidNode{serviceName='druid/middlemanager', host='sandbox-hdp.hortonworks.com', port=8100}]
2018-03-29T14:09:30,774 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.query.lookup.LookupConfig] from props[druid.lookup.] as [LookupConfig{snapshotWorkingDir=''}]
2018-03-29T14:09:30,844 INFO [main] io.druid.server.metrics.MetricsModule - Adding monitor[com.metamx.metrics.JvmMonitor@485caa8f]
2018-03-29T14:09:30,847 INFO [main] io.druid.server.metrics.MetricsModule - Adding monitor[io.druid.query.ExecutorServiceMonitor@4d157493]
2018-03-29T14:09:30,848 INFO [main] io.druid.server.metrics.MetricsModule - Adding monitor[io.druid.server.initialization.jetty.JettyServerModule$JettyMonitor@76e3b45b]
2018-03-29T14:09:30,854 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.server.log.StartupLoggingConfig] from props[druid.startup.logging.] as [io.druid.server.log.StartupLoggingConfig@3e28fee1]
2018-03-29T14:09:30,854 INFO [main] io.druid.cli.CliPeon - Starting up with processors[4], memory[308,281,344], maxMemory[1,908,932,608].
2018-03-29T14:09:30,857 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.storage.hdfs.HdfsKerberosConfig] from props[druid.hadoop.security.kerberos.] as [io.druid.storage.hdfs.HdfsKerberosConfig@0]
2018-03-29T14:09:30,864 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.indexing.common.config.TaskConfig] from props[druid.indexer.task.] as [io.druid.indexing.common.config.TaskConfig@2eda072]
2018-03-29T14:09:30,867 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.guice.http.DruidHttpClientConfig] from props[druid.global.http.] as [io.druid.guice.http.DruidHttpClientConfig@4091b9c3]
2018-03-29T14:09:30,928 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.client.indexing.IndexingServiceSelectorConfig] from props[druid.selectors.indexing.] as [io.druid.client.indexing.IndexingServiceSelectorConfig@7e7e962d]
2018-03-29T14:09:30,931 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.curator.CuratorConfig] from props[druid.zk.service.] as [io.druid.curator.CuratorConfig@7499eac7]
2018-03-29T14:09:30,934 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.curator.ExhibitorConfig] from props[druid.exhibitor.service.] as [io.druid.curator.ExhibitorConfig@d88f893]
2018-03-29T14:09:30,938 WARN [main] org.apache.curator.retry.ExponentialBackoffRetry - maxRetries too large (30). Pinning to 29
2018-03-29T14:09:30,959 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.server.initialization.CuratorDiscoveryConfig] from props[druid.discovery.curator.] as [io.druid.server.initialization.CuratorDiscoveryConfig@5b000fe6]
2018-03-29T14:09:30,980 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.indexing.common.RetryPolicyConfig] from props[druid.peon.taskActionClient.retry.] as [io.druid.indexing.common.RetryPolicyConfig@7186b202]
2018-03-29T14:09:30,982 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.storage.hdfs.HdfsDataSegmentPusherConfig] from props[druid.storage.] as [io.druid.storage.hdfs.HdfsDataSegmentPusherConfig@1a538ed8]
2018-03-29T14:09:30,983 INFO [main] io.druid.storage.hdfs.HdfsDataSegmentPusher - Configured HDFS as deep storage
2018-03-29T14:09:30,986 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.segment.loading.LocalDataSegmentPusherConfig] from props[druid.storage.] as [io.druid.segment.loading.LocalDataSegmentPusherConfig@270a620]
2018-03-29T14:09:30,991 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.client.DruidServerConfig] from props[druid.server.] as [io.druid.client.DruidServerConfig@4dfe8b37]
2018-03-29T14:09:30,994 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.server.initialization.BatchDataSegmentAnnouncerConfig] from props[druid.announcer.] as [io.druid.server.initialization.BatchDataSegmentAnnouncerConfig@3c232051]
2018-03-29T14:09:30,996 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.server.initialization.ZkPathsConfig] from props[druid.zk.paths.] as [io.druid.server.initialization.ZkPathsConfig@22e2266d]
2018-03-29T14:09:31,002 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[interface io.druid.server.coordination.DataSegmentAnnouncerProvider] from props[druid.announcer.] as [io.druid.server.coordination.BatchDataSegmentAnnouncerProvider@59fc6d05]
2018-03-29T14:09:31,004 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.client.coordinator.CoordinatorSelectorConfig] from props[druid.selectors.coordinator.] as [io.druid.client.coordinator.CoordinatorSelectorConfig@7e87ef9e]
2018-03-29T14:09:31,007 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.segment.realtime.plumber.CoordinatorBasedSegmentHandoffNotifierConfig] from props[druid.segment.handoff.] as [io.druid.segment.realtime.plumber.CoordinatorBasedSegmentHandoffNotifierConfig@7c241a9f]
2018-03-29T14:09:31,009 INFO [main] org.skife.config.ConfigurationObjectFactory - Assigning value [50000000] for [druid.processing.buffer.sizeBytes] on [io.druid.query.DruidProcessingConfig#intermediateComputeSizeBytes()]
2018-03-29T14:09:31,011 INFO [main] org.skife.config.ConfigurationObjectFactory - Using method itself for [druid.computation.buffer.poolCacheMaxCount, ${base_path}.buffer.poolCacheMaxCount] on [io.druid.query.DruidProcessingConfig#poolCacheMaxCount()]
2018-03-29T14:09:31,012 INFO [main] org.skife.config.ConfigurationObjectFactory - Using method itself for [${base_path}.fifo] on [io.druid.query.DruidProcessingConfig#isFifo()]
2018-03-29T14:09:31,012 INFO [main] org.skife.config.ConfigurationObjectFactory - Assigning value [2] for [druid.processing.numThreads] on [io.druid.query.DruidProcessingConfig#getNumThreadsConfigured()]
2018-03-29T14:09:31,012 INFO [main] org.skife.config.ConfigurationObjectFactory - Using method itself for [${base_path}.numMergeBuffers] on [io.druid.query.DruidProcessingConfig#getNumMergeBuffersConfigured()]
2018-03-29T14:09:31,013 INFO [main] org.skife.config.ConfigurationObjectFactory - Using method itself for [${base_path}.columnCache.sizeBytes] on [io.druid.query.DruidProcessingConfig#columnCacheSizeBytes()]
2018-03-29T14:09:31,013 INFO [main] org.skife.config.ConfigurationObjectFactory - Using method itself for [${base_path}.tmpDir] on [io.druid.query.DruidProcessingConfig#getTmpDir()]
2018-03-29T14:09:31,013 INFO [main] org.skife.config.ConfigurationObjectFactory - Assigning default value [processing-%s] for [${base_path}.formatString] on [io.druid.java.util.common.concurrent.ExecutorServiceConfig#getFormatString()]
2018-03-29T14:09:31,083 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[interface io.druid.client.cache.CacheProvider] from props[druid.cache.] as [io.druid.client.cache.LocalCacheProvider@182e7eda]
2018-03-29T14:09:31,091 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.client.cache.CacheConfig] from props[druid.realtime.cache.] as [io.druid.client.cache.CacheConfig@27dc627a]
2018-03-29T14:09:31,099 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.server.initialization.ServerConfig] from props[druid.server.http.] as [ServerConfig{numThreads=50, maxIdleTime=PT5M, defaultQueryTimeout=300000, maxScatterGatherBytes=9223372036854775807}]
2018-03-29T14:09:31,107 INFO [main] org.eclipse.jetty.util.log - Logging initialized @3834ms
2018-03-29T14:09:31,221 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.query.search.search.SearchQueryConfig] from props[druid.query.search.] as [io.druid.query.search.search.SearchQueryConfig@622d7e4]
2018-03-29T14:09:31,226 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.query.metadata.SegmentMetadataQueryConfig] from props[druid.query.segmentMetadata.] as [io.druid.query.metadata.SegmentMetadataQueryConfig@26582ca]
2018-03-29T14:09:31,229 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.query.groupby.GroupByQueryConfig] from props[druid.query.groupBy.] as [GroupByQueryConfig{defaultStrategy='v2', singleThreaded=false, maxIntermediateRows=50000, maxResults=500000, bufferGrouperMaxSize=2147483647, bufferGrouperMaxLoadFactor=0.0, bufferGrouperInitialBuckets=0, maxMergingDictionarySize=100000000, maxOnDiskStorage=0, forcePushDownLimit=false, forceHashAggregation=false}]
2018-03-29T14:09:31,244 INFO [main] io.druid.offheap.OffheapBufferGenerator - Allocating new intermediate processing buffer[0] of size[50,000,000]
2018-03-29T14:09:31,254 INFO [main] io.druid.offheap.OffheapBufferGenerator - Allocating new intermediate processing buffer[1] of size[50,000,000]
2018-03-29T14:09:31,263 INFO [main] io.druid.offheap.OffheapBufferGenerator - Allocating new result merging buffer[0] of size[50,000,000]
2018-03-29T14:09:31,314 INFO [main] io.druid.offheap.OffheapBufferGenerator - Allocating new result merging buffer[1] of size[50,000,000]
2018-03-29T14:09:31,370 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.query.scan.ScanQueryConfig] from props[druid.query.scan.] as [ScanQueryConfig{legacy=false}]
2018-03-29T14:09:31,373 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.query.select.SelectQueryConfig] from props[druid.query.select.] as [io.druid.query.select.SelectQueryConfig@707865bd]
2018-03-29T14:09:31,375 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.query.topn.TopNQueryConfig] from props[druid.query.topN.] as [io.druid.query.topn.TopNQueryConfig@3ebc955b]
2018-03-29T14:09:31,379 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[interface io.druid.server.log.RequestLoggerProvider] from props[druid.request.logging.] as [io.druid.server.log.NoopRequestLoggerProvider@6ffd4c0d]
2018-03-29T14:09:31,380 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.server.security.AuthConfig] from props[druid.auth.] as [AuthConfig{enabled=false}]
2018-03-29T14:09:31,388 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.query.lookup.LookupListeningAnnouncerConfig] from props[druid.lookup.] as [ListeningAnnouncerConfig{listenersPath='/druid/listeners'}]
2018-03-29T14:09:31,389 INFO [main] io.druid.java.util.common.lifecycle.Lifecycle$AnnotationBasedHandler - Invoking start method[public void io.druid.initialization.Log4jShutterDownerModule$Log4jShutterDowner.start()] on object[io.druid.initialization.Log4jShutterDownerModule$Log4jShutterDowner@3dbd7107].
2018-03-29T14:09:31,391 INFO [main] io.druid.java.util.common.lifecycle.Lifecycle$AnnotationBasedHandler - Invoking start method[public void com.metamx.emitter.service.ServiceEmitter.start()] on object[com.metamx.emitter.service.ServiceEmitter@5a090f62].
2018-03-29T14:09:31,392 INFO [main] io.druid.java.util.common.lifecycle.Lifecycle$AnnotationBasedHandler - Invoking start method[public void io.druid.query.lookup.LookupReferencesManager.start()] on object[io.druid.query.lookup.LookupReferencesManager@89caf47].
2018-03-29T14:09:31,393 INFO [main] io.druid.query.lookup.LookupReferencesManager - LookupReferencesManager is starting.
2018-03-29T14:09:31,393 INFO [main] io.druid.query.lookup.LookupReferencesManager - LookupReferencesManager is started.
2018-03-29T14:09:31,394 INFO [main] io.druid.java.util.common.lifecycle.Lifecycle$AnnotationBasedHandler - Invoking start method[public void com.metamx.metrics.MonitorScheduler.start()] on object[com.metamx.metrics.MonitorScheduler@be164d8].
2018-03-29T14:09:31,403 INFO [main] io.druid.java.util.common.lifecycle.Lifecycle$AnnotationBasedHandler - Invoking start method[public void io.druid.storage.hdfs.HdfsStorageAuthentication.authenticate()] on object[io.druid.storage.hdfs.HdfsStorageAuthentication@2e56b4d].
2018-03-29T14:09:31,405 INFO [main] com.metamx.common.lifecycle.Lifecycle$AnnotationBasedHandler - Invoking start method[public void com.metamx.http.client.NettyHttpClient.start()] on object[com.metamx.http.client.NettyHttpClient@36ad229b].
2018-03-29T14:09:31,405 INFO [main] io.druid.curator.CuratorModule - Starting Curator
2018-03-29T14:09:31,405 INFO [main] org.apache.curator.framework.imps.CuratorFrameworkImpl - Starting
2018-03-29T14:09:31,413 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:zookeeper.version=3.4.6-91--1, built on 01/04/2018 09:27 GMT
2018-03-29T14:09:31,413 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:host.name=sandbox-hdp.hortonworks.com
2018-03-29T14:09:31,413 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:java.version=1.8.0_161
2018-03-29T14:09:31,413 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:java.vendor=Oracle Corporation
2018-03-29T14:09:31,413 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:java.home=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.161-3.b14.el6_9.x86_64/jre
2018-03-29T14:09:31,413 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:java.class.path=/usr/hdp/current/druid-middlemanager/conf/_common:/usr/hdp/current/druid-middlemanager/conf/middleManager:/usr/hdp/current/druid-middlemanager/lib/antlr4-runtime-4.5.1.jar:/usr/hdp/current/druid-middlemanager/lib/druid-sql-0.10.1.2.6.4.0-91.jar:/usr/hdp/current/druid-middlemanager/lib/plexus-interpolation-1.19.jar:/usr/hdp/current/druid-middlemanager/lib/guice-servlet-4.1.0.jar:/usr/hdp/current/druid-middlemanager/lib/jackson-jaxrs-smile-provider-2.4.6.jar:/usr/hdp/current/druid-middlemanager/lib/curator-client-2.11.0.jar:/usr/hdp/current/druid-middlemanager/lib/tesla-aether-0.0.5.jar:/usr/hdp/current/druid-middlemanager/lib/wagon-provider-api-2.4.jar:/usr/hdp/current/druid-middlemanager/lib/extendedset-0.10.1.2.6.4.0-91.jar:/usr/hdp/current/druid-middlemanager/lib/jackson-dataformat-smile-2.4.6.jar:/usr/hdp/current/druid-middlemanager/lib/maven-model-3.1.1.jar:/usr/hdp/current/druid-middlemanager/lib/json-path-2.1.0.jar:/usr/hdp/current/druid-middlemanager/lib/jetty-util-9.3.19.v20170502.jar:/usr/hdp/current/druid-middlemanager/lib/commons-compiler-2.7.6.jar:/usr/hdp/current/druid-middlemanager/lib/log4j-api-2.5.jar:/usr/hdp/current/druid-middlemanager/lib/jcl-over-slf4j-1.7.12.jar:/usr/hdp/current/druid-middlemanager/lib/druid-server-0.10.1.2.6.4.0-91.jar:/usr/hdp/current/druid-middlemanager/lib/jetty-proxy-9.3.19.v20170502.jar:/usr/hdp/current/druid-middlemanager/lib/commons-pool-1.6.jar:/usr/hdp/current/druid-middlemanager/lib/jsr311-api-1.1.1.jar:/usr/hdp/current/druid-middlemanager/lib/jsr305-2.0.1.jar:/usr/hdp/current/druid-middlemanager/lib/druid-indexing-service-0.10.1.2.6.4.0-91.jar:/usr/hdp/current/druid-middlemanager/lib/commons-logging-1.1.1.jar:/usr/hdp/current/druid-middlemanager/lib/okhttp-1.0.2.jar:/usr/hdp/current/druid-middlemanager/lib/RoaringBitmap-0.5.18.jar:/usr/hdp/current/druid-middlemanager/lib/http-client-1.1.0.jar:/usr/hdp/current/druid-middlemanager/lib/curator-recipes-2.11.0.jar:/usr/hdp/current/druid-middlemanager/lib/aopalliance-1.0.jar:/usr/hdp/current/druid-middlemanager/lib/netty-all-4.1.11.Final.jar:/usr/hdp/current/druid-middlemanager/lib/derbyclient-10.11.1.1.jar:/usr/hdp/current/druid-middlemanager/lib/bytebuffer-collections-0.10.1.2.6.4.0-91.jar:/usr/hdp/current/druid-middlemanager/lib/protobuf-java-3.1.0.jar:/usr/hdp/current/druid-middlemanager/lib/aether-spi-0.9.0.M2.jar:/usr/hdp/current/druid-middlemanager/lib/derbynet-10.11.1.1.jar:/usr/hdp/current/druid-middlemanager/lib/compress-lzf-1.0.3.jar:/usr/hdp/current/druid-middlemanager/lib/disruptor-3.3.6.jar:/usr/hdp/current/druid-middlemanager/lib/jetty-http-9.3.19.v20170502.jar:/usr/hdp/current/druid-middlemanager/lib/maven-settings-3.1.1.jar:/usr/hdp/current/druid-middlemanager/lib/bcprov-jdk15on-1.52.jar:/usr/hdp/current/druid-middlemanager/lib/classmate-1.0.0.jar:/usr/hdp/current/druid-middlemanager/lib/opencsv-2.3.jar:/usr/hdp/current/druid-middlemanager/lib/druid-aws-common-0.10.1.2.6.4.0-91.jar:/usr/hdp/current/druid-middlemanager/lib/aws-java-sdk-ec2-1.10.77.jar:/usr/hdp/current/druid-middlemanager/lib/druid-services-0.10.1.2.6.4.0-91.jar:/usr/hdp/current/druid-middlemanager/lib/javax.inject-1.jar:/usr/hdp/current/druid-middlemanager/lib/jackson-datatype-joda-2.4.6.jar:/usr/hdp/current/druid-middlemanager/lib/commons-dbcp-1.4.jar:/usr/hdp/current/druid-middlemanager/lib/jackson-datatype-guava-2.4.6.jar:/usr/hdp/current/druid-middlemanager/lib/jackson-core-2.4.6.jar:/usr/hdp/current/druid-middlemanager/lib/jline-0.9.94.jar:/usr/hdp/current/druid-middlemanager/lib/base64-2.3.8.jar:/usr/hdp/current/druid-middlemanager/lib/calcite-core-1.12.0.jar:/usr/hdp/current/druid-middlemanager/lib/mapdb-1.0.8.jar:/usr/hdp/current/druid-middlemanager/lib/curator-framework-2.11.0.jar:/usr/hdp/current/druid-middlemanager/lib/log4j-core-2.5.jar:/usr/hdp/current/druid-middlemanager/lib/jetty-servlets-9.3.19.v20170502.jar:/usr/hdp/current/druid-middlemanager/lib/druid-console-0.0.2.jar:/usr/hdp/current/druid-middlemanager/lib/maven-model-builder-3.1.1.jar:/usr/hdp/current/druid-middlemanager/lib/config-magic-0.9.jar:/usr/hdp/current/druid-middlemanager/lib/emitter-0.4.5.jar:/usr/hdp/current/druid-middlemanager/lib/slf4j-api-1.6.4.jar:/usr/hdp/current/druid-middlemanager/lib/jboss-logging-3.1.3.GA.jar:/usr/hdp/current/druid-middlemanager/lib/commons-cli-1.2.jar:/usr/hdp/current/druid-middlemanager/lib/guice-4.1.0.jar:/usr/hdp/current/druid-middlemanager/lib/druid-common-0.10.1.2.6.4.0-91.jar:/usr/hdp/current/druid-middlemanager/lib/jackson-dataformat-cbor-2.5.3.jar:/usr/hdp/current/druid-middlemanager/lib/derby-10.11.1.1.jar:/usr/hdp/current/druid-middlemanager/lib/aether-api-0.9.0.M2.jar:/usr/hdp/current/druid-middlemanager/lib/commons-io-2.5.jar:/usr/hdp/current/druid-middlemanager/lib/netty-3.10.6.Final.jar:/usr/hdp/current/druid-middlemanager/lib/javax.el-3.0.0.jar:/usr/hdp/current/druid-middlemanager/lib/janino-2.7.6.jar:/usr/hdp/current/druid-middlemanager/lib/plexus-utils-3.0.15.jar:/usr/hdp/current/druid-middlemanager/lib/jackson-jaxrs-base-2.4.6.jar:/usr/hdp/current/druid-middlemanager/lib/aws-java-sdk-core-1.10.77.jar:/usr/hdp/current/druid-middlemanager/lib/javax.servlet-api-3.1.0.jar:/usr/hdp/current/druid-middlemanager/lib/log4j-1.2-api-2.5.jar:/usr/hdp/current/druid-middlemanager/lib/avatica-core-1.9.0.jar:/usr/hdp/current/druid-middlemanager/lib/aether-impl-0.9.0.M2.jar:/usr/hdp/current/druid-middlemanager/lib/jetty-security-9.3.19.v20170502.jar:/usr/hdp/current/druid-middlemanager/lib/spymemcached-2.12.3.jar:/usr/hdp/current/druid-middlemanager/lib/maven-aether-provider-3.1.1.jar:/usr/hdp/current/druid-middlemanager/lib/aggdesigner-algorithm-6.0.jar:/usr/hdp/current/druid-middlemanager/lib/aether-connector-okhttp-0.0.9.jar:/usr/hdp/current/druid-middlemanager/lib/commons-pool2-2.2.jar:/usr/hdp/current/druid-middlemanager/lib/jackson-databind-2.4.6.jar:/usr/hdp/current/druid-middlemanager/lib/asm-tree-5.2.jar:/usr/hdp/current/druid-middlemanager/lib/jetty-continuation-9.3.19.v20170502.jar:/usr/hdp/current/druid-middlemanager/lib/java-util-0.28.2.jar:/usr/hdp/current/druid-middlemanager/lib/maven-repository-metadata-3.1.1.jar:/usr/hdp/current/druid-middlemanager/lib/jackson-annotations-2.4.6.jar:/usr/hdp/current/druid-middlemanager/lib/calcite-linq4j-1.12.0.jar:/usr/hdp/current/druid-middlemanager/lib/jersey-servlet-1.19.3.jar:/usr/hdp/current/druid-middlemanager/lib/jackson-mapper-asl-1.9.13.jar:/usr/hdp/current/druid-middlemanager/lib/asm-5.2.jar:/usr/hdp/current/druid-middlemanager/lib/asm-commons-5.2.jar:/usr/hdp/current/druid-middlemanager/lib/httpclient-4.5.1.jar:/usr/hdp/current/druid-middlemanager/lib/lz4-1.3.0.jar:/usr/hdp/current/druid-middlemanager/lib/avatica-metrics-1.9.0.jar:/usr/hdp/current/druid-middlemanager/lib/jetty-servlet-9.3.19.v20170502.jar:/usr/hdp/current/druid-middlemanager/lib/jackson-core-asl-1.9.13.jar:/usr/hdp/current/druid-middlemanager/lib/log4j-slf4j-impl-2.5.jar:/usr/hdp/current/druid-middlemanager/lib/javax.el-api-3.0.0.jar:/usr/hdp/current/druid-middlemanager/lib/irc-api-1.0-0014.jar:/usr/hdp/current/druid-middlemanager/lib/java-util-0.10.1.2.6.4.0-91.jar:/usr/hdp/current/druid-middlemanager/lib/jersey-guice-1.19.3.jar:/usr/hdp/current/druid-middlemanager/lib/activation-1.1.1.jar:/usr/hdp/current/druid-middlemanager/lib/druid-indexing-hadoop-0.10.1.2.6.4.0-91.jar:/usr/hdp/current/druid-middlemanager/lib/commons-lang-2.6.jar:/usr/hdp/current/druid-middlemanager/lib/jersey-server-1.19.3.jar:/usr/hdp/current/druid-middlemanager/lib/aether-connector-file-0.9.0.M2.jar:/usr/hdp/current/druid-middlemanager/lib/geoip2-0.4.0.jar:/usr/hdp/current/druid-middlemanager/lib/joda-time-2.9.9.jar:/usr/hdp/current/druid-middlemanager/lib/jetty-server-9.3.19.v20170502.jar:/usr/hdp/current/druid-middlemanager/lib/druid-hll-0.10.1.2.6.4.0-91.jar:/usr/hdp/current/druid-middlemanager/lib/guice-multibindings-4.1.0.jar:/usr/hdp/current/druid-middlemanager/lib/curator-x-discovery-2.11.0.jar:/usr/hdp/current/druid-middlemanager/lib/druid-api-0.10.1.2.6.4.0-91.jar:/usr/hdp/current/druid-middlemanager/lib/commons-lang3-3.2.jar:/usr/hdp/current/druid-middlemanager/lib/maxminddb-0.2.0.jar:/usr/hdp/current/druid-middlemanager/lib/jetty-io-9.3.19.v20170502.jar:/usr/hdp/current/druid-middlemanager/lib/maven-settings-builder-3.1.1.jar:/usr/hdp/current/druid-middlemanager/lib/jdbi-2.63.1.jar:/usr/hdp/current/druid-middlemanager/lib/aether-util-0.9.0.M2.jar:/usr/hdp/current/druid-middlemanager/lib/icu4j-54.1.1.jar:/usr/hdp/current/druid-middlemanager/lib/validation-api-1.1.0.Final.jar:/usr/hdp/current/druid-middlemanager/lib/jets3t-0.9.4.jar:/usr/hdp/current/druid-middlemanager/lib/jackson-module-jaxb-annotations-2.4.6.jar:/usr/hdp/current/druid-middlemanager/lib/server-metrics-0.2.8.jar:/usr/hdp/current/druid-middlemanager/lib/jetty-client-9.3.19.v20170502.jar:/usr/hdp/current/druid-middlemanager/lib/avatica-server-1.9.0.jar:/usr/hdp/current/druid-middlemanager/lib/httpcore-4.4.3.jar:/usr/hdp/current/druid-middlemanager/lib/airline-0.7.jar:/usr/hdp/current/druid-middlemanager/lib/jackson-jaxrs-json-provider-2.4.6.jar:/usr/hdp/current/druid-middlemanager/lib/druid-processing-0.10.1.2.6.4.0-91.jar:/usr/hdp/current/druid-middlemanager/lib/zookeeper-3.4.6.2.6.4.0-91.jar:/usr/hdp/current/druid-middlemanager/lib/guava-16.0.1.jar:/usr/hdp/current/druid-middlemanager/lib/commons-codec-1.7.jar:/usr/hdp/current/druid-middlemanager/lib/jersey-core-1.19.3.jar:/usr/hdp/current/druid-middlemanager/lib/commons-dbcp2-2.0.1.jar:/usr/hdp/current/druid-middlemanager/lib/hibernate-validator-5.1.3.Final.jar:/usr/hdp/current/druid-middlemanager/lib/java-xmlbuilder-1.1.jar:/usr/hdp/current/druid-middlemanager/lib/commons-math3-3.6.1.jar:/usr/hdp/current/druid-middlemanager/lib/rhino-1.7R5.jar:/usr/hdp/current/druid-middlemanager/lib/log4j-jul-2.5.jar:/usr/hdp/current/druid-middlemanager/lib/fastutil-7.2.0.jar:/usr/hdp/2.6.4.0-91/hadoop/conf
2018-03-29T14:09:31,413 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
2018-03-29T14:09:31,413 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:java.io.tmpdir=/tmp
2018-03-29T14:09:31,413 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:java.compiler=<NA>
2018-03-29T14:09:31,413 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:os.name=Linux
2018-03-29T14:09:31,413 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:os.arch=amd64
2018-03-29T14:09:31,413 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:os.version=4.4.114-1.el7.elrepo.x86_64
2018-03-29T14:09:31,413 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:user.name=druid
2018-03-29T14:09:31,413 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:user.home=/home/druid
2018-03-29T14:09:31,413 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:user.dir=/home/druid
2018-03-29T14:09:31,414 INFO [main] org.apache.zookeeper.ZooKeeper - Initiating client connection, connectString=sandbox-hdp.hortonworks.com:2181 sessionTimeout=30000 watcher=org.apache.curator.ConnectionState@29013ef2
2018-03-29T14:09:31,434 INFO [main] io.druid.java.util.common.lifecycle.Lifecycle$AnnotationBasedHandler - Invoking start method[public void io.druid.curator.discovery.ServerDiscoverySelector.start() throws java.lang.Exception] on object[io.druid.curator.discovery.ServerDiscoverySelector@13cc0b90].
2018-03-29T14:09:31,474 INFO [main-SendThread(sandbox-hdp.hortonworks.com:2181)] org.apache.zookeeper.ClientCnxn - Opening socket connection to server sandbox-hdp.hortonworks.com/172.17.0.2:2181. Will not attempt to authenticate using SASL (unknown error)
2018-03-29T14:09:31,481 INFO [main-SendThread(sandbox-hdp.hortonworks.com:2181)] org.apache.zookeeper.ClientCnxn - Socket connection established, initiating session, client: /172.17.0.2:42090, server: sandbox-hdp.hortonworks.com/172.17.0.2:2181
2018-03-29T14:09:31,491 INFO [main-SendThread(sandbox-hdp.hortonworks.com:2181)] org.apache.zookeeper.ClientCnxn - Session establishment complete on server sandbox-hdp.hortonworks.com/172.17.0.2:2181, sessionid = 0x16271e552860012, negotiated timeout = 30000
2018-03-29T14:09:31,494 INFO [main-EventThread] org.apache.curator.framework.state.ConnectionStateManager - State change: CONNECTED
2018-03-29T14:09:31,590 INFO [main] io.druid.java.util.common.lifecycle.Lifecycle$AnnotationBasedHandler - Invoking start method[public void io.druid.curator.announcement.Announcer.start()] on object[io.druid.curator.announcement.Announcer@1f129467].
2018-03-29T14:09:31,591 INFO [main] io.druid.java.util.common.lifecycle.Lifecycle$AnnotationBasedHandler - Invoking start method[public void io.druid.curator.discovery.ServerDiscoverySelector.start() throws java.lang.Exception] on object[io.druid.curator.discovery.ServerDiscoverySelector@646cd766].
2018-03-29T14:09:31,612 INFO [main] io.druid.java.util.common.lifecycle.Lifecycle$AnnotationBasedHandler - Invoking start method[public void io.druid.indexing.worker.executor.ExecutorLifecycle.start() throws java.lang.InterruptedException] on object[io.druid.indexing.worker.executor.ExecutorLifecycle@65eb3597].
2018-03-29T14:09:31,662 INFO [main] io.druid.indexing.worker.executor.ExecutorLifecycle - Running with task: {
  "type" : "index",
  "id" : "index_usgs_2018-03-29T14:09:27.159Z",
  "resource" : {
    "availabilityGroup" : "index_usgs_2018-03-29T14:09:27.159Z",
    "requiredCapacity" : 1
  },
  "spec" : {
    "dataSchema" : {
      "dataSource" : "usgs",
      "parser" : {
        "type" : "string",
        "parseSpec" : {
          "format" : "tsv",
          "timestampSpec" : {
            "column" : "dates",
            "format" : "auto"
          },
          "dimensionsSpec" : {
            "dimensions" : [ "staid", "val", "dates" ],
            "dimensionExclusions" : [ ],
            "spatialDimensions" : [ ]
          }
        }
      },
      "metricsSpec" : [ {
        "type" : "count",
        "name" : "count"
      }, {
        "type" : "doubleSum",
        "name" : "avgFlowCuFtsec",
        "fieldName" : "val",
        "expression" : null
      } ],
      "granularitySpec" : {
        "type" : "uniform",
        "segmentGranularity" : "MONTH",
        "queryGranularity" : {
          "type" : "none"
        },
        "rollup" : true,
        "intervals" : [ "1963-01-01T00:00:00.000Z/2013-12-31T00:00:00.000Z" ]
      }
    },
    "ioConfig" : {
      "type" : "index",
      "firehose" : {
        "type" : "local",
        "baseDir" : "/tmp/usgs",
        "filter" : "napa-flow.tsv",
        "parser" : null
      },
      "appendToExisting" : false
    },
    "tuningConfig" : {
      "type" : "index",
      "targetPartitionSize" : 10000,
      "maxRowsInMemory" : 75000,
      "numShards" : null,
      "indexSpec" : {
        "bitmap" : {
          "type" : "concise"
        },
        "dimensionCompression" : "lz4",
        "metricCompression" : "lz4",
        "longEncoding" : "longs"
      },
      "maxPendingPersists" : 0,
      "buildV9Directly" : true,
      "forceExtendableShardSpecs" : false,
      "reportParseExceptions" : false,
      "publishTimeout" : 0
    }
  },
  "context" : null,
  "groupId" : "index_usgs_2018-03-29T14:09:27.159Z",
  "dataSource" : "usgs"
}
2018-03-29T14:09:31,664 INFO [main] io.druid.indexing.worker.executor.ExecutorLifecycle - Attempting to lock file[/apps/druid/tasks/index_usgs_2018-03-29T14:09:27.159Z/lock].
2018-03-29T14:09:31,666 INFO [main] io.druid.indexing.worker.executor.ExecutorLifecycle - Acquired lock file[/apps/druid/tasks/index_usgs_2018-03-29T14:09:27.159Z/lock] in 1ms.
2018-03-29T14:09:31,671 INFO [main] io.druid.indexing.common.actions.RemoteTaskActionClient - Performing action for task[index_usgs_2018-03-29T14:09:27.159Z]: LockTryAcquireAction{interval=1963-01-01T00:00:00.000Z/2014-01-01T00:00:00.000Z}
2018-03-29T14:09:31,679 INFO [main] io.druid.indexing.common.actions.RemoteTaskActionClient - Submitting action for task[index_usgs_2018-03-29T14:09:27.159Z] to overlord[http://sandbox-hdp.hortonworks.com:8090/druid/indexer/v1/action]: LockTryAcquireAction{interval=1963-01-01T00:00:00.000Z/2014-01-01T00:00:00.000Z}
2018-03-29T14:09:31,688 INFO [main] com.metamx.http.client.pool.ChannelResourceFactory - Generating: http://sandbox-hdp.hortonworks.com:8090
2018-03-29T14:09:31,728 INFO [main] com.metamx.http.client.pool.ChannelResourceFactory - Generating: http://sandbox-hdp.hortonworks.com:8090
2018-03-29T14:09:31,729 INFO [main] com.metamx.http.client.pool.ChannelResourceFactory - Generating: http://sandbox-hdp.hortonworks.com:8090
2018-03-29T14:09:31,730 INFO [main] com.metamx.http.client.pool.ChannelResourceFactory - Generating: http://sandbox-hdp.hortonworks.com:8090
2018-03-29T14:09:31,730 INFO [main] com.metamx.http.client.pool.ChannelResourceFactory - Generating: http://sandbox-hdp.hortonworks.com:8090
2018-03-29T14:09:31,731 INFO [main] com.metamx.http.client.pool.ChannelResourceFactory - Generating: http://sandbox-hdp.hortonworks.com:8090
2018-03-29T14:09:31,731 INFO [main] com.metamx.http.client.pool.ChannelResourceFactory - Generating: http://sandbox-hdp.hortonworks.com:8090
2018-03-29T14:09:31,732 INFO [main] com.metamx.http.client.pool.ChannelResourceFactory - Generating: http://sandbox-hdp.hortonworks.com:8090
2018-03-29T14:09:31,732 INFO [main] com.metamx.http.client.pool.ChannelResourceFactory - Generating: http://sandbox-hdp.hortonworks.com:8090
2018-03-29T14:09:31,732 INFO [main] com.metamx.http.client.pool.ChannelResourceFactory - Generating: http://sandbox-hdp.hortonworks.com:8090
2018-03-29T14:09:31,732 INFO [main] com.metamx.http.client.pool.ChannelResourceFactory - Generating: http://sandbox-hdp.hortonworks.com:8090
2018-03-29T14:09:31,733 INFO [main] com.metamx.http.client.pool.ChannelResourceFactory - Generating: http://sandbox-hdp.hortonworks.com:8090
2018-03-29T14:09:31,733 INFO [main] com.metamx.http.client.pool.ChannelResourceFactory - Generating: http://sandbox-hdp.hortonworks.com:8090
2018-03-29T14:09:31,733 INFO [main] com.metamx.http.client.pool.ChannelResourceFactory - Generating: http://sandbox-hdp.hortonworks.com:8090
2018-03-29T14:09:31,734 INFO [main] com.metamx.http.client.pool.ChannelResourceFactory - Generating: http://sandbox-hdp.hortonworks.com:8090
2018-03-29T14:09:31,734 INFO [main] com.metamx.http.client.pool.ChannelResourceFactory - Generating: http://sandbox-hdp.hortonworks.com:8090
2018-03-29T14:09:31,734 INFO [main] com.metamx.http.client.pool.ChannelResourceFactory - Generating: http://sandbox-hdp.hortonworks.com:8090
2018-03-29T14:09:31,735 INFO [main] com.metamx.http.client.pool.ChannelResourceFactory - Generating: http://sandbox-hdp.hortonworks.com:8090
2018-03-29T14:09:31,735 INFO [main] com.metamx.http.client.pool.ChannelResourceFactory - Generating: http://sandbox-hdp.hortonworks.com:8090
2018-03-29T14:09:31,735 INFO [main] com.metamx.http.client.pool.ChannelResourceFactory - Generating: http://sandbox-hdp.hortonworks.com:8090
2018-03-29T14:09:31,821 INFO [task-runner-0-priority-0] io.druid.indexing.overlord.ThreadPoolTaskRunner - Running task: index_usgs_2018-03-29T14:09:27.159Z
2018-03-29T14:09:31,824 INFO [task-runner-0-priority-0] io.druid.indexing.overlord.TaskRunnerUtils - Task [index_usgs_2018-03-29T14:09:27.159Z] location changed to [TaskLocation{host='sandbox-hdp.hortonworks.com', port=8100}].
2018-03-29T14:09:31,824 INFO [task-runner-0-priority-0] io.druid.indexing.overlord.TaskRunnerUtils - Task [index_usgs_2018-03-29T14:09:27.159Z] status changed to [RUNNING].
2018-03-29T14:09:31,829 INFO [task-runner-0-priority-0] io.druid.indexing.common.task.IndexTask - Determining intervals and shardSpecs
2018-03-29T14:09:31,831 INFO [main] org.eclipse.jetty.server.Server - jetty-9.3.19.v20170502
2018-03-29T14:09:31,880 ERROR [task-runner-0-priority-0] io.druid.indexing.overlord.ThreadPoolTaskRunner - Exception while running task[IndexTask{id=index_usgs_2018-03-29T14:09:27.159Z, type=index, dataSource=usgs}]
java.lang.IllegalArgumentException: Instantiation of [simple type, class io.druid.data.input.impl.DelimitedParseSpec] value failed: If columns field is not set, the first row of your data must have your header and hasHeaderRow must be set to true.
	at com.fasterxml.jackson.databind.ObjectMapper._convert(ObjectMapper.java:2774) ~[jackson-databind-2.4.6.jar:2.4.6]
	at com.fasterxml.jackson.databind.ObjectMapper.convertValue(ObjectMapper.java:2700) ~[jackson-databind-2.4.6.jar:2.4.6]
	at io.druid.segment.indexing.DataSchema.getParser(DataSchema.java:111) ~[druid-server-0.10.1.2.6.4.0-91.jar:0.10.1.2.6.4.0-91]
	at io.druid.indexing.common.task.IndexTask.determineShardSpecs(IndexTask.java:269) ~[druid-indexing-service-0.10.1.2.6.4.0-91.jar:0.10.1.2.6.4.0-91]
	at io.druid.indexing.common.task.IndexTask.run(IndexTask.java:188) ~[druid-indexing-service-0.10.1.2.6.4.0-91.jar:0.10.1.2.6.4.0-91]
	at io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:436) [druid-indexing-service-0.10.1.2.6.4.0-91.jar:0.10.1.2.6.4.0-91]
	at io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:408) [druid-indexing-service-0.10.1.2.6.4.0-91.jar:0.10.1.2.6.4.0-91]
	at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_161]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_161]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_161]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_161]
Caused by: com.fasterxml.jackson.databind.JsonMappingException: Instantiation of [simple type, class io.druid.data.input.impl.DelimitedParseSpec] value failed: If columns field is not set, the first row of your data must have your header and hasHeaderRow must be set to true.
	at com.fasterxml.jackson.databind.deser.std.StdValueInstantiator.wrapException(StdValueInstantiator.java:405) ~[jackson-databind-2.4.6.jar:2.4.6]
	at com.fasterxml.jackson.databind.deser.std.StdValueInstantiator.createFromObjectWith(StdValueInstantiator.java:234) ~[jackson-databind-2.4.6.jar:2.4.6]
	at com.fasterxml.jackson.databind.deser.impl.PropertyBasedCreator.build(PropertyBasedCreator.java:167) ~[jackson-databind-2.4.6.jar:2.4.6]
	at com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeUsingPropertyBased(BeanDeserializer.java:398) ~[jackson-databind-2.4.6.jar:2.4.6]
	at com.fasterxml.jackson.databind.deser.BeanDeserializerBase.deserializeFromObjectUsingNonDefault(BeanDeserializerBase.java:1064) ~[jackson-databind-2.4.6.jar:2.4.6]
	at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserializeFromObject(BeanDeserializer.java:264) ~[jackson-databind-2.4.6.jar:2.4.6]
	at com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeOther(BeanDeserializer.java:156) ~[jackson-databind-2.4.6.jar:2.4.6]
	at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:126) ~[jackson-databind-2.4.6.jar:2.4.6]
	at com.fasterxml.jackson.databind.jsontype.impl.AsPropertyTypeDeserializer._deserializeTypedForId(AsPropertyTypeDeserializer.java:113) ~[jackson-databind-2.4.6.jar:2.4.6]
	at com.fasterxml.jackson.databind.jsontype.impl.AsPropertyTypeDeserializer.deserializeTypedFromObject(AsPropertyTypeDeserializer.java:84) ~[jackson-databind-2.4.6.jar:2.4.6]
	at com.fasterxml.jackson.databind.deser.AbstractDeserializer.deserializeWithType(AbstractDeserializer.java:132) ~[jackson-databind-2.4.6.jar:2.4.6]
	at com.fasterxml.jackson.databind.deser.SettableBeanProperty.deserialize(SettableBeanProperty.java:536) ~[jackson-databind-2.4.6.jar:2.4.6]
	at com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeUsingPropertyBased(BeanDeserializer.java:344) ~[jackson-databind-2.4.6.jar:2.4.6]
	at com.fasterxml.jackson.databind.deser.BeanDeserializerBase.deserializeFromObjectUsingNonDefault(BeanDeserializerBase.java:1064) ~[jackson-databind-2.4.6.jar:2.4.6]
	at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserializeFromObject(BeanDeserializer.java:264) ~[jackson-databind-2.4.6.jar:2.4.6]
	at com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeOther(BeanDeserializer.java:156) ~[jackson-databind-2.4.6.jar:2.4.6]
	at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:126) ~[jackson-databind-2.4.6.jar:2.4.6]
	at com.fasterxml.jackson.databind.jsontype.impl.AsPropertyTypeDeserializer._deserializeTypedForId(AsPropertyTypeDeserializer.java:113) ~[jackson-databind-2.4.6.jar:2.4.6]
	at com.fasterxml.jackson.databind.jsontype.impl.AsPropertyTypeDeserializer.deserializeTypedFromObject(AsPropertyTypeDeserializer.java:84) ~[jackson-databind-2.4.6.jar:2.4.6]
	at com.fasterxml.jackson.databind.deser.AbstractDeserializer.deserializeWithType(AbstractDeserializer.java:132) ~[jackson-databind-2.4.6.jar:2.4.6]
	at com.fasterxml.jackson.databind.deser.impl.TypeWrappedDeserializer.deserialize(TypeWrappedDeserializer.java:41) ~[jackson-databind-2.4.6.jar:2.4.6]
	at com.fasterxml.jackson.databind.ObjectMapper._convert(ObjectMapper.java:2769) ~[jackson-databind-2.4.6.jar:2.4.6]
	... 10 more
Caused by: java.lang.IllegalArgumentException: If columns field is not set, the first row of your data must have your header and hasHeaderRow must be set to true.
	at com.google.common.base.Preconditions.checkArgument(Preconditions.java:125) ~[guava-16.0.1.jar:?]
	at io.druid.data.input.impl.DelimitedParseSpec.<init>(DelimitedParseSpec.java:65) ~[druid-api-0.10.1.2.6.4.0-91.jar:0.10.1.2.6.4.0-91]
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:1.8.0_161]
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:1.8.0_161]
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:1.8.0_161]
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_161]
	at com.fasterxml.jackson.databind.introspect.AnnotatedConstructor.call(AnnotatedConstructor.java:125) ~[jackson-databind-2.4.6.jar:2.4.6]
	at com.fasterxml.jackson.databind.deser.std.StdValueInstantiator.createFromObjectWith(StdValueInstantiator.java:230) ~[jackson-databind-2.4.6.jar:2.4.6]
	at com.fasterxml.jackson.databind.deser.impl.PropertyBasedCreator.build(PropertyBasedCreator.java:167) ~[jackson-databind-2.4.6.jar:2.4.6]
	at com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeUsingPropertyBased(BeanDeserializer.java:398) ~[jackson-databind-2.4.6.jar:2.4.6]
	at com.fasterxml.jackson.databind.deser.BeanDeserializerBase.deserializeFromObjectUsingNonDefault(BeanDeserializerBase.java:1064) ~[jackson-databind-2.4.6.jar:2.4.6]
	at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserializeFromObject(BeanDeserializer.java:264) ~[jackson-databind-2.4.6.jar:2.4.6]
	at com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeOther(BeanDeserializer.java:156) ~[jackson-databind-2.4.6.jar:2.4.6]
	at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:126) ~[jackson-databind-2.4.6.jar:2.4.6]
	at com.fasterxml.jackson.databind.jsontype.impl.AsPropertyTypeDeserializer._deserializeTypedForId(AsPropertyTypeDeserializer.java:113) ~[jackson-databind-2.4.6.jar:2.4.6]
	at com.fasterxml.jackson.databind.jsontype.impl.AsPropertyTypeDeserializer.deserializeTypedFromObject(AsPropertyTypeDeserializer.java:84) ~[jackson-databind-2.4.6.jar:2.4.6]
	at com.fasterxml.jackson.databind.deser.AbstractDeserializer.deserializeWithType(AbstractDeserializer.java:132) ~[jackson-databind-2.4.6.jar:2.4.6]
	at com.fasterxml.jackson.databind.deser.SettableBeanProperty.deserialize(SettableBeanProperty.java:536) ~[jackson-databind-2.4.6.jar:2.4.6]
	at com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeUsingPropertyBased(BeanDeserializer.java:344) ~[jackson-databind-2.4.6.jar:2.4.6]
	at com.fasterxml.jackson.databind.deser.BeanDeserializerBase.deserializeFromObjectUsingNonDefault(BeanDeserializerBase.java:1064) ~[jackson-databind-2.4.6.jar:2.4.6]
	at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserializeFromObject(BeanDeserializer.java:264) ~[jackson-databind-2.4.6.jar:2.4.6]
	at com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeOther(BeanDeserializer.java:156) ~[jackson-databind-2.4.6.jar:2.4.6]
	at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:126) ~[jackson-databind-2.4.6.jar:2.4.6]
	at com.fasterxml.jackson.databind.jsontype.impl.AsPropertyTypeDeserializer._deserializeTypedForId(AsPropertyTypeDeserializer.java:113) ~[jackson-databind-2.4.6.jar:2.4.6]
	at com.fasterxml.jackson.databind.jsontype.impl.AsPropertyTypeDeserializer.deserializeTypedFromObject(AsPropertyTypeDeserializer.java:84) ~[jackson-databind-2.4.6.jar:2.4.6]
	at com.fasterxml.jackson.databind.deser.AbstractDeserializer.deserializeWithType(AbstractDeserializer.java:132) ~[jackson-databind-2.4.6.jar:2.4.6]
	at com.fasterxml.jackson.databind.deser.impl.TypeWrappedDeserializer.deserialize(TypeWrappedDeserializer.java:41) ~[jackson-databind-2.4.6.jar:2.4.6]
	at com.fasterxml.jackson.databind.ObjectMapper._convert(ObjectMapper.java:2769) ~[jackson-databind-2.4.6.jar:2.4.6]
	... 10 more
2018-03-29T14:09:31,887 INFO [task-runner-0-priority-0] io.druid.indexing.overlord.TaskRunnerUtils - Task [index_usgs_2018-03-29T14:09:27.159Z] status changed to [FAILED].
2018-03-29T14:09:31,898 INFO [task-runner-0-priority-0] io.druid.indexing.worker.executor.ExecutorLifecycle - Task completed with status: {
  "id" : "index_usgs_2018-03-29T14:09:27.159Z",
  "status" : "FAILED",
  "duration" : 66
}
2018-03-29T14:09:32,015 INFO [main] com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory - Registering com.fasterxml.jackson.jaxrs.json.JacksonJsonProvider as a provider class
2018-03-29T14:09:32,015 INFO [main] com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory - Registering com.fasterxml.jackson.jaxrs.smile.JacksonSmileProvider as a provider class
2018-03-29T14:09:32,016 INFO [main] com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory - Registering io.druid.server.initialization.jetty.CustomExceptionMapper as a provider class
2018-03-29T14:09:32,016 INFO [main] com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory - Registering io.druid.server.StatusResource as a root resource class
2018-03-29T14:09:32,019 INFO [main] com.sun.jersey.server.impl.application.WebApplicationImpl - Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:43 PM'
2018-03-29T14:09:32,086 INFO [main] com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory - Binding io.druid.server.initialization.jetty.CustomExceptionMapper to GuiceManagedComponentProvider with the scope "Singleton"
2018-03-29T14:09:32,089 INFO [main] com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory - Binding com.fasterxml.jackson.jaxrs.json.JacksonJsonProvider to GuiceManagedComponentProvider with the scope "Singleton"
2018-03-29T14:09:32,096 INFO [main] com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory - Binding com.fasterxml.jackson.jaxrs.smile.JacksonSmileProvider to GuiceManagedComponentProvider with the scope "Singleton"
2018-03-29T14:09:32,368 INFO [main] com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory - Binding io.druid.server.http.security.StateResourceFilter to GuiceInstantiatedComponentProvider
2018-03-29T14:09:32,387 INFO [main] com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory - Binding io.druid.server.http.SegmentListerResource to GuiceInstantiatedComponentProvider
2018-03-29T14:09:32,398 INFO [main] com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory - Binding io.druid.server.QueryResource to GuiceInstantiatedComponentProvider
2018-03-29T14:09:32,400 INFO [main] com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory - Binding io.druid.segment.realtime.firehose.ChatHandlerResource to GuiceInstantiatedComponentProvider
2018-03-29T14:09:32,403 INFO [main] com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory - Binding io.druid.query.lookup.LookupListeningResource to GuiceInstantiatedComponentProvider
2018-03-29T14:09:32,405 INFO [main] com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory - Binding io.druid.query.lookup.LookupIntrospectionResource to GuiceInstantiatedComponentProvider
2018-03-29T14:09:32,406 INFO [main] com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory - Binding io.druid.server.StatusResource to GuiceManagedComponentProvider with the scope "Undefined"
2018-03-29T14:09:32,420 WARN [main] com.sun.jersey.spi.inject.Errors - The following warnings have been detected with resource and/or provider classes:
  WARNING: A HTTP GET method, public void io.druid.server.http.SegmentListerResource.getSegments(long,long,long,javax.servlet.http.HttpServletRequest) throws java.io.IOException, MUST return a non-void type.
2018-03-29T14:09:32,428 INFO [main] org.eclipse.jetty.server.handler.ContextHandler - Started o.e.j.s.ServletContextHandler@63de4fa{/,null,AVAILABLE}
2018-03-29T14:09:32,435 INFO [main] org.eclipse.jetty.server.AbstractConnector - Started ServerConnector@2fba0dac{HTTP/1.1,[http/1.1]}{0.0.0.0:8100}
2018-03-29T14:09:32,435 INFO [main] org.eclipse.jetty.server.Server - Started @5163ms
2018-03-29T14:09:32,436 INFO [main] io.druid.java.util.common.lifecycle.Lifecycle$AnnotationBasedHandler - Invoking start method[public void io.druid.server.listener.announcer.ListenerResourceAnnouncer.start()] on object[io.druid.query.lookup.LookupResourceListenerAnnouncer@7be38eba].
2018-03-29T14:09:32,450 INFO [main] io.druid.server.listener.announcer.ListenerResourceAnnouncer - Announcing start time on [/druid/listeners/lookups/__default/sandbox-hdp.hortonworks.com:8100]
2018-03-29T14:09:32,455 INFO [main] io.druid.java.util.common.lifecycle.Lifecycle$AnnotationBasedHandler - Invoking stop method[public void io.druid.server.listener.announcer.ListenerResourceAnnouncer.stop()] on object[io.druid.query.lookup.LookupResourceListenerAnnouncer@7be38eba].
2018-03-29T14:09:32,455 INFO [main] io.druid.curator.announcement.Announcer - unannouncing [/druid/listeners/lookups/__default/sandbox-hdp.hortonworks.com:8100]
2018-03-29T14:09:32,484 INFO [main] io.druid.server.listener.announcer.ListenerResourceAnnouncer - Unannouncing start time on [/druid/listeners/lookups/__default/sandbox-hdp.hortonworks.com:8100]
2018-03-29T14:09:32,488 INFO [main] org.eclipse.jetty.server.AbstractConnector - Stopped ServerConnector@2fba0dac{HTTP/1.1,[http/1.1]}{0.0.0.0:8100}
2018-03-29T14:09:32,489 INFO [main] org.eclipse.jetty.server.handler.ContextHandler - Stopped o.e.j.s.ServletContextHandler@63de4fa{/,null,UNAVAILABLE}
2018-03-29T14:09:32,496 INFO [main] io.druid.java.util.common.lifecycle.Lifecycle$AnnotationBasedHandler - Invoking stop method[public void io.druid.indexing.worker.executor.ExecutorLifecycle.stop() throws java.lang.Exception] on object[io.druid.indexing.worker.executor.ExecutorLifecycle@65eb3597].
2018-03-29T14:09:32,497 INFO [main] io.druid.java.util.common.lifecycle.Lifecycle$AnnotationBasedHandler - Invoking stop method[public void io.druid.indexing.overlord.ThreadPoolTaskRunner.stop()] on object[io.druid.indexing.overlord.ThreadPoolTaskRunner@37b1149b].
2018-03-29T14:09:32,497 INFO [main] io.druid.java.util.common.lifecycle.Lifecycle$AnnotationBasedHandler - Invoking stop method[public void io.druid.curator.discovery.ServerDiscoverySelector.stop() throws java.io.IOException] on object[io.druid.curator.discovery.ServerDiscoverySelector@646cd766].
2018-03-29T14:09:32,501 INFO [main] io.druid.java.util.common.lifecycle.Lifecycle$AnnotationBasedHandler - Invoking stop method[public void io.druid.curator.announcement.Announcer.stop()] on object[io.druid.curator.announcement.Announcer@1f129467].
2018-03-29T14:09:32,501 INFO [main] io.druid.java.util.common.lifecycle.Lifecycle$AnnotationBasedHandler - Invoking stop method[public void io.druid.curator.discovery.ServerDiscoverySelector.stop() throws java.io.IOException] on object[io.druid.curator.discovery.ServerDiscoverySelector@13cc0b90].
2018-03-29T14:09:32,501 INFO [main] io.druid.curator.CuratorModule - Stopping Curator
2018-03-29T14:09:32,502 INFO [Curator-Framework-0] org.apache.curator.framework.imps.CuratorFrameworkImpl - backgroundOperationsLoop exiting
2018-03-29T14:09:32,506 INFO [main] org.apache.zookeeper.ZooKeeper - Session: 0x16271e552860012 closed
2018-03-29T14:09:32,506 INFO [main-EventThread] org.apache.zookeeper.ClientCnxn - EventThread shut down
2018-03-29T14:09:32,506 INFO [main] com.metamx.common.lifecycle.Lifecycle$AnnotationBasedHandler - Invoking stop method[public void com.metamx.http.client.NettyHttpClient.stop()] on object[com.metamx.http.client.NettyHttpClient@36ad229b].
2018-03-29T14:09:32,527 INFO [main] io.druid.java.util.common.lifecycle.Lifecycle$AnnotationBasedHandler - Invoking stop method[public void io.druid.storage.hdfs.HdfsStorageAuthentication.stop()] on object[io.druid.storage.hdfs.HdfsStorageAuthentication@2e56b4d].
2018-03-29T14:09:32,527 INFO [main] io.druid.java.util.common.lifecycle.Lifecycle$AnnotationBasedHandler - Invoking stop method[public void com.metamx.metrics.MonitorScheduler.stop()] on object[com.metamx.metrics.MonitorScheduler@be164d8].
2018-03-29T14:09:32,528 INFO [main] io.druid.java.util.common.lifecycle.Lifecycle$AnnotationBasedHandler - Invoking stop method[public void io.druid.query.lookup.LookupReferencesManager.stop()] on object[io.druid.query.lookup.LookupReferencesManager@89caf47].
2018-03-29T14:09:32,528 INFO [main] io.druid.query.lookup.LookupReferencesManager - LookupReferencesManager is stopping.
2018-03-29T14:09:32,528 INFO [LookupReferencesManager-MainThread] io.druid.query.lookup.LookupReferencesManager - Lookup Management loop exited, Lookup notices are not handled anymore.
2018-03-29T14:09:32,529 INFO [main] io.druid.query.lookup.LookupReferencesManager - LookupReferencesManager is stopped.
2018-03-29T14:09:32,529 INFO [main] io.druid.java.util.common.lifecycle.Lifecycle$AnnotationBasedHandler - Invoking stop method[public void com.metamx.emitter.service.ServiceEmitter.close() throws java.io.IOException] on object[com.metamx.emitter.service.ServiceEmitter@5a090f62].
2018-03-29T14:09:32,529 INFO [main] io.druid.java.util.common.lifecycle.Lifecycle$AnnotationBasedHandler - Invoking stop method[public void io.druid.initialization.Log4jShutterDownerModule$Log4jShutterDowner.stop()] on object[io.druid.initialization.Log4jShutterDownerModule$Log4jShutterDowner@3dbd7107].
2018-03-29 14:09:32,567 pool-1-thread-1 ERROR Unable to register shutdown hook because JVM is shutting down. java.lang.IllegalStateException: Not started
	at io.druid.common.config.Log4jShutdown.addShutdownCallback(Log4jShutdown.java:46)
	at org.apache.logging.log4j.core.impl.Log4jContextFactory.addShutdownCallback(Log4jContextFactory.java:273)
	at org.apache.logging.log4j.core.LoggerContext.setUpShutdownHook(LoggerContext.java:256)
	at org.apache.logging.log4j.core.LoggerContext.start(LoggerContext.java:216)
	at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:145)
	at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:41)
	at org.apache.logging.log4j.LogManager.getContext(LogManager.java:182)
	at org.apache.logging.log4j.spi.AbstractLoggerAdapter.getContext(AbstractLoggerAdapter.java:103)
	at org.apache.logging.slf4j.Log4jLoggerFactory.getContext(Log4jLoggerFactory.java:43)
	at org.apache.logging.log4j.spi.AbstractLoggerAdapter.getLogger(AbstractLoggerAdapter.java:42)
	at org.apache.logging.slf4j.Log4jLoggerFactory.getLogger(Log4jLoggerFactory.java:29)
	at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:253)
	at org.apache.commons.logging.impl.SLF4JLogFactory.getInstance(SLF4JLogFactory.java:155)
	at org.apache.commons.logging.impl.SLF4JLogFactory.getInstance(SLF4JLogFactory.java:132)
	at org.apache.commons.logging.LogFactory.getLog(LogFactory.java:273)
	at org.apache.hadoop.hdfs.LeaseRenewer.<clinit>(LeaseRenewer.java:72)
	at org.apache.hadoop.hdfs.DFSClient.getLeaseRenewer(DFSClient.java:830)
	at org.apache.hadoop.hdfs.DFSClient.close(DFSClient.java:968)
	at org.apache.hadoop.hdfs.DistributedFileSystem.close(DistributedFileSystem.java:1214)
	at org.apache.hadoop.fs.FileSystem$Cache.closeAll(FileSystem.java:2886)
	at org.apache.hadoop.fs.FileSystem$Cache$ClientFinalizer.run(FileSystem.java:2903)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)


Any help would be very much appreciated.

1 ACCEPTED SOLUTION

Explorer

@jmedel

The root cause of failure is invalid parse spec

java.lang.IllegalArgumentException: Instantiation of [simple type, class io.druid.data.input.impl.DelimitedParseSpec] value failed: If columns field is not set, the first row of your data must have your header and hasHeaderRow must be set to true.

Check if your input file has a header row or not.

If yes, set hasHeaderRow = true in parseSpec, otherwise you need to specify list of columns in your parseSpec so that druid knows about the columns present in the file.

View solution in original post

3 REPLIES 3

Explorer

@jmedel

The root cause of failure is invalid parse spec

java.lang.IllegalArgumentException: Instantiation of [simple type, class io.druid.data.input.impl.DelimitedParseSpec] value failed: If columns field is not set, the first row of your data must have your header and hasHeaderRow must be set to true.

Check if your input file has a header row or not.

If yes, set hasHeaderRow = true in parseSpec, otherwise you need to specify list of columns in your parseSpec so that druid knows about the columns present in the file.

Contributor

overlord-log.txtHi Nishant,

The input file doesn't have a header row.napa-flowtsv.txt

But, I utilized your suggestion about adding the "hasHeaderRow = true" and that caused the task to complete successfully.

Here is my latest Stack trace with the ERROR I noticed:

2018-03-29 15:54:37,879 pool-1-thread-1 ERROR Unable to register shutdown hook because JVM is shutting down. java.lang.IllegalStateException: Not started
	at io.druid.common.config.Log4jShutdown.addShutdownCallback(Log4jShutdown.java:46)
	at org.apache.logging.log4j.core.impl.Log4jContextFactory.addShutdownCallback(Log4jContextFactory.java:273)
	at org.apache.logging.log4j.core.LoggerContext.setUpShutdownHook(LoggerContext.java:256)
	at org.apache.logging.log4j.core.LoggerContext.start(LoggerContext.java:216)
	at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:145)
	at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:41)
	at org.apache.logging.log4j.LogManager.getContext(LogManager.java:182)
	at org.apache.logging.log4j.spi.AbstractLoggerAdapter.getContext(AbstractLoggerAdapter.java:103)
	at org.apache.logging.slf4j.Log4jLoggerFactory.getContext(Log4jLoggerFactory.java:43)
	at org.apache.logging.log4j.spi.AbstractLoggerAdapter.getLogger(AbstractLoggerAdapter.java:42)
	at org.apache.logging.slf4j.Log4jLoggerFactory.getLogger(Log4jLoggerFactory.java:29)
	at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:253)
	at org.apache.commons.logging.impl.SLF4JLogFactory.getInstance(SLF4JLogFactory.java:155)
	at org.apache.commons.logging.impl.SLF4JLogFactory.getInstance(SLF4JLogFactory.java:132)
	at org.apache.commons.logging.LogFactory.getLog(LogFactory.java:273)
	at org.apache.hadoop.hdfs.LeaseRenewer.<clinit>(LeaseRenewer.java:72)
	at org.apache.hadoop.hdfs.DFSClient.getLeaseRenewer(DFSClient.java:830)
	at org.apache.hadoop.hdfs.DFSClient.close(DFSClient.java:968)
	at org.apache.hadoop.hdfs.DistributedFileSystem.close(DistributedFileSystem.java:1214)
	at org.apache.hadoop.fs.FileSystem$Cache.closeAll(FileSystem.java:2886)
	at org.apache.hadoop.fs.FileSystem$Cache$ClientFinalizer.run(FileSystem.java:2903)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

Contributor

Hi Nishant,

I have an update, so utiliizing your suggestion of adding the "columns": [ ... ]

solved my problem and now I am able to successfully ingest my data into Druid. For users who want to see the ingestion spec, I have included it below:

{
    "type" : "index_hadoop",
    "spec" : {
      "dataSchema" : {
        "dataSource" : "usgs",
        "parser" : {
          "type" : "hadoopyString",
          "parseSpec" : {
            "format" : "tsv",
            "timestampSpec" : {
              "column" : "dates",
              "format" : "auto"
            },
            "dimensionsSpec" : {
              "dimensions": ["staid","val","dates"],
              "dimensionExclusions" : [],
              "spatialDimensions" : []
            },
            "columns" : ["staid","val","dates"]
          }
        },
        "metricsSpec" : [
          {
            "type" : "count",
            "name" : "count"
          },
          {
            "type" : "doubleSum",
            "name" : "avgFlowCuFtsec",
            "fieldName" : "val"
          }
        ],
        "granularitySpec" : {
          "type" : "uniform",
          "segmentGranularity" : "MONTH",
          "queryGranularity" : "NONE",
          "intervals" : [ "1963-01-01/2013-12-31" ]
        }
      },
      "ioConfig" : {
        "type" : "hadoop",
        "inputSpec" : {
          "type" : "static",
          "paths" : "/tmp/druid/napa-flow.tsv.gz"
        }
      },
      "tuningConfig" : {
        "type": "hadoop",
        "targetPartitionSize" : 10000,
        "maxRowsInMemory" : 75000
      }
    }
}


Note: I will add the "index" ingestion spec soon too, so users can ingest data into Druid from their local file system or they can go with hadoop file system.