Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Hive druid handler "java.lang.NoClassDefFoundError: org/apache/hadoop/hive/metastore/DefaultHiveMetaHook"

avatar
Explorer

Hi!

I've installed HDP 2.6 and installed druid. Trying to create external table to a wikiticker datasource gives me an error:

Error: java.lang.NoClassDefFoundError: org/apache/hadoop/hive/metastore/DefaultHiveMetaHook (state=,code=0)

I've added druid handler (lib/hive-druid-handler.jar) with "add jar" before. Tried to add hive metastore jar - it did not help. How to use this handler? Thank you!

14 REPLIES 14

avatar
New Contributor
Error: java.lang.NoClassDefFoundError: org/apache/hadoop/hive/metastore/DefaultHiveMetaHook (state=,code=0)

is due to another missing JAR:

hive-metastore-2.3.0.jar

which is extended by the DruidStorageHandler class. After adding it, the exception is fixed, and now running the CREATE I get:

SemanticException [Error 10043]: Either list of columns or a custom serializer should be specified

The fact that we get the exceptions, and we fixed them, implies that the STORED BY is using DruidStorageHandler, but this message implies that for some reason it does not supply the column data from the Druid table. Anyone got past this point? I am thinking maybe its now a problem connecting to the Druid broker?

avatar
Expert Contributor

@Adrian Gay either your broker address setting is wrong, this can be fixed by running hive with this config

--hiveconf hive.druid.broker.address.default=hostname_of_broker:8082 or the data source you are referring to doesn't exist in druid yet ("druid.datasource"="wikiticker")

avatar
New Contributor

Appreciate u replying. I have been poking around in Hive Github and I see you are a significant contributor to Druid integration 🙂 I set

  <property>
      <name>hive.druid.broker.address.default</name>
      <value>10.93.206.102:8082</value>
  </property>

in hive-site.xml and I am sure the datasource name I am using is correct because I can access Druid from Imply Pivot and Apache Superset using these params. Looking in the code for DruidStorageHandler I wonder if it may be something to do with me not setting the db type property anywhere?

    final String dbType = HiveConf            .getVar(SessionState.getSessionConf(), HiveConf.ConfVars.DRUID_METADATA_DB_TYPE);

as this would cause an exception in the constructor of the class?

    if (dbType.equals("mysql")) {      connector = new MySQLConnector(storageConnectorConfigSupplier,              Suppliers.ofInstance(druidMetadataStorageTablesConfig)      );    } else if (dbType.equals("postgresql")) {      connector = new PostgreSQLConnector(storageConnectorConfigSupplier,              Suppliers.ofInstance(druidMetadataStorageTablesConfig)      );    } else {      throw new IllegalStateException(String.format("Unknown metadata storage type [%s]", dbType));    }

I would have expected to see the exception logged at the hiveserver2 console on startup or when the [first] query is run... maybe? but I don't see anything but the semantic exception above.

avatar
Expert Contributor

@Adrian Gay I don't think the metadata setup is the issue since it is set by default to mysql. What do you see in your hive server logs ? for instance if the hive server can connect to druid broker you should see something like this/

 hive.log: error in initSerDe: org.apache.hadoop.hive.serde2.SerDeException Connected to Druid but could not retrieve datasource information
org.apache.hadoop.hive.serde2.SerDeException: Connected to Druid but could not retrieve datasource information
        at org.apache.hadoop.hive.druid.serde.DruidSerDe.submitMetadataRequest(DruidSerDe.java:296) ~[hive-druid-handler-2.1.0.2.6.3.0-SNAPSHOT.jar:2.1.0.2.6.3.0-SNAPSHOT]
        at org.apache.hadoop.hive.druid.serde.DruidSerDe.initialize(DruidSerDe.java:178) ~[hive-druid-handler-2.1.0.2.6.3.0-SNAPSHOT.jar:2.1.0.2.6.3.0-SNAPSHOT]
        at org.apache.hadoop.hive.serde2.AbstractSerDe.initialize(AbstractSerDe.java:54) ~[hive-exec-2.1.0.2.6.3.0-SNAPSHOT.jar:2.1.0.2.6.3.0-SNAPSHOT]
        at org.apache.hadoop.hive.serde2.SerDeUtils.initializeSerDe(SerDeUtils.java:531) ~[hive-exec-2.1.0.2.6.3.0-SNAPSHOT.jar:2.1.0.2.6.3.0-SNAPSHOT]
        at org.apache.hadoop.hive.metastore.MetaStoreUtils.getDeserializer(MetaStoreUtils.java:436) ~[hive-exec-2.1.0.2.6.3.0-SNAPSHOT.jar:2.1.0.2.6.3.0-SNAPSHOT]
        at org.apache.hadoop.hive.metastore.MetaStoreUtils.getDeserializer(MetaStoreUtils.java:423) ~[hive-exec-2.1.0.2.6.3.0-SNAPSHOT.jar:2.1.0.2.6.3.0-SNAPSHOT]
        at org.apache.hadoop.hive.ql.metadata.Table.getDeserializerFromMetaStore(Table.java:279) ~[hive-exec-2.1.0.2.6.3.0-SNAPSHOT.jar:2.1.0.2.6.3.0-SNAPSHOT]
        at org.apache.hadoop.hive.ql.metadata.Table.getDeserializer(Table.java:261) ~[hive-exec-2.1.0.2.6.3.0-SNAPSHOT.jar:2.1.0.2.6.3.0-SNAPSHOT]
        at org.apache.hadoop.hive.ql.metadata.Table.getColsInternal(Table.java:639) ~[hive-exec-2.1.0.2.6.3.0-SNAPSHOT.jar:2.1.0.2.6.3.0-SNAPSHOT]
        at org.apache.hadoop.hive.ql.metadata.Table.getCols(Table.java:622) ~[hive-exec-2.1.0.2.6.3.0-SNAPSHOT.jar:2.1.0.2.6.3.0-SNAPSHOT]
        at org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:833) ~[hive-exec-2.1.0.2.6.3.0-SNAPSHOT.jar:2.1.0.2.6.3.0-SNAPSHOT]
        at org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:869) ~[hive-exec-2.1.0.2.6.3.0-SNAPSHOT.jar:2.1.0.2.6.3.0-SNAPSHOT]
        at org.apache.hadoop.hive.ql.exec.DDLTask.createTable(DDLTask.java:4227) ~[hive-exec-2.1.0.2.6.3.0-SNAPSHOT.jar:2.1.0.2.6.3.0-SNAPSHOT]
        at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:347) ~[hive-exec-2.1.0.2.6.3.0-SNAPSHOT.jar:2.1.0.2.6.3.0-SNAPSHOT]
        at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:197) ~[hive-exec-2.1.0.2.6.3.0-SNAPSHOT.jar:2.1.0.2.6.3.0-SNAPSHOT]
        at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:100) ~[hive-exec-2.1.0.2.6.3.0-SNAPSHOT.jar:2.1.0.2.6.3.0-SNAPSHOT]
        at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1905) ~[hive-exec-2.1.0.2.6.3.0-SNAPSHOT.jar:2.1.0.2.6.3.0-SNAPSHOT]
        at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1607) ~[hive-exec-2.1.0.2.6.3.0-SNAPSHOT.jar:2.1.0.2.6.3.0-SNAPSHOT]
        at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1354) ~[hive-exec-2.1.0.2.6.3.0-SNAPSHOT.jar:2.1.0.2.6.3.0-SNAPSHOT]
        at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1123) ~[hive-exec-2.1.0.2.6.3.0-SNAPSHOT.jar:2.1.0.2.6.3.0-SNAPSHOT]
        at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1111) ~[hive-exec-2.1.0.2.6.3.0-SNAPSHOT.jar:2.1.0.2.6.3.0-SNAPSHOT]
        at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:233) ~[hive-cli-2.1.0.2.6.3.0-SNAPSHOT.jar:2.1.0.2.6.3.0-SNAPSHOT]
        at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:184) ~[hive-cli-2.1.0.2.6.3.0-SNAPSHOT.jar:2.1.0.2.6.3.0-SNAPSHOT]
        at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:403) ~[hive-cli-2.1.0.2.6.3.0-SNAPSHOT.jar:2.1.0.2.6.3.0-SNAPSHOT]
        at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:821) ~[hive-cli-2.1.0.2.6.3.0-SNAPSHOT.jar:2.1.0.2.6.3.0-SNAPSHOT]
        at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:759) ~[hive-cli-2.1.0.2.6.3.0-SNAPSHOT.jar:2.1.0.2.6.3.0-SNAPSHOT]
        at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:686) ~[hive-cli-2.1.0.2.6.3.0-SNAPSHOT.jar:2.1.0.2.6.3.0-SNAPSHOT]
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_77]
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_77]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_77]
        at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_77]
        at org.apache.hadoop.util.RunJar.run(RunJar.java:233) ~[hadoop-common-2.7.3.2.5.0.1-216.jar:?]
        at org.apache.hadoop.util.RunJar.main(RunJar.java:148) ~[hadoop-common-2.7.3.2.5.0.1-216.jar:?]
2017-09-13T10:16:51,941 ERROR [6f8f271e-3af1-4385-a79a-4d37aecb9fde main] metadata.Table: Unable to get field from serde: org.apache.hadoop.hive.druid.serde.DruidSerDe
java.lang.RuntimeException: MetaException(message:org.apache.hadoop.hive.serde2.SerDeException Connected to Druid but could not retrieve datasource information)
        at org.apache.hadoop.hive.ql.metadata.Table.getDeserializerFromMetaStore(Table.java:281) ~[hive-exec-2.1.0.2.6.3.0-SNAPSHOT.jar:2.1.0.2.6.3.0-SNAPSHOT]
        at org.apache.hadoop.hive.ql.metadata.Table.getDeserializer(Table.java:261) ~[hive-exec-2.1.0.2.6.3.0-SNAPSHOT.jar:2.1.0.2.6.3.0-SNAPSHOT]
        at org.apache.hadoop.hive.ql.metadata.Table.getColsInternal(Table.java:639) ~[hive-exec-2.1.0.2.6.3.0-SNAPSHOT.jar:2.1.0.2.6.3.0-SNAPSHOT]
        at org.apache.hadoop.hive.ql.metadata.Table.getCols(Table.java:622) ~[hive-exec-2.1.0.2.6.3.0-SNAPSHOT.jar:2.1.0.2.6.3.0-SNAPSHOT]
        at org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:833) ~[hive-exec-2.1.0.2.6.3.0-SNAPSHOT.jar:2.1.0.2.6.3.0-SNAPSHOT]
        at org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:869) ~[hive-exec-2.1.0.2.6.3.0-SNAPSHOT.jar:2.1.0.2.6.3.0-SNAPSHOT]
        at org.apache.hadoop.hive.ql.exec.DDLTask.createTable(DDLTask.java:4227) ~[hive-exec-2.1.0.2.6.3.0-SNAPSHOT.jar:2.1.0.2.6.3.0-SNAPSHOT]
        at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:347) ~[hive-exec-2.1.0.2.6.3.0-SNAPSHOT.jar:2.1.0.2.6.3.0-SNAPSHOT]
        at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:197) ~[hive-exec-2.1.0.2.6.3.0-SNAPSHOT.jar:2.1.0.2.6.3.0-SNAPSHOT]
        at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:100) ~[hive-exec-2.1.0.2.6.3.0-SNAPSHOT.jar:2.1.0.2.6.3.0-SNAPSHOT]
        at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1905) ~[hive-exec-2.1.0.2.6.3.0-SNAPSHOT.jar:2.1.0.2.6.3.0-SNAPSHOT]
        at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1607) ~[hive-exec-2.1.0.2.6.3.0-SNAPSHOT.jar:2.1.0.2.6.3.0-SNAPSHOT]
        at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1354) ~[hive-exec-2.1.0.2.6.3.0-SNAPSHOT.jar:2.1.0.2.6.3.0-SNAPSHOT]
        at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1123) ~[hive-exec-2.1.0.2.6.3.0-SNAPSHOT.jar:2.1.0.2.6.3.0-SNAPSHOT]
        at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1111) ~[hive-exec-2.1.0.2.6.3.0-SNAPSHOT.jar:2.1.0.2.6.3.0-SNAPSHOT]
        at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:233) ~[hive-cli-2.1.0.2.6.3.0-SNAPSHOT.jar:2.1.0.2.6.3.0-SNAPSHOT]
        at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:184) ~[hive-cli-2.1.0.2.6.3.0-SNAPSHOT.jar:2.1.0.2.6.3.0-SNAPSHOT]
        at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:403) ~[hive-cli-2.1.0.2.6.3.0-SNAPSHOT.jar:2.1.0.2.6.3.0-SNAPSHOT]
        at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:821) ~[hive-cli-2.1.0.2.6.3.0-SNAPSHOT.jar:2.1.0.2.6.3.0-SNAPSHOT]
        at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:759) ~[hive-cli-2.1.0.2.6.3.0-SNAPSHOT.jar:2.1.0.2.6.3.0-SNAPSHOT]
        at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:686) ~[hive-cli-2.1.0.2.6.3.0-SNAPSHOT.jar:2.1.0.2.6.3.0-SNAPSHOT]
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_77]
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_77]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_77]
        at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_77]
        at org.apache.hadoop.util.RunJar.run(RunJar.java:233) ~[hadoop-common-2.7.3.2.5.0.1-216.jar:?]
        at org.apache.hadoop.util.RunJar.main(RunJar.java:148) ~[hadoop-common-2.7.3.2.5.0.1-216.jar:?]

avatar

HI @Slim

I'm seeing "Connected to Druid but could not retrieve datasource information" when I create a table.

Could you have any idea where I should check?