Created on 05-15-2017 01:50 PM - edited 09-16-2022 04:36 AM
Hi!
I've installed HDP 2.6 and installed druid. Trying to create external table to a wikiticker datasource gives me an error:
Error: java.lang.NoClassDefFoundError: org/apache/hadoop/hive/metastore/DefaultHiveMetaHook (state=,code=0)
I've added druid handler (lib/hive-druid-handler.jar) with "add jar" before. Tried to add hive metastore jar - it did not help. How to use this handler? Thank you!
Created 09-13-2017 01:47 PM
Error: java.lang.NoClassDefFoundError: org/apache/hadoop/hive/metastore/DefaultHiveMetaHook (state=,code=0)
is due to another missing JAR:
hive-metastore-2.3.0.jar
which is extended by the DruidStorageHandler class. After adding it, the exception is fixed, and now running the CREATE I get:
SemanticException [Error 10043]: Either list of columns or a custom serializer should be specified
The fact that we get the exceptions, and we fixed them, implies that the STORED BY is using DruidStorageHandler, but this message implies that for some reason it does not supply the column data from the Druid table. Anyone got past this point? I am thinking maybe its now a problem connecting to the Druid broker?
Created 09-13-2017 02:22 PM
@Adrian Gay either your broker address setting is wrong, this can be fixed by running hive with this config
--hiveconf hive.druid.broker.address.default=hostname_of_broker:8082 or the data source you are referring to doesn't exist in druid yet ("druid.datasource"="wikiticker")
Created 09-13-2017 02:51 PM
Appreciate u replying. I have been poking around in Hive Github and I see you are a significant contributor to Druid integration 🙂 I set
<property> <name>hive.druid.broker.address.default</name> <value>10.93.206.102:8082</value> </property>
in hive-site.xml and I am sure the datasource name I am using is correct because I can access Druid from Imply Pivot and Apache Superset using these params. Looking in the code for DruidStorageHandler I wonder if it may be something to do with me not setting the db type property anywhere?
final String dbType = HiveConf .getVar(SessionState.getSessionConf(), HiveConf.ConfVars.DRUID_METADATA_DB_TYPE);
as this would cause an exception in the constructor of the class?
if (dbType.equals("mysql")) { connector = new MySQLConnector(storageConnectorConfigSupplier, Suppliers.ofInstance(druidMetadataStorageTablesConfig) ); } else if (dbType.equals("postgresql")) { connector = new PostgreSQLConnector(storageConnectorConfigSupplier, Suppliers.ofInstance(druidMetadataStorageTablesConfig) ); } else { throw new IllegalStateException(String.format("Unknown metadata storage type [%s]", dbType)); }
I would have expected to see the exception logged at the hiveserver2 console on startup or when the [first] query is run... maybe? but I don't see anything but the semantic exception above.
Created 09-13-2017 05:24 PM
@Adrian Gay I don't think the metadata setup is the issue since it is set by default to mysql. What do you see in your hive server logs ? for instance if the hive server can connect to druid broker you should see something like this/
hive.log: error in initSerDe: org.apache.hadoop.hive.serde2.SerDeException Connected to Druid but could not retrieve datasource information org.apache.hadoop.hive.serde2.SerDeException: Connected to Druid but could not retrieve datasource information at org.apache.hadoop.hive.druid.serde.DruidSerDe.submitMetadataRequest(DruidSerDe.java:296) ~[hive-druid-handler-2.1.0.2.6.3.0-SNAPSHOT.jar:2.1.0.2.6.3.0-SNAPSHOT] at org.apache.hadoop.hive.druid.serde.DruidSerDe.initialize(DruidSerDe.java:178) ~[hive-druid-handler-2.1.0.2.6.3.0-SNAPSHOT.jar:2.1.0.2.6.3.0-SNAPSHOT] at org.apache.hadoop.hive.serde2.AbstractSerDe.initialize(AbstractSerDe.java:54) ~[hive-exec-2.1.0.2.6.3.0-SNAPSHOT.jar:2.1.0.2.6.3.0-SNAPSHOT] at org.apache.hadoop.hive.serde2.SerDeUtils.initializeSerDe(SerDeUtils.java:531) ~[hive-exec-2.1.0.2.6.3.0-SNAPSHOT.jar:2.1.0.2.6.3.0-SNAPSHOT] at org.apache.hadoop.hive.metastore.MetaStoreUtils.getDeserializer(MetaStoreUtils.java:436) ~[hive-exec-2.1.0.2.6.3.0-SNAPSHOT.jar:2.1.0.2.6.3.0-SNAPSHOT] at org.apache.hadoop.hive.metastore.MetaStoreUtils.getDeserializer(MetaStoreUtils.java:423) ~[hive-exec-2.1.0.2.6.3.0-SNAPSHOT.jar:2.1.0.2.6.3.0-SNAPSHOT] at org.apache.hadoop.hive.ql.metadata.Table.getDeserializerFromMetaStore(Table.java:279) ~[hive-exec-2.1.0.2.6.3.0-SNAPSHOT.jar:2.1.0.2.6.3.0-SNAPSHOT] at org.apache.hadoop.hive.ql.metadata.Table.getDeserializer(Table.java:261) ~[hive-exec-2.1.0.2.6.3.0-SNAPSHOT.jar:2.1.0.2.6.3.0-SNAPSHOT] at org.apache.hadoop.hive.ql.metadata.Table.getColsInternal(Table.java:639) ~[hive-exec-2.1.0.2.6.3.0-SNAPSHOT.jar:2.1.0.2.6.3.0-SNAPSHOT] at org.apache.hadoop.hive.ql.metadata.Table.getCols(Table.java:622) ~[hive-exec-2.1.0.2.6.3.0-SNAPSHOT.jar:2.1.0.2.6.3.0-SNAPSHOT] at org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:833) ~[hive-exec-2.1.0.2.6.3.0-SNAPSHOT.jar:2.1.0.2.6.3.0-SNAPSHOT] at org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:869) ~[hive-exec-2.1.0.2.6.3.0-SNAPSHOT.jar:2.1.0.2.6.3.0-SNAPSHOT] at org.apache.hadoop.hive.ql.exec.DDLTask.createTable(DDLTask.java:4227) ~[hive-exec-2.1.0.2.6.3.0-SNAPSHOT.jar:2.1.0.2.6.3.0-SNAPSHOT] at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:347) ~[hive-exec-2.1.0.2.6.3.0-SNAPSHOT.jar:2.1.0.2.6.3.0-SNAPSHOT] at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:197) ~[hive-exec-2.1.0.2.6.3.0-SNAPSHOT.jar:2.1.0.2.6.3.0-SNAPSHOT] at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:100) ~[hive-exec-2.1.0.2.6.3.0-SNAPSHOT.jar:2.1.0.2.6.3.0-SNAPSHOT] at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1905) ~[hive-exec-2.1.0.2.6.3.0-SNAPSHOT.jar:2.1.0.2.6.3.0-SNAPSHOT] at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1607) ~[hive-exec-2.1.0.2.6.3.0-SNAPSHOT.jar:2.1.0.2.6.3.0-SNAPSHOT] at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1354) ~[hive-exec-2.1.0.2.6.3.0-SNAPSHOT.jar:2.1.0.2.6.3.0-SNAPSHOT] at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1123) ~[hive-exec-2.1.0.2.6.3.0-SNAPSHOT.jar:2.1.0.2.6.3.0-SNAPSHOT] at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1111) ~[hive-exec-2.1.0.2.6.3.0-SNAPSHOT.jar:2.1.0.2.6.3.0-SNAPSHOT] at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:233) ~[hive-cli-2.1.0.2.6.3.0-SNAPSHOT.jar:2.1.0.2.6.3.0-SNAPSHOT] at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:184) ~[hive-cli-2.1.0.2.6.3.0-SNAPSHOT.jar:2.1.0.2.6.3.0-SNAPSHOT] at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:403) ~[hive-cli-2.1.0.2.6.3.0-SNAPSHOT.jar:2.1.0.2.6.3.0-SNAPSHOT] at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:821) ~[hive-cli-2.1.0.2.6.3.0-SNAPSHOT.jar:2.1.0.2.6.3.0-SNAPSHOT] at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:759) ~[hive-cli-2.1.0.2.6.3.0-SNAPSHOT.jar:2.1.0.2.6.3.0-SNAPSHOT] at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:686) ~[hive-cli-2.1.0.2.6.3.0-SNAPSHOT.jar:2.1.0.2.6.3.0-SNAPSHOT] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_77] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_77] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_77] at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_77] at org.apache.hadoop.util.RunJar.run(RunJar.java:233) ~[hadoop-common-2.7.3.2.5.0.1-216.jar:?] at org.apache.hadoop.util.RunJar.main(RunJar.java:148) ~[hadoop-common-2.7.3.2.5.0.1-216.jar:?] 2017-09-13T10:16:51,941 ERROR [6f8f271e-3af1-4385-a79a-4d37aecb9fde main] metadata.Table: Unable to get field from serde: org.apache.hadoop.hive.druid.serde.DruidSerDe java.lang.RuntimeException: MetaException(message:org.apache.hadoop.hive.serde2.SerDeException Connected to Druid but could not retrieve datasource information) at org.apache.hadoop.hive.ql.metadata.Table.getDeserializerFromMetaStore(Table.java:281) ~[hive-exec-2.1.0.2.6.3.0-SNAPSHOT.jar:2.1.0.2.6.3.0-SNAPSHOT] at org.apache.hadoop.hive.ql.metadata.Table.getDeserializer(Table.java:261) ~[hive-exec-2.1.0.2.6.3.0-SNAPSHOT.jar:2.1.0.2.6.3.0-SNAPSHOT] at org.apache.hadoop.hive.ql.metadata.Table.getColsInternal(Table.java:639) ~[hive-exec-2.1.0.2.6.3.0-SNAPSHOT.jar:2.1.0.2.6.3.0-SNAPSHOT] at org.apache.hadoop.hive.ql.metadata.Table.getCols(Table.java:622) ~[hive-exec-2.1.0.2.6.3.0-SNAPSHOT.jar:2.1.0.2.6.3.0-SNAPSHOT] at org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:833) ~[hive-exec-2.1.0.2.6.3.0-SNAPSHOT.jar:2.1.0.2.6.3.0-SNAPSHOT] at org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:869) ~[hive-exec-2.1.0.2.6.3.0-SNAPSHOT.jar:2.1.0.2.6.3.0-SNAPSHOT] at org.apache.hadoop.hive.ql.exec.DDLTask.createTable(DDLTask.java:4227) ~[hive-exec-2.1.0.2.6.3.0-SNAPSHOT.jar:2.1.0.2.6.3.0-SNAPSHOT] at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:347) ~[hive-exec-2.1.0.2.6.3.0-SNAPSHOT.jar:2.1.0.2.6.3.0-SNAPSHOT] at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:197) ~[hive-exec-2.1.0.2.6.3.0-SNAPSHOT.jar:2.1.0.2.6.3.0-SNAPSHOT] at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:100) ~[hive-exec-2.1.0.2.6.3.0-SNAPSHOT.jar:2.1.0.2.6.3.0-SNAPSHOT] at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1905) ~[hive-exec-2.1.0.2.6.3.0-SNAPSHOT.jar:2.1.0.2.6.3.0-SNAPSHOT] at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1607) ~[hive-exec-2.1.0.2.6.3.0-SNAPSHOT.jar:2.1.0.2.6.3.0-SNAPSHOT] at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1354) ~[hive-exec-2.1.0.2.6.3.0-SNAPSHOT.jar:2.1.0.2.6.3.0-SNAPSHOT] at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1123) ~[hive-exec-2.1.0.2.6.3.0-SNAPSHOT.jar:2.1.0.2.6.3.0-SNAPSHOT] at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1111) ~[hive-exec-2.1.0.2.6.3.0-SNAPSHOT.jar:2.1.0.2.6.3.0-SNAPSHOT] at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:233) ~[hive-cli-2.1.0.2.6.3.0-SNAPSHOT.jar:2.1.0.2.6.3.0-SNAPSHOT] at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:184) ~[hive-cli-2.1.0.2.6.3.0-SNAPSHOT.jar:2.1.0.2.6.3.0-SNAPSHOT] at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:403) ~[hive-cli-2.1.0.2.6.3.0-SNAPSHOT.jar:2.1.0.2.6.3.0-SNAPSHOT] at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:821) ~[hive-cli-2.1.0.2.6.3.0-SNAPSHOT.jar:2.1.0.2.6.3.0-SNAPSHOT] at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:759) ~[hive-cli-2.1.0.2.6.3.0-SNAPSHOT.jar:2.1.0.2.6.3.0-SNAPSHOT] at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:686) ~[hive-cli-2.1.0.2.6.3.0-SNAPSHOT.jar:2.1.0.2.6.3.0-SNAPSHOT] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_77] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_77] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_77] at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_77] at org.apache.hadoop.util.RunJar.run(RunJar.java:233) ~[hadoop-common-2.7.3.2.5.0.1-216.jar:?] at org.apache.hadoop.util.RunJar.main(RunJar.java:148) ~[hadoop-common-2.7.3.2.5.0.1-216.jar:?]
Created 02-15-2018 06:48 AM
HI @Slim
I'm seeing "Connected to Druid but could not retrieve datasource information" when I create a table.
Could you have any idea where I should check?