Does HiveKafkaStorageHandler support SSL?

[#1]

Hi all,

I've been looking for solutions to expose Kafka data in Hive. I found a few things, however HiveKafkaStorageHandler caught my attention :).

I was looking for trying it out, but I don't find anywhere in the documentation how does it support a connection to Kafka through SSL (which is a requirement for my setup).

Can someone tell me for sure if this storage handler supports a broker listener that uses SSL?

Thanks a lot in advance !

[#2]

yes it supports SSL, you can pass required params in TBLPROPERTIES

Example:

CREATE EXTERNAL TABLE kafka_hive_table_edgenode_SASL_SSL(`Country Name` string , `Language` string, `_id` struct<`$oid`:string,`name`:string>, `account` struct<`$accountstat`:string>) STORED BY 'org.apache.hadoop.hive.kafka.KafkaStorageHandler' TBLPROPERTIES("kafka.topic" = "kafka_hive_topic", "kafka.bootstrap.servers"="<kafka-broker>:<port>","kafka.consumer.security.protocol" = "SASL_SSL","kafka.consumer.ssl.truststore.location"="<truststore-path>","kafka.consumer.ssl.truststore.password"="<password>");

[#3]

Hi @Manohar Vanam,

Thanks a lot for your quick answer.

I was trying to test KafkaStorageHandler but I was not able to get it to work yet. I am having the following error:

Exception in thread “main” java.lang.NoClassDefFoundError: org/apache/hadoop/hive/ql/metadata/StorageHandlerInfo
       at java.lang.Class.forName0(Native Method)
       at java.lang.Class.forName(Class.java:348)
       at org.apache.hadoop.hive.ql.parse.ParseUtils.ensureClassExists(ParseUtils.java:261)
       at org.apache.hadoop.hive.ql.parse.StorageFormat.fillStorageFormat(StorageFormat.java:64)
       at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeCreateTable(SemanticAnalyzer.java:11907)
       at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genResolvedParseTree(SemanticAnalyzer.java:11040)
       at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:11153)
       at org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:286)
       at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:258)
       at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:512)
       at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1317)
       at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1457)
       at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1237)
       at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1227)
       at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:233)
       at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:184)
       at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:403)
       at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:821)
       at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:759)
       at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:686)
       at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
       at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
       at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
       at java.lang.reflect.Method.invoke(Method.java:498)
       at org.apache.hadoop.util.RunJar.run(RunJar.java:239)
       at org.apache.hadoop.util.RunJar.main(RunJar.java:153)
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.hive.ql.metadata.StorageHandlerInfo
       at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
       at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
       at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
       at java.lang.ClassLoader.loadClass(ClassLoader.java:357)

I think this might be due to the version of Hive I am using. The setup I have uses Hive 2.3.x but apparently the lib for KafkaStorageHandler is for 3.1.x.

Is this the issue? Have you had the same problem?

Thanks!

[#4 · Accepted solution]

@saulo_sobreiro 

Are you using HDP 2.3? if so, you can try with HDP 3.1 which has KafkaStorageHandler.

If you are using opensource hive 2.3 then try to use any version above 3.1.

[#5]

Hi,

I am actually using an Amazon EMR, which does not yet support hive 3.x in any version of EMR clusters, so this might make me drop this idea :(.

Is there any workaround? Perhaps something like a set of libs we can import in Hive 2.x so it supports HiveKafkaStorageHandler?

Thanks again!