Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Does HiveKafkaStorageHandler support SSL?

Solved Go to solution
Highlighted

Does HiveKafkaStorageHandler support SSL?

New Contributor

Hi all,


I've been looking for solutions to expose Kafka data in Hive. I found a few things, however HiveKafkaStorageHandler caught my attention :).

I was looking for trying it out, but I don't find anywhere in the documentation how does it support a connection to Kafka through SSL (which is a requirement for my setup).


Can someone tell me for sure if this storage handler supports a broker listener that uses SSL?


Thanks a lot in advance !

1 ACCEPTED SOLUTION

Accepted Solutions

Re: Does HiveKafkaStorageHandler support SSL?

Cloudera Employee

@saulo_sobreiro 

Are you using HDP 2.3? if so, you can try with HDP 3.1 which has KafkaStorageHandler.

If you are using opensource hive 2.3 then try to use any version above 3.1.

4 REPLIES 4

Re: Does HiveKafkaStorageHandler support SSL?

Cloudera Employee

yes it supports SSL, you can pass required params in TBLPROPERTIES

Example:

CREATE EXTERNAL TABLE kafka_hive_table_edgenode_SASL_SSL(`Country Name` string , `Language` string, `_id` struct<`$oid`:string,`name`:string>, `account` struct<`$accountstat`:string>) STORED BY 'org.apache.hadoop.hive.kafka.KafkaStorageHandler' TBLPROPERTIES("kafka.topic" = "kafka_hive_topic", "kafka.bootstrap.servers"="<kafka-broker>:<port>","kafka.consumer.security.protocol" = "SASL_SSL","kafka.consumer.ssl.truststore.location"="<truststore-path>","kafka.consumer.ssl.truststore.password"="<password>");

Re: Does HiveKafkaStorageHandler support SSL?

New Contributor

Hi @Manohar Vanam,


Thanks a lot for your quick answer.

I was trying to test KafkaStorageHandler but I was not able to get it to work yet. I am having the following error:

Exception in thread “main” java.lang.NoClassDefFoundError: org/apache/hadoop/hive/ql/metadata/StorageHandlerInfo

       at java.lang.Class.forName0(Native Method)

       at java.lang.Class.forName(Class.java:348)

       at org.apache.hadoop.hive.ql.parse.ParseUtils.ensureClassExists(ParseUtils.java:261)

       at org.apache.hadoop.hive.ql.parse.StorageFormat.fillStorageFormat(StorageFormat.java:64)

       at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeCreateTable(SemanticAnalyzer.java:11907)

       at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genResolvedParseTree(SemanticAnalyzer.java:11040)

       at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:11153)

       at org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:286)

       at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:258)

       at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:512)

       at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1317)

       at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1457)

       at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1237)

       at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1227)

       at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:233)

       at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:184)

       at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:403)

       at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:821)

       at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:759)

       at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:686)

       at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

       at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

       at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

       at java.lang.reflect.Method.invoke(Method.java:498)

       at org.apache.hadoop.util.RunJar.run(RunJar.java:239)

       at org.apache.hadoop.util.RunJar.main(RunJar.java:153)

Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.hive.ql.metadata.StorageHandlerInfo

       at java.net.URLClassLoader.findClass(URLClassLoader.java:382)

       at java.lang.ClassLoader.loadClass(ClassLoader.java:424)

       at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)

       at java.lang.ClassLoader.loadClass(ClassLoader.java:357)


I think this might be due to the version of Hive I am using. The setup I have uses Hive 2.3.x but apparently the lib for KafkaStorageHandler is for 3.1.x.

Is this the issue? Have you had the same problem?


Thanks!

Re: Does HiveKafkaStorageHandler support SSL?

Cloudera Employee

@saulo_sobreiro 

Are you using HDP 2.3? if so, you can try with HDP 3.1 which has KafkaStorageHandler.

If you are using opensource hive 2.3 then try to use any version above 3.1.

Re: Does HiveKafkaStorageHandler support SSL?

New Contributor

Hi,

 

I am actually using an Amazon EMR, which does not yet support hive 3.x in any version of EMR clusters, so this might make me drop this idea :(.

 

Is there any workaround? Perhaps something like a set of libs we can import in Hive 2.x so it supports HiveKafkaStorageHandler?

 

Thanks again!