Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Kafka Handler for CDH 6.1.1 (Hive 2.1.1)

avatar
New Contributor

We have CDH 6.1.1 cluster and Kafka cluster. I want to make possible to select data from Kafka topics using Hive. Is there any way to do it? Where I can find Kafka storage handler version for Hive 2.1.1 ?

1 ACCEPTED SOLUTION

avatar
Expert Contributor

Hello @d33play 

I checked the documents again and found there is an internal jira CDH-72683 which was tracked for CDH versions to support Kafka Storage Handler. 
Kafka Storage Handler is a part of Hive 3.x version but CDH 6.1.1 comes with Apache Hive 2.1.1

Hence it is not supported in CDH versions but is available with CDP.

https://docs.cloudera.com/runtime/7.2.0/release-notes/topics/rt-runtime-component-versions.html

https://docs.cloudera.com/cdp-private-cloud-base/7.1.5/runtime-release-notes/topics/rt-pvc-runtime-c...

 

So, the KafkaStorageHandler package will not work with Hive 2.1.1.

 

Hope this information helps.

 

Thanks and Regards,

Nandini

 

SME || Kafka | Schema Registry | SMM | SRM

View solution in original post

3 REPLIES 3

avatar
Expert Contributor

I believe this is what you are looking for 

https://docs.cloudera.com/cdp-private-cloud-base/7.1.5/integrating-hive-and-bi/topics/hive-kafka-int...

https://blog.cloudera.com/introducing-hive-kafka-sql/

 

And a more comprehensive doc - https://docs.cloudera.com/HDPDocuments/HDF3/HDF-3.5.1/kafka-hive-integration/hdf-kafka-hive-integrat...

 

You would not have to worry about the versions as the components are tested for cross compatibility. 

SME || Kafka | Schema Registry | SMM | SRM

avatar
New Contributor

Thank you for reply. Is there any installation guide (which jars, where to place etc.)?

I've tried ADD JAR from hive CLI (not beeline), then I trying to create kafka storage table it's returns error (Error: Error running query: java.lang.NoClassDefFoundError: org/apache/hadoop/hive/metastore/DefaultHiveMetaHook).

avatar
Expert Contributor

Hello @d33play 

I checked the documents again and found there is an internal jira CDH-72683 which was tracked for CDH versions to support Kafka Storage Handler. 
Kafka Storage Handler is a part of Hive 3.x version but CDH 6.1.1 comes with Apache Hive 2.1.1

Hence it is not supported in CDH versions but is available with CDP.

https://docs.cloudera.com/runtime/7.2.0/release-notes/topics/rt-runtime-component-versions.html

https://docs.cloudera.com/cdp-private-cloud-base/7.1.5/runtime-release-notes/topics/rt-pvc-runtime-c...

 

So, the KafkaStorageHandler package will not work with Hive 2.1.1.

 

Hope this information helps.

 

Thanks and Regards,

Nandini

 

SME || Kafka | Schema Registry | SMM | SRM