Support Questions
Find answers, ask questions, and share your expertise

Kafka Handler for CDH 6.1.1 (Hive 2.1.1)

New Contributor

We have CDH 6.1.1 cluster and Kafka cluster. I want to make possible to select data from Kafka topics using Hive. Is there any way to do it? Where I can find Kafka storage handler version for Hive 2.1.1 ?

1 ACCEPTED SOLUTION

Contributor

Hello @d33play 

I checked the documents again and found there is an internal jira CDH-72683 which was tracked for CDH versions to support Kafka Storage Handler. 
Kafka Storage Handler is a part of Hive 3.x version but CDH 6.1.1 comes with Apache Hive 2.1.1

Hence it is not supported in CDH versions but is available with CDP.

https://docs.cloudera.com/runtime/7.2.0/release-notes/topics/rt-runtime-component-versions.html

https://docs.cloudera.com/cdp-private-cloud-base/7.1.5/runtime-release-notes/topics/rt-pvc-runtime-c...

 

So, the KafkaStorageHandler package will not work with Hive 2.1.1.

 

Hope this information helps.

 

Thanks and Regards,

Nandini

 

SME || Kafka | Schema Registry | SMM | SRM

View solution in original post

3 REPLIES 3

Contributor

I believe this is what you are looking for 

https://docs.cloudera.com/cdp-private-cloud-base/7.1.5/integrating-hive-and-bi/topics/hive-kafka-int...

https://blog.cloudera.com/introducing-hive-kafka-sql/

 

And a more comprehensive doc - https://docs.cloudera.com/HDPDocuments/HDF3/HDF-3.5.1/kafka-hive-integration/hdf-kafka-hive-integrat...

 

You would not have to worry about the versions as the components are tested for cross compatibility. 

SME || Kafka | Schema Registry | SMM | SRM

New Contributor

Thank you for reply. Is there any installation guide (which jars, where to place etc.)?

I've tried ADD JAR from hive CLI (not beeline), then I trying to create kafka storage table it's returns error (Error: Error running query: java.lang.NoClassDefFoundError: org/apache/hadoop/hive/metastore/DefaultHiveMetaHook).

Contributor

Hello @d33play 

I checked the documents again and found there is an internal jira CDH-72683 which was tracked for CDH versions to support Kafka Storage Handler. 
Kafka Storage Handler is a part of Hive 3.x version but CDH 6.1.1 comes with Apache Hive 2.1.1

Hence it is not supported in CDH versions but is available with CDP.

https://docs.cloudera.com/runtime/7.2.0/release-notes/topics/rt-runtime-component-versions.html

https://docs.cloudera.com/cdp-private-cloud-base/7.1.5/runtime-release-notes/topics/rt-pvc-runtime-c...

 

So, the KafkaStorageHandler package will not work with Hive 2.1.1.

 

Hope this information helps.

 

Thanks and Regards,

Nandini

 

SME || Kafka | Schema Registry | SMM | SRM
; ;