question Kafka Handler for CDH 6.1.1 (Hive 2.1.1) in Support Questions

Kafka Handler for CDH 6.1.1 (Hive 2.1.1)

d33play — Fri, 12 Mar 2021 13:49:44 GMT

We have CDH 6.1.1 cluster and Kafka cluster. I want to make possible to select data from Kafka topics using Hive. Is there any way to do it? Where I can find Kafka storage handler version for Hive 2.1.1 ?

Re: Kafka Handler for CDH 6.1.1 (Hive 2.1.1)

Nandinin — Fri, 12 Mar 2021 17:00:14 GMT

I believe this is what you are looking for

https://docs.cloudera.com/cdp-private-cloud-base/7.1.5/integrating-hive-and-bi/topics/hive-kafka-integration.html

https://blog.cloudera.com/introducing-hive-kafka-sql/

And a more comprehensive doc - https://docs.cloudera.com/HDPDocuments/HDF3/HDF-3.5.1/kafka-hive-integration/hdf-kafka-hive-integration.pdf

You would not have to worry about the versions as the components are tested for cross compatibility.

Re: Kafka Handler for CDH 6.1.1 (Hive 2.1.1)

d33play — Fri, 12 Mar 2021 19:38:13 GMT

Thank you for reply. Is there any installation guide (which jars, where to place etc.)?

I've tried ADD JAR from hive CLI (not beeline), then I trying to create kafka storage table it's returns error (Error: Error running query: java.lang.NoClassDefFoundError: org/apache/hadoop/hive/metastore/DefaultHiveMetaHook).

Re: Kafka Handler for CDH 6.1.1 (Hive 2.1.1)

Nandinin — Tue, 16 Mar 2021 08:28:11 GMT

Hello @d33play

I checked the documents again and found there is an internal jira CDH-72683 which was tracked for CDH versions to support Kafka Storage Handler.
Kafka Storage Handler is a part of Hive 3.x version but CDH 6.1.1 comes with Apache Hive 2.1.1

Hence it is not supported in CDH versions but is available with CDP.

https://docs.cloudera.com/runtime/7.2.0/release-notes/topics/rt-runtime-component-versions.html

https://docs.cloudera.com/cdp-private-cloud-base/7.1.5/runtime-release-notes/topics/rt-pvc-runtime-component-versions.html

So, the KafkaStorageHandler package will not work with Hive 2.1.1.

Hope this information helps.

Thanks and Regards,

Nandini