Kafka Hive integration error related most possibly to different guava versions

[#1]

Hello all,

I tried to search Hive related topics but did not find the issue I face so I wanted to ask here.

This is CDH 6.3.2 (Parcels) on Linux (CentOS)

I have Kafka and Hive and I am able to create external Kafka tables after putting hive-exec-3.1.3000.7.1.3.0-100.jar and kafka-handler-3.1.3000.7.1.3.0-100.jar in Hive under Hive Auxiliary JARs Directory

However, when I try to select data from the external table I get this error:

0: jdbc:hive2://localhost:10000/testdb> select * from kafka_table;
Error: java.io.IOException: java.io.IOException: java.util.concurrent.ExecutionException: java.lang.NoSuchMethodError: com.google.common.base.Preconditions.checkArgument(ZLjava/lang/String;JJ)V (state=,code=0)

After looking for this error I've found that it is most likely related to issues with different versions of guava in Hadoop and Hive: https://issues.apache.org/jira/browse/HIVE-22915

I see that Hadoop is using 

/opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/lib/hadoop/client/guava.jar
/opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/lib/hadoop/client/guava-11.0.2.jar
/opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/lib/hadoop/lib/guava-11.0.2.jar
/opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/lib/hadoop-hdfs/lib/guava-11.0.2.jar

And Hive is using

/opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/lib/hive/lib/guava-14.0.2.jar

And Kafka is using

/opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/lib/kafka/libs/guava-20.0.jar

However following the advice from the HIVE Jira above if I equalize Hive and Hadoop versions (either downgrade Hive to guava-11.0.2.jar or upgrade Hadoop in all locations above to guava-14.0.2.jar) the errors persist.

If I upgrade Hive to guava-20.0.jar I get multiple issues and I am not even able to read local tables:

0: jdbc:hive2://localhost:10000/testdb> desc kafka_table;
Error: Error running query: java.lang.IllegalAccessError: tried to access method com.google.common.collect.Iterators.emptyIterator()Lcom/google/common/collect/UnmodifiableIterator; from class org.apache.hadoop.hive.ql.exec.FetchOperator (state=,code=0)

If I restore Hive to either guava-11.0.2.jar or guava-14.0.2.jar I am again able to select data from tables but initial error is still in place and I cannot read Kafka topic messages in Hive.

Has anyone else faced this already and if yes what is recommended way to resolve this in Cloudera environment?

Thanks a lot in advance.

Warm regards,

Yavor

[#2 · Accepted solution]

I believe you need to be running HDP 3.1 or CDP for the Hive-Kafka integration since it's part of Hive 3.x and CDH has Hive 1 or 2.

[#3]

https://docs.cloudera.com/runtime/7.2.1/integrating-hive-and-bi/topics/hive-kafka-integration.html

[#4]

Thanks a lot @TimothySpann for your time and insight. Your advice saves me from future futile efforts to hack/mess up with guava jar libs. I will probably try different approach to load Kafka topics into Hive. Thanks again.