Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Kafka Hive integration error related most possibly to different guava versions

Solved Go to solution

Kafka Hive integration error related most possibly to different guava versions

New Contributor

Hello all,

 

I tried to search Hive related topics but did not find the issue I face so I wanted to ask here.

 

This is CDH 6.3.2 (Parcels) on Linux (CentOS)

 

I have Kafka and Hive and I am able to create external Kafka tables after putting hive-exec-3.1.3000.7.1.3.0-100.jar and kafka-handler-3.1.3000.7.1.3.0-100.jar in Hive under Hive Auxiliary JARs Directory

 

However, when I try to select data from the external table I get this error:

 

0: jdbc:hive2://localhost:10000/testdb> select * from kafka_table;
Error: java.io.IOException: java.io.IOException: java.util.concurrent.ExecutionException: java.lang.NoSuchMethodError: com.google.common.base.Preconditions.checkArgument(ZLjava/lang/String;JJ)V (state=,code=0)

 

After looking for this error I've found that it is most likely related to issues with different versions of guava in Hadoop and Hive: https://issues.apache.org/jira/browse/HIVE-22915

 

I see that Hadoop is using 

 

/opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/lib/hadoop/client/guava.jar
/opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/lib/hadoop/client/guava-11.0.2.jar
/opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/lib/hadoop/lib/guava-11.0.2.jar
/opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/lib/hadoop-hdfs/lib/guava-11.0.2.jar

 

And Hive is using

 

/opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/lib/hive/lib/guava-14.0.2.jar

 

And Kafka is using

 

/opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/lib/kafka/libs/guava-20.0.jar

 

However following the advice from the HIVE Jira above if I equalize Hive and Hadoop versions (either downgrade Hive to guava-11.0.2.jar or upgrade Hadoop in all locations above to guava-14.0.2.jar) the errors persist.

 

If I upgrade Hive to guava-20.0.jar I get multiple issues and I am not even able to read local tables:

 

0: jdbc:hive2://localhost:10000/testdb> desc kafka_table;
Error: Error running query: java.lang.IllegalAccessError: tried to access method com.google.common.collect.Iterators.emptyIterator()Lcom/google/common/collect/UnmodifiableIterator; from class org.apache.hadoop.hive.ql.exec.FetchOperator (state=,code=0)

 

If I restore Hive to either guava-11.0.2.jar or guava-14.0.2.jar I am again able to select data from tables but initial error is still in place and I cannot read Kafka topic messages in Hive.

 

Has anyone else faced this already and if yes what is recommended way to resolve this in Cloudera environment?

 

Thanks a lot in advance.

 

Warm regards,

Yavor

1 ACCEPTED SOLUTION

Accepted Solutions
Highlighted

Re: Kafka Hive integration error related most possibly to different guava versions

Super Guru

I believe you need to be running HDP 3.1 or CDP for the Hive-Kafka integration since it's part of Hive 3.x and CDH has Hive 1 or 2.

View solution in original post

3 REPLIES 3
Highlighted

Re: Kafka Hive integration error related most possibly to different guava versions

Super Guru

I believe you need to be running HDP 3.1 or CDP for the Hive-Kafka integration since it's part of Hive 3.x and CDH has Hive 1 or 2.

View solution in original post

Highlighted

Re: Kafka Hive integration error related most possibly to different guava versions

Super Guru
Highlighted

Re: Kafka Hive integration error related most possibly to different guava versions

New Contributor

Thanks a lot @TimothySpann for your time and insight. Your advice saves me from future futile efforts to hack/mess up with guava jar libs. I will probably try different approach to load Kafka topics into Hive. Thanks again.

Don't have an account?
Coming from Hortonworks? Activate your account here