Hello all,
I tried to search Hive related topics but did not find the issue I face so I wanted to ask here.
This is CDH 6.3.2 (Parcels) on Linux (CentOS)
I have Kafka and Hive and I am able to create external Kafka tables after putting hive-exec-3.1.3000.7.1.3.0-100.jar and kafka-handler-3.1.3000.7.1.3.0-100.jar in Hive under Hive Auxiliary JARs Directory
However, when I try to select data from the external table I get this error:
0: jdbc:hive2://localhost:10000/testdb> select * from kafka_table;
Error: java.io.IOException: java.io.IOException: java.util.concurrent.ExecutionException: java.lang.NoSuchMethodError: com.google.common.base.Preconditions.checkArgument(ZLjava/lang/String;JJ)V (state=,code=0)
After looking for this error I've found that it is most likely related to issues with different versions of guava in Hadoop and Hive: https://issues.apache.org/jira/browse/HIVE-22915
I see that Hadoop is using
/opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/lib/hadoop/client/guava.jar
/opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/lib/hadoop/client/guava-11.0.2.jar
/opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/lib/hadoop/lib/guava-11.0.2.jar
/opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/lib/hadoop-hdfs/lib/guava-11.0.2.jar
And Hive is using
/opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/lib/hive/lib/guava-14.0.2.jar
And Kafka is using
/opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/lib/kafka/libs/guava-20.0.jar
However following the advice from the HIVE Jira above if I equalize Hive and Hadoop versions (either downgrade Hive to guava-11.0.2.jar or upgrade Hadoop in all locations above to guava-14.0.2.jar) the errors persist.
If I upgrade Hive to guava-20.0.jar I get multiple issues and I am not even able to read local tables:
0: jdbc:hive2://localhost:10000/testdb> desc kafka_table;
Error: Error running query: java.lang.IllegalAccessError: tried to access method com.google.common.collect.Iterators.emptyIterator()Lcom/google/common/collect/UnmodifiableIterator; from class org.apache.hadoop.hive.ql.exec.FetchOperator (state=,code=0)
If I restore Hive to either guava-11.0.2.jar or guava-14.0.2.jar I am again able to select data from tables but initial error is still in place and I cannot read Kafka topic messages in Hive.
Has anyone else faced this already and if yes what is recommended way to resolve this in Cloudera environment?
Thanks a lot in advance.
Warm regards,
Yavor
