Hi experts:
The Hadoop version coming with CDH-6.3.4 is Hadoop 3.0.0-cdh6.3.4. The Apache Spark web site does not have a prebuilt tarball for Hadoop 3.0.0, so I downloaded "spark-3.0.1-bin-hadoop3.2.tgz". Untar'red and tried it on our CDH 6.3.4 cluster.
Simple Spark line counting works fine. But in a pyspark session 'show tables' in a hive database working fine, but creating a table fails with an error as:
pyspark.sql.utils.AnalysisException: org.apache.hadoop.hive.ql.metadata.HiveException: Unable to fetch table messages1. Invalid method name: 'get_table_req';
That is very similar to what is described here:
https://stackoverflow.com/questions/63476121/hive-queries-failing-with-unable-to-fetch-table-test-ta...
I tried to replace these hive related jars under Spark 3.0.1 jars subdirectory with the correspondent ones in /opt/cloudera/parcels/CDH-6.3.4-1.cdh6.3.4.p0.6626826/jars, it does not help - failed with different error.
Does anyone have some experience with running Spark 3 in a CDH 6.3.x cluster? Can you suggest anything to try?
Your help is greatly appreciated!
Regards.
Vincent