Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Spark 3.0.1 on a CDH-6.3.4 cluster

avatar
Contributor

Hi experts:

 

The Hadoop version coming with CDH-6.3.4 is Hadoop 3.0.0-cdh6.3.4. The Apache Spark web site does not have a prebuilt tarball for Hadoop 3.0.0, so I downloaded "spark-3.0.1-bin-hadoop3.2.tgz". Untar'red and tried it on our CDH 6.3.4 cluster. 

 

Simple Spark line counting works fine. But in a pyspark session 'show tables' in a hive database working fine, but creating a table fails with an error as:

pyspark.sql.utils.AnalysisException: org.apache.hadoop.hive.ql.metadata.HiveException: Unable to fetch table messages1. Invalid method name: 'get_table_req';

 

That is very similar to what is described here:

https://stackoverflow.com/questions/63476121/hive-queries-failing-with-unable-to-fetch-table-test-ta...

 

I tried to replace these hive related jars under Spark 3.0.1 jars subdirectory with the correspondent ones in /opt/cloudera/parcels/CDH-6.3.4-1.cdh6.3.4.p0.6626826/jars, it does not help - failed with different error.

 

Does anyone have some experience with running Spark 3 in a CDH 6.3.x cluster? Can you suggest anything to try?

 

Your help is greatly appreciated!

 

Regards.

Vincent 

 

2 REPLIES 2

avatar
Expert Contributor

avatar
Contributor

Thanks very much @MyNamesNotRick . We will check them out.