Spark Connect to CDP Warehouse using Hive JDBC Method not supported error

Question

Hello, I've been trying to use Spark jdbc to connect to a CDP cloud virtual warehouse using the Hive jar (hive-jdbc-3.1.0-SNAPSHOT-standalone.jar) but I have the following error

Caused by: java.sql.SQLFeatureNotSupportedException: Method not supported
at org.apache.hive.jdbc.HivePreparedStatement.addBatch(HivePreparedStatement.java:78)

Looking into the jar sources I see that the addBatch method throws the exception by default. Spark needs to use this method to insert the data into hive. Is there any other driver that can be used to overcome this problem.If no, does cloudera will support spark to connect to CDP virtual warehouse.

Thanks in advance

Best regards,

Diego

PaulDM · Accepted Answer

Hi @dcardenas
In CDH/CDP clusters, the integration between Spark and Hive service is get done by implementing Spark SQL API for Hive External tables through HiveMetastore and HWC (Hive Warehouse Connector) to Hive Managed/ACID tables.

Therefore, Spark accessing Hive with JDBC is not supported, please check Unsupported Apache Spark Features

DianaTorres · Answer

@dcardenas Welcome to the Cloudera Community!

To help you get the best possible solution, I have tagged our Spark experts @Bharati and @jagadeesan who may be able to assist you further.

Please keep us updated on your post, and we hope you find a satisfactory solution to your query.

Sign up

Please use your Cloudera account.

Login to the community

Please use your Cloudera account.