Spark unsupported fearutes in CDP

Hi @Rajeshhadoop ,

The Spark DataSource API has the following syntax:

val jdbcDF = spark.read.format("jdbc").option("url", "jdbc:...")...load()

Please see:

https://spark.apache.org/docs/2.4.0/sql-data-sources-jdbc.html 

The problem with this approach - using with Hive or Impala is that since the above may run on multiple executors, this could overwhelm and essentially DDOS the Hive / Impala service.  As the documentation states this is not a supported way of connecting from Spark to Hive/Impala.

However you should be able to still connect to Hive and Impala through a simple JDBC connection using "java.sql.DriverManager" or "java.sql.Connection". That in contrast runs only on a single thread, on the Spark driver side - and will create a single connection to a HiveServer2 / Impala daemon instance. The throughput between the Spark driver and Hive/Impala of course is limited with this approach, please use it for simple queries or submitting DDL/DML  queries. Please see 

https://www.cloudera.com/downloads/connectors/hive/jdbc.html

https://www.cloudera.com/downloads/connectors/impala/jdbc.html 

for the JDBC drivers and for examples.

Independently of the above, you can still access Hive tables' data through SparkSQL with

val df = spark.sql("select ... from ...")

which is the recommended way of accessing and manipulating Hive table data from Spark as it is parallelized through the Spark executors. See docs:

https://spark.apache.org/docs/2.4.0/sql-data-sources-hive-tables.html 

I hope this clarifies it.

Best regards

 Miklos