Reply
Expert Contributor
Posts: 253
Registered: ‎01-25-2017

Re: Having Spark 1.6.0 and 2.1 in the same CDH

i already restarted them as the process of adding the spark2 procedure.

I will give it another restart and check
Cloudera Employee
Posts: 16
Registered: ‎11-16-2015

Re: Having Spark 1.6.0 and 2.1 in the same CDH

+1 @mbigelow 

This (not seeing Spark2 in 'Add a Service' wizard) is generally a result of Cloudera Management Services not being restarted (or CM not recognizing the CSD)

Explorer
Posts: 6
Registered: ‎04-11-2017

Re: Having Spark 1.6.0 and 2.1 in the same CDH

How do I connect to the Hive metastore from Spark2.

 

Thx

Renjith

Highlighted
Posts: 642
Topics: 3
Kudos: 103
Solutions: 66
Registered: ‎08-16-2016

Re: Having Spark 1.6.0 and 2.1 in the same CDH

In the Spark2 configs, ensure that the Hive service is enabled. This will include the Hive client configs for the Spark2 service. This will allow the SparkSession created by spark2-shell to have Hive support for the HMS on the cluster.

I haven't tested actual Spark2 applications but with the above setup it should be as simple as using the .enableHiveSupport in the SparkSession builder method.

Outside of that you would probably need to include the hive-site.xml or Hive HMS settings in the Spark Context configuration object and then us .enableHiveSupport.
Announcements