Reply
Expert Contributor
Posts: 277
Registered: ‎01-25-2017

Re: Having Spark 1.6.0 and 2.1 in the same CDH

i already restarted them as the process of adding the spark2 procedure.

I will give it another restart and check
Cloudera Employee
Posts: 31
Registered: ‎11-16-2015

Re: Having Spark 1.6.0 and 2.1 in the same CDH

+1 @mbigelow 

This (not seeing Spark2 in 'Add a Service' wizard) is generally a result of Cloudera Management Services not being restarted (or CM not recognizing the CSD)

Explorer
Posts: 6
Registered: ‎04-11-2017

Re: Having Spark 1.6.0 and 2.1 in the same CDH

How do I connect to the Hive metastore from Spark2.

 

Thx

Renjith

Posts: 642
Topics: 3
Kudos: 105
Solutions: 67
Registered: ‎08-16-2016

Re: Having Spark 1.6.0 and 2.1 in the same CDH

In the Spark2 configs, ensure that the Hive service is enabled. This will include the Hive client configs for the Spark2 service. This will allow the SparkSession created by spark2-shell to have Hive support for the HMS on the cluster.

I haven't tested actual Spark2 applications but with the above setup it should be as simple as using the .enableHiveSupport in the SparkSession builder method.

Outside of that you would probably need to include the hive-site.xml or Hive HMS settings in the Spark Context configuration object and then us .enableHiveSupport.
New Contributor
Posts: 4
Registered: ‎10-18-2017

Re: Having Spark 1.6.0 and 2.1 in the same CDH

Did this work for you ? I'm facing same issue, after successfully distributing and activating parcel, not able to see spark 2 service in CM.

Expert Contributor
Posts: 277
Registered: ‎01-25-2017

Re: Having Spark 1.6.0 and 2.1 in the same CDH

@Hitesh88 Did you restart CM and CMS?

New Contributor
Posts: 4
Registered: ‎10-18-2017

Re: Having Spark 1.6.0 and 2.1 in the same CDH

Yes I restart both CM server and cloudera management service both and still it is not showing Spark2 as service in cluster.

 

I see JDK 1.7 used with CDH 5.9 on cluster I'm on. I read somewhere that Spark 2 requires JDK 1.8. Could that be stopping to get spark2 service.

 

Regards,

Hitesh

Expert Contributor
Posts: 277
Registered: ‎01-25-2017

Re: Having Spark 1.6.0 and 2.1 in the same CDH

Yes, Spark2 need JDK8.
New Contributor
Posts: 4
Registered: ‎10-18-2017

Re: Having Spark 1.6.0 and 2.1 in the same CDH

Okay, thanks for the update. Do you think there is anything else which could be causing this issue and not displaying Spark2. 

 

Regards,

Hitesh

New Contributor
Posts: 4
Registered: ‎10-18-2017

Re: Having Spark 1.6.0 and 2.1 in the same CDH

Also, is there way to confirm csd file is properly deployed. Also, I don't see scala 11 libraries under /opt/cloudera/parcels/CDH/jars and only scala 10 libraries. 

 

I heard that scala 10 and 11 both are installed with CDH 5.7 and later. Shouldn't scala 11 be available, Is this also cause for spark2 service not appearing.

 

I did all steps as mentioned and all steps did completely successfully, spark2 parcel is activated now.

 

Regards,

Hitesh

Announcements