Created local repository for CDH 6.2.0 but unable to get Spark-2 now

[#1]

Hi Team, 

@jsensharma 

@MattWho 

@Shelton 

@venkatsambath 

I have installed CDH 6.2.0 on CM 6.2.0 with the help of local repository parcel file created by me following the Cloudera documents ( https://docs.cloudera.com/documentation/enterprise/6/6.3/topics/cm_ig_create_local_parcel_repo.html ). I have successfully installed the CDH. however, CDH 6.2 comes with in-built parcels for Spark-2. which I am unable to find in the parcels. 

Please assist me on how do I add Spark-2 as a service in my Cloudera 6.2.0 now.

Appreciate your assistance. 

[#2 · Accepted solution]

Hi Ansar,

For CDH5.x version the spark service that is bundled with the CDH parcel was spark 1.6 version hence there was a need to add Spark2 as a separate parcel.

You need not add spark2 separately for CDH6.2. When you just add spark service in CDH6.2, it will install spark2.4.0 -- For your reference, This link contains the individual service packaged inside cdh6.2.x https://docs.cloudera.com/documentation/enterprise/6/release-notes/topics/rg_cdh_62_packaging.html#c... 

If you would like to validate the version of spark installed alternatively you can run spark-shell --> This will show the spark version like this

Spark session available as 'spark'.
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 2.4.0-cdh6.3.3
      /_/

Using Scala version 2.11.12 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_141)
Type in expressions to have them evaluated.

[#3]

Thanks @venkatsambath 

That helped. 

Regards, 
Ansar