Created local repository for CDH 6.2.0 but unable to get Spark-2 now

Question

Hi Team,

@jsensharma

@MattWho

@Shelton

@venkatsambath

I have installed CDH 6.2.0 on CM 6.2.0 with the help of local repository parcel file created by me following the Cloudera documents ( https://docs.cloudera.com/documentation/enterprise/6/6.3/topics/cm_ig_create_local_parcel_repo.html ). I have successfully installed the CDH. however, CDH 6.2 comes with in-built parcels for Spark-2. which I am unable to find in the parcels.

Please assist me on how do I add Spark-2 as a service in my Cloudera 6.2.0 now.

Appreciate your assistance.

venkatsambath · Accepted Answer

Hi Ansar,

For CDH5.x version the spark service that is bundled with the CDH parcel was spark 1.6 version hence there was a need to add Spark2 as a separate parcel.

You need not add spark2 separately for CDH6.2. When you just add spark service in CDH6.2, it will install spark2.4.0 -- For your reference, This link contains the individual service packaged inside cdh6.2.x https://docs.cloudera.com/documentation/enterprise/6/release-notes/topics/rg_cdh_62_packaging.html#cdh_621_packaging

If you would like to validate the version of spark installed alternatively you can run spark-shell --> This will show the spark version like this

Spark session available as 'spark'.
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 2.4.0-cdh6.3.3
      /_/

Using Scala version 2.11.12 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_141)
Type in expressions to have them evaluated.

Sign up

Please use your Cloudera account.

Login to the community

Please use your Cloudera account.