Reply
New Contributor
Posts: 3
Registered: ‎02-17-2017

Spark2 service is not on the 'Add Service' list on cdh5.9.1

using this document to install spark2 

https://www.cloudera.com/documentation/spark2/latest/topics/spark2_installing.html

 

install csd

-------------

 

ls -ltr /opt/cloudera/csd/
total 16
-rw-r--r-- 1 cloudera-scm cloudera-scm 15765 Feb 17 11:56 SPARK2_ON_YARN-2.0.0.cloudera1.jar

 

service cloudera-scm-agent restart

 

I see in the /var/log/cloudera-scm-server/cloudera-scm-server.log

 

2017-02-17 12:03:57,728 INFO main:org.mortbay.log: Registered SubjectType SPARK2_ON_YARN
2017-02-17 12:03:57,728 INFO main:com.cloudera.cmon.TimeSeriesEntityType: Registered TimeSeriesEntityType SPARK2_ON_YARN
2017-02-17 12:03:57,728 INFO main:org.mortbay.log: Registered SubjectType SPARK2_ON_YARN-SPARK2_YARN_HISTORY_SERVER
2017-02-17 12:03:57,728 INFO main:com.cloudera.cmon.TimeSeriesEntityType: Registered TimeSeriesEntityType SPARK2_ON_YARN-SPARK2_YARN_HISTORY_SERVER
2017-02-17 12:03:57,729 INFO main:com.cloudera.cmon.TimeSeriesEntityType: Registered TimeSeriesEntityType IMPALA_DAEMON_POOL
2017-02-17 12:03:57,729 INFO main:com.cloudera.cmon.TimeSeriesEntityType: Registered TimeSeriesEntityType KAFKA_TOPIC
2017-02-17 12:03:57,729 INFO main:com.cloudera.cmon.TimeSeriesEntityType: Registered TimeSeriesEntityType KAFKA_BROKER_TOPIC
2017-02-17 12:03:57,729 INFO main:com.cloudera.cmon.TimeSeriesEntityType: Registered TimeSeriesEntityType KAFKA_REPLICA
2017-02-17 12:03:57,935 INFO main:com.cloudera.enterprise.Translator: Translations for bundle [SPARK2_ON_YARN] and locale [de] are missing.
2017-02-17 12:03:57,935 INFO main:com.cloudera.enterprise.Translator: Translations for bundle [SPARK2_ON_YARN] and locale [es] are missing.
2017-02-17 12:03:57,935 INFO main:com.cloudera.enterprise.Translator: Translations for bundle [SPARK2_ON_YARN] and locale [fr] are missing.
2017-02-17 12:03:57,935 INFO main:com.cloudera.enterprise.Translator: Translations for bundle [SPARK2_ON_YARN] and locale [pt_PT] are missing.
2017-02-17 12:03:57,935 INFO main:com.cloudera.enterprise.Translator: Translations for bundle [SPARK2_ON_YARN] and locale [pt_BR] are missing.
2017-02-17 12:03:57,935 INFO main:com.cloudera.enterprise.Translator: Translations for bundle [SPARK2_ON_YARN] and locale [ja] are missing.
2017-02-17 12:03:57,935 INFO main:com.cloudera.enterprise.Translator: Translations for bundle [SPARK2_ON_YARN] and locale [ko] are missing.
2017-02-17 12:03:57,936 INFO main:com.cloudera.enterprise.Translator: Translations for bundle [SPARK2_ON_YARN] and locale [zh_CN] are missing.
2017-02-17 12:03:57,999 INFO main:com.cloudera.csd.components.CsdRegistryImpl: Installed 1 handlers for CSD [SPARK2_ON_YARN-2.0.0.cloudera1]
2017-02-17 12:03:57,999 INFO main:com.cloudera.server.cmf.descriptor.components.DescriptorFactory: CSD SPARK2_ON_YARN-2.0.0.cloudera1 installed, refreshing descriptors
2017-02-17 12:03:57,999 INFO main:com.cloudera.server.cmf.descriptor.components.DescriptorFactory: Bumping configuration defaults generation.
2017-02-17 12:03:58,019 INFO main:com.cloudera.parcel.components.ParcelManagerImpl: Adding remote parcel repository : http://archive.cloudera.com/spark2/parcels/latest/

install parcel

------------------

 

ls -l /opt/cloudera/parcel-repo/

-rw-r----- 1 cloudera-scm cloudera-scm 173901094 Feb 17 13:02 SPARK2-2.0.0.cloudera1-1.cdh5.7.0.p0.113931-el7.parcel
-rw-r----- 1 cloudera-scm cloudera-scm 41 Feb 17 13:02 SPARK2-2.0.0.cloudera1-1.cdh5.7.0.p0.113931-el7.parcel.sha
-rw-r----- 1 cloudera-scm cloudera-scm 6820 Feb 17 13:02 SPARK2-2.0.0.cloudera1-1.cdh5.7.0.p0.113931-el7.parcel.torrent

It is downloaded, distributed, and activated.

 

Cluster > 'Add service' does not show 'spark2'

 

As well, I do not see the package in  /opt/cloudera/parcel-repo/manifest.json

 

Please advise.

Thanks

 

 

Cloudera Employee
Posts: 89
Registered: ‎05-10-2016

Re: Spark2 service is not on the 'Add Service' list on cdh5.9.1

You have restarted the cloudera manager agent, but did you also restart cloudera manager server and did you confirm CSD was deployed to the same cloudera manager server you are attempting to connect?

New Contributor
Posts: 3
Registered: ‎02-17-2017

Re: Spark2 service is not on the 'Add Service' list on cdh5.9.1

[ Edited ]

I have a  2 node cluster

 

node 1 runs server and agent

node 2 runs agent only

 

node 1

ls -l /opt/cloudera/csd/
total 16
-rw-r--r-- 1 cloudera-scm cloudera-scm 15765 Feb 17 11:56 SPARK2_ON_YARN-2.0.0.cloudera1.jar

 

 ls -l /opt/cloudera/parcel-repo

-rw-r----- 1 cloudera-scm cloudera-scm 173901094 Feb 17 13:02 SPARK2-2.0.0.cloudera1-1.cdh5.7.0.p0.113931-el7.parcel
-rw-r----- 1 cloudera-scm cloudera-scm 41 Feb 17 13:02 SPARK2-2.0.0.cloudera1-1.cdh5.7.0.p0.113931-el7.parcel.sha
-rw-r----- 1 cloudera-scm cloudera-scm 6820 Feb 17 13:02 SPARK2-2.0.0.cloudera1-1.cdh5.7.0.p0.113931-el7.parcel.torrent

node 2

ls -l /opt/cloudera/parcels/SPARK2-2.0.0.cloudera1-1.cdh5.7.0.p0.113931/
total 0
drwxr-xr-x 2 root root 63 Dec 6 13:54 bin
drwxr-xr-x 3 root root 20 Dec 6 13:54 etc
drwxr-xr-x 3 root root 20 Dec 6 13:54 lib
drwxrwxr-x 2 root root 71 Dec 6 13:54 meta

 

I have restarted server

systemctl restart cloudera-scm-server.service

 

I have restart on both nodes1

 systemctl restart cloudera-scm-agent.service

 

All the logs show that the parcels is active and distributed.

diagnostics > logs

 

Registered SubjectType SPARK2_ON_YARN
Registered TimeSeriesEntityType SPARK2_ON_YARN
Registered SubjectType SPARK2_ON_YARN-SPARK2_YARN_HISTORY_SERVER
Registered TimeSeriesEntityType SPARK2_ON_YARN-SPARK2_YARN_HISTORY_SERVER

 

hosts > parcels

Parcel Name Version Status Actions

SPARK2 2.0.0.cloudera1-1.cdh5.7.0.p0.113931
Distributed, Activated

 

Not sure what else to look for.

New Contributor
Posts: 1
Registered: ‎02-28-2017

Re: Spark2 service is not on the 'Add Service' list on cdh5.9.1

[ Edited ]
New Contributor
Posts: 3
Registered: ‎02-17-2017

Re: Spark2 service is not on the 'Add Service' list on cdh5.9.1

Never figured it out.

 

I rebuilt the 3 node cluster with cm. This time it worked fine as per the instructions.

Highlighted
New Contributor
Posts: 2
Registered: ‎05-17-2017

Re: Spark2 service is not on the 'Add Service' list on cdh5.9.1

I have same situation.

Do you have any solution?

New Contributor
Posts: 1
Registered: ‎05-22-2017

Re: Spark2 service is not on the 'Add Service' list on cdh5.9.1

I had the same issue today.

Any suggestion?

 

Announcements