Reply
Contributor
Posts: 59
Registered: ‎10-02-2017
Accepted Solution

Download Client Configuration link for Spark_On_Yarn not available

[ Edited ]

We are trying to configure a host to allow client connections to various services on our cluster.  For Hive, HDFS and YARN we use the provided "Download Client Configuration" link exposed in the actions dropdown menu for each service.  This link isn't active in the actions menu for Spark_On_Yarn, however there is a "Deploy Client Configuration" link available. 

Posts: 866
Topics: 1
Kudos: 200
Solutions: 107
Registered: ‎04-22-2014

Re: Download Client Configuration link for Spark_On_Yarn not available

@dturner,

 

Perhaps you don't have Spark_On_Yarn Gateway roles on any hosts?  Usually other roles depend on it, but I can't think of any other reasons the Deploy Client Configuration link wouldn't be available.

Contributor
Posts: 59
Registered: ‎10-02-2017

Re: Download Client Configuration link for Spark_On_Yarn not available

[ Edited ]

Hi. Thank you for the reply.

Just to clarify, the “Deploy” configuration link is available. It’s the “Download” link that I’m asking about. It is not available.

 

However - good point about the gateway role.  I notice that I have " 8 none" next to the Gateway hosts status.  

Posts: 866
Topics: 1
Kudos: 200
Solutions: 107
Registered: ‎04-22-2014

Re: Download Client Configuration link for Spark_On_Yarn not available

Ah, thanks for pointing that out... I did miss the "Download" part.

Indeed, that feature is not available for Spark since the client configuration requires script-based post-processing. The best way for you to get a working Spark client configuration, then, is to copy the spark configuration from a cluster host that has the Spark Gateway role on it.

 

 

Contributor
Posts: 59
Registered: ‎10-02-2017

Re: Download Client Configuration link for Spark_On_Yarn not available

[ Edited ]

Ah. I see.  In the Cloudera Director bootstrap config file for this deployment, I have the Spark_On_Yarn Gateway role enabled for all cluster hosts.  The hosts are associated with the role, however the role didn't start after deployment.  I tried starting the gateway role on one of the nodes using Cloudera Manager and receive the following error:

 

"Command Start is not currently available for execution"

 

All other Spark roles (and remaining cluster services / roles) are functioning normally.

 

We are running CDH 5.14.2 

 

New Contributor
Posts: 1
Registered: ‎05-21-2018

Re: Download Client Configuration link for Spark_On_Yarn not available

Download and Install Spark Binaries
Spark binaries are available from the Apache Spark download page. Adjust each command below to match the correct version number.

Get the download URL from the Spark download page, download it, and uncompress it.

For Spark 2.2.0 with Hadoop 2.7 or later, log on node-master as the hadoop user, and run:

cd /home/hadoop
wget https://d3kbcqa49mib13.cloudfront.net/spark-2.2.0-bin-hadoop2.7.tgz
tar -xvf spark-2.2.0-bin-hadoop2.7.tgz
mv spark-2.2.0-bin-hadoop2.7 spark
Add the Spark binaries directory to your PATH. Edit /home/hadoop/.profile and add the following line:

Highlighted
Contributor
Posts: 59
Registered: ‎10-02-2017

Re: Download Client Configuration link for Spark_On_Yarn not available

Thanks for your reply.  The following link leads me to believe that what I'm seeing is expected:

 

https://community.cloudera.com/t5/Advanced-Analytics-Apache-Spark/Spark-gateway-not-starting/m-p/490...

Announcements