Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Configure Spark2 with Ambari

Solved Go to solution
Highlighted

Configure Spark2 with Ambari

New Contributor

Hi everyone.

We have HDP cluster with several Spark2 clients. I want to perform a remote connection, the manual says that I need to set the SPARK_MASTER_HOST

Is there a way to make such configuration with Ambari UI or I have to edit spark-env.sh? If second, which one I have to choose (namenode, datanode etc)?

1 ACCEPTED SOLUTION

Accepted Solutions

Re: Configure Spark2 with Ambari

Expert Contributor
@Chingiz Akmatov

spark-env.sh is ambari managed file and expose on UI. So you can make changes to it from Amabri UI.

To do so Go to Spark2->Configs->Advanced spark2-env section->content property and add the env variable

34709-spark-env.png

4 REPLIES 4

Re: Configure Spark2 with Ambari

Expert Contributor
@Chingiz Akmatov

spark-env.sh is ambari managed file and expose on UI. So you can make changes to it from Amabri UI.

To do so Go to Spark2->Configs->Advanced spark2-env section->content property and add the env variable

34709-spark-env.png

Re: Configure Spark2 with Ambari

New Contributor

@jaimin

I set SPARK_MASTER_HOST from the UI as you said, spark-env.sh updated with the line I put, but there is no listening port 7077. What am I missing? And I forgot to mention that we have zookeeper and spark along with spark2 if it makes a sense

Re: Configure Spark2 with Ambari

New Contributor

My mistake, forgot to run

start-master.sh

after configuring spark-env.sh

Re: Configure Spark2 with Ambari

New Contributor

Hi,

I can not edit my spark config finals, am i missing something ?

Don't have an account?
Coming from Hortonworks? Activate your account here