question Re: Configure Spark2 with Ambari in Archives of Support Questions (Read Only)

Configure Spark2 with Ambari

chingiz3101 — Thu, 07 Sep 2017 11:18:13 GMT

Hi everyone.

We have HDP cluster with several Spark2 clients. I want to perform a remote connection, the manual says that I need to set the SPARK_MASTER_HOST

Is there a way to make such configuration with Ambari UI or I have to edit spark-env.sh? If second, which one I have to choose (namenode, datanode etc)?

Re: Configure Spark2 with Ambari

jaimin — Sun, 18 Aug 2019 09:44:41 GMT

@Chingiz Akmatov

spark-env.sh is ambari managed file and expose on UI. So you can make changes to it from Amabri UI.

To do so Go to Spark2->Configs->Advanced spark2-env section->content property and add the env variable

Re: Configure Spark2 with Ambari

chingiz3101 — Fri, 08 Sep 2017 10:42:27 GMT

@jaimin

I set SPARK_MASTER_HOST from the UI as you said, spark-env.sh updated with the line I put, but there is no listening port 7077. What am I missing? And I forgot to mention that we have zookeeper and spark along with spark2 if it makes a sense

Re: Configure Spark2 with Ambari

chingiz3101 — Mon, 25 Sep 2017 14:11:24 GMT

My mistake, forgot to run

start-master.sh

after configuring spark-env.sh

Re: Configure Spark2 with Ambari

engineeromerhay — Tue, 30 Jul 2019 07:31:44 GMT

Hi,

I can not edit my spark config finals, am i missing something ?