What happens inside the Spark Component Adding through Ambari ?

[#1]

Hi there,

I am trying to install Spark on 16 node HDP cluster. When I select spark component in the Ambari (Add components page), what happens. Does it install spark library on all the nodes and install Spark server on the master node?

Previously I ran Spark on local mode but want to know how does the installation work under the hood?

Regards.

[#2 · Accepted solution]

@Smart Solutions you will have to select where you want Spark History Server to run and which machines to install Spark clients on. There is a page further down the road that will allow you to pick which nodes to select what on. For example on Sandbox, if I click either URL below, will take me to sandbox.hortonworks.com, on your cluster, URLs will point to whatever servers you defined.

[#3]

Thanks does it mean we install just Spark clients and nothing related to master processes or server on HDP? While configuring through Ambari?

[#4]

no, you install master services as well. Spark History for example is a service. I don't have access to an installation right now otherwise I'd shown you install steps. Basically components are categorized by clients and master services. You check and uncheck which machines will serve as what. @Smart Solutions

[#5]

@Smart Solutions

Take a look on this

http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.4/bk_installing_manually_book/content/ch_insta...

You can pick nodes where you want your clients.

When you install Spark, the following directories will be created:

• /usr/hdp/current/spark-client for submitting Spark jobs
• /usr/hdp/current/spark-history for launching Spark master processes, such as the Spark History Server
• /usr/hdp/current/spark-thriftserver for the Spark Thrift Server

[#6]

@Smart Solutions See this

You need master services and clients.

You can see that I have spark client in 3 nodes

when I clicked spark clients and then clicks one of the nodes then you can see all the clients installed in that node

See : Clients

[#7]

@Smart Solutions

When you add Spark through Ambari, you will be asked to choose where to deploy master service (Spark History Service)

And then to choose where to deploy clients services

Finally you will be asked for several properties

screen-shot-2016-03-03-at-61725-pm.png