question Spark Master UI in Archives of Support Questions (Read Only)

Spark Master UI

aboomarrifai — Wed, 01 Jun 2016 22:46:02 GMT

I have Sandbox 2.4 on a VirtualBox

when I try to access Spark Master UI by the browser from my windows machine I get access denied as you can see from the pic I attached

can you tell my what wrong with it.?access-denide.png

Re: Spark Master UI

kkhan — Wed, 01 Jun 2016 22:47:22 GMT

Are you able to ping sandbox.hortonworks.com from your windows machine? If not, try using IP address instead of hostname.

Re: Spark Master UI

jyadav — Wed, 01 Jun 2016 22:52:05 GMT

@omar harb

I think HDP comes with Spark on Yarn not standalone unless you installed it manually therefore you won't find the spark master UI inside sandbox.

Please start with below doc.

http://hortonworks.com/hadoop-tutorial/a-lap-around-apache-spark/

Re: Spark Master UI

aboomarrifai — Wed, 01 Jun 2016 22:59:37 GMT

@Jitendra Yadav

so how can I see INFO . about the workers?

Re: Spark Master UI

jyadav — Thu, 02 Jun 2016 00:30:28 GMT

@omar harb

There is no separate workers when spark runs on YARN, it will run normal as a normal app within YARN in the form of YARN containers. please refer below doc for spark on yarn architecture.

https://spark-summit.org/2014/wp-content/uploads/2014/07/Spark-on-YARN-A-Deep-Dive-Sandy-Ryza.pdf

http://spark.apache.org/docs/latest/running-on-yarn.html

Re: Spark Master UI

jyadav — Thu, 02 Jun 2016 20:48:12 GMT

Hi @omar harb Please let me know if you need more info on this or kindly accept the answer to close this thread.

Re: Spark Master UI

aboomarrifai — Thu, 02 Jun 2016 20:57:24 GMT

@Jitendra Yadav

I just need to know why when

I opened the URL

http://localhost:8088/cluster/app/application_1464857740650_0009

and clicked on the log link to see the log, I got

Access Denied.

please see the attached photo clusterurl.png logurlpng.png

thank you.

Re: Spark Master UI

jyadav — Thu, 02 Jun 2016 21:26:06 GMT

@omar harb

If RM UI showing enough available resources then I would suggest you to stop the spark application and run it again with below options.

spark-submit --class com.Spark.MainClass --master yarn-client  /home/Test-0.0.1-SNAPSHOT.jar --executor-cores 2 --executor-memory 2g --num-executors 2

If you still see same issue then kindly share the Spark AM logs from this screen for that job. clusterurl.png