Created 12-24-2016 02:15 PM
Hi,
I have a self-contained application with Spark 1.4.1, scala 2.11 and sbt 0.13.14. I run my application by sbt run then sbt package to genere a file jar.
I don't know where my application is runing, how much number of servers and into who cluster. I know that from Spark UI I can access to the interface, where I can see all my servers and I can change the sizes of a memory of the executor and driver. But the problem I don't know how I can access to a Spark web UI ?
I tried by 127.0.0.1:4040 but the page is inaccessible, when runing of my application it display that my driver in this address 10.0.2.15, I tried also by 10.0.2.15:4040 but in vein. Knowing that I'm using the Apache Ambari where I can access to all the clusters of hadoop ( I use Hadoop because my data is stored into HDFS).
Can you please tell me how I can access to the Saprk UI.
Thank you.
Created 12-24-2016 09:14 PM
if I understand correctly, you run Spark locally. Could you check what is running on your local port 4040?
I assume that you use a Linux-like kernel, including mac OS:
lsof -i :4040 | grep LISTEN
Write down the PID and use the following command to learn more
ps -ef WhatEverPID
If you have something else running on the port, then that's your issue. You have to stop and start Spark to use a different port for Web UI. Please make sure that is available.
It is unlikely that you have a firewall blocking port 4040, but that is another potential cause.
Created 12-24-2016 10:12 PM
Thak you for your answer. Yes I checked on availability of the port 4040 is reserved:
the result of checking is the following:
Active connexion local address Remote address state
TCP 127.0.0.1:4040 DESKTOP-FF4U.....:0 Listening
Created 12-26-2016 02:51 AM
What is on your 8080 port?
Per https://spark.apache.org/docs/1.4.1/spark-standalone.html, spark master web ui is 8080
Created on 12-29-2016 11:06 AM - edited 08-18-2019 03:39 AM
Thank you for your answer, I'm sorry for this deplay in response.
In fact, the port 8080 is reserved for Ambari, http://localhost:8080, this is a print-screen:
Created 01-03-2017 09:41 PM
Alicia, please see my answer above on Oct 24. If you are running Spark on YARN you will have to go through the YARN RM UI to get to the Spark UI for a running job. Link for YARN UI is available from Ambari YARN service. For a completed job, you will need to go through Spark History Server. Link for Spark history server is available from the Ambari Spark service.