How access to Spark Web UI ?

[#1]

Hi,

I have a self-contained application with Spark 1.4.1, scala 2.11 and sbt 0.13.14. I run my application by sbt run then sbt package to genere a file jar.

I don't know where my application is runing, how much number of servers and into who cluster. I know that from Spark UI I can access to the interface, where I can see all my servers and I can change the sizes of a memory of the executor and driver. But the problem I don't know how I can access to a Spark web UI ?

I tried by 127.0.0.1:4040 but the page is inaccessible, when runing of my application it display that my driver in this address 10.0.2.15, I tried also by 10.0.2.15:4040 but in vein. Knowing that I'm using the Apache Ambari where I can access to all the clusters of hadoop ( I use Hadoop because my data is stored into HDFS).

Can you please tell me how I can access to the Saprk UI.

Thank you.

[#2]

@Alicia Alicia

sbt package is to generate application jar then you need to submit this jar on spark cluster suggesting what master to use in local,yarn-client, yarn-cluster or standalone.

have you submitted the application with spark-submit specifying --master parameter?

[#3]

yes I submit my application by spark submit: spark-submit --class "simpleApp" --master local[4] target/scala-2.11/my-project_2.11-1.0.jar

[#4]

@Alicia Alicia did you see any port bind exception in driver logs, spark ui try to bind 4040 and if it fail then try for 4041,4042 to 44. how long your application run there might be a chance your application exit quickly after submission.

could you please post driver logs?

[#5]

@Rajkumar Singh this following my small code:

/* My first application in Spark scala has an idea to build a system of recommandation for the users */ 
import java.io.File
import org.apache.spark.SparkContext 
import org.apache.spark.SparkContext._
import org.apache.spark.SparkConf 
import org.apache.spark.rdd._ 
import org.apache.spark.mllib.recommendation.ALS
import org.apache.spark.mllib.recommendation.Rating 
import org.apache.spark.mllib.recommendation.MatrixFactorizationModel 
object appFilms { 
val conf = new SparkConf().setAppName("system of recommandation").setMaster("local[4]") 
val sc = new SparkContext(conf) 
def main(args: Array[String]) { 
 // Load file rating and parse it
val Ratingfiles = sc.textFile("hdfs://sandbox.hortonworks.com:8020/tmp/ProjetFilm/ratings.dat").map {line =>
val splitRating =  line.split("::").toString
Rating(splitRating(0).toInt, splitRating(1).toInt, splitRating(2).toDouble)
}
//load file rating and parse it 
val Moviefile = sc.textFile("hdfs://sandbox.hortonworks.com:8020/tmp/ProjetFilm/movies.dat").map { line =>
val splitMovie = line.split("::").toString
(splitMovie(0).toInt, splitMovie(1).toString)
} 
//Split the Rating files into three parties 
val training_RDD = Ratingfiles.filter(x => x.rating < 6).cache() 
val validation_RDD = Ratingfiles.filter(x => x.rating >= 6 && x.rating < 8).cache() 
val test_RDD = Ratingfiles.filter(x => x.rating > 8).cache() 
// the model of training 
val rank = 10 
val numIteration = 4
val model = ALS.train(test_RDD, rank, numIteration, 0.01) 
//val result = model.save(sc, "hdfs://sandbox.hortonworks.com:8020/tmp/ProjetFilm/Model")
println("helooooooooooo")
sc.stop()
}
}

[#6]

@Alicia Alicia

I dont think that this application will run for a long, could you please introduce Thread sleep method in main method like this (jar build required in this case)

def main(args: Array[String]) {
  Thread.sleep(30000);
  // Load file rating and parse itval Ratingfiles = sc.textFile("hdfs://sandbox.hortonworks.com:8020/tmp/ProjetFilm/ratings.dat")

this will allow your app to remain there for atleast 30 secs and you can quickly grab the driver url from the driver logs.

[#7]

@Alicia Alicia are you able to access the driver ui after suggested change?

[#8]

@Alicia Alicia well I could not see any problem with the code it execute succesfully.

did you see any event in driver logs like

SparkUI: Bound SparkUI to 0.0.0.0, and started at http://172.26.81.127:4041

[#9]

@Alicia Alicia one more thing to add here, if you want to see the application logs for the completed application you can access the following url

http://<hostname_spark_history_server>:18080/

[#10]

@Alicia Alicia

hope this help you,if yes then please accept best answer so that other can refer it.