I have HDP2.3.2 cluster installed in Amazon EC2.
I want to update the IP adress of spark.driver.appUIAddress,which is currently mapped to private IP of EC2.
Searched in spark config in ambari,could find spark.driver.appUIAddress property.
Because of this private IP mapping I am unable to connect to Spark web UI.
Would really appreciate the help.
Attached the screen shot for reference
I think you can leave that address unchanged. Instead you can open the port for external access and access it using your instance public address. To open ports, click on your security group, then select "Inbound" tab, edit and add a new "Custom TCP rule" selecting your port (4040) and optionally source IP address ("anywhere", or selecting only the IP of your machine you are going to use to access the cluster). Finally, try to access, for example http://ec2-<your-ip-adr>.ap-southeast-1.compute.amazonaws.com:4040.
I configured below properties
SPARK_LOCAL_IPIP address of the machine to bind to.
SPARK_PUBLIC_DNSHostname your Spark program will advertise to other machines.Now In spark-shell console log
15/12/29 02:11:18 INFO Utils: Successfully started service 'SparkUI' on port 4040. 15/12/29 02:11:18 INFO SparkUI: Started SparkUI at http://public-IP:4040Now when I go this address http://public-IP:4040 Its redirecting to http://ip-private-ip.ap-southeast-1.compute.internal:8088/proxy/application_1451351202944_0007/ which is mapped to
in Spark shell environment variable spark.driver.appUIAddress http://private-ip:4040
Am I mising any configuration ?
Hi @Divya Gehlot My suggestion assumes your cluster and Spark operate on local ip addresses, and you just open ports you want to access remotely. In that case there is no need to set SPARK_LOCAL_IP or SPARK_PUBLIC_DNS (unless Spark has some internal issues). In the same way you can open ports to access Ambari (8080), RM UI (8080), etc. Another approach is to install VNC, login and access all Web UIs using local IPs.