Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

map spark.driver.appUIAddress IP to different IP

map spark.driver.appUIAddress IP to different IP

Rising Star

1105-spark-appui-config.png

Hi,

I have HDP2.3.2 cluster installed in Amazon EC2.

I want to update the IP adress of spark.driver.appUIAddress,which is currently mapped to private IP of EC2.

Searched in spark config in ambari,could find spark.driver.appUIAddress property.

Because of this private IP mapping I am unable to connect to Spark web UI.

Would really appreciate the help.

Attached the screen shot for reference

Thanks,

Divya

4 REPLIES 4

Re: map spark.driver.appUIAddress IP to different IP

I think you can leave that address unchanged. Instead you can open the port for external access and access it using your instance public address. To open ports, click on your security group, then select "Inbound" tab, edit and add a new "Custom TCP rule" selecting your port (4040) and optionally source IP address ("anywhere", or selecting only the IP of your machine you are going to use to access the cluster). Finally, try to access, for example http://ec2-<your-ip-adr>.ap-southeast-1.compute.amazonaws.com:4040.

Highlighted

Re: map spark.driver.appUIAddress IP to different IP

Rising Star

Hi,

I configured below properties

SPARK_LOCAL_IPIP address of the machine to bind to.SPARK_PUBLIC_DNSHostname your Spark program will advertise to other machines.Now In spark-shell console log
15/12/29 02:11:18 INFO Utils: Successfully started service 'SparkUI' on port 4040.
15/12/29 02:11:18 INFO SparkUI: Started SparkUI at http://public-IP:4040
Now when I go this address http://public-IP:4040 Its redirecting to http://ip-private-ip.ap-southeast-1.compute.internal:8088/proxy/application_1451351202944_0007/ which is mapped to
spark.org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter.param.PROXY_URI_BASES

in Spark shell environment variable spark.driver.appUIAddress http://private-ip:4040

Am I mising any configuration ?

Re: map spark.driver.appUIAddress IP to different IP

Hi @Divya Gehlot My suggestion assumes your cluster and Spark operate on local ip addresses, and you just open ports you want to access remotely. In that case there is no need to set SPARK_LOCAL_IP or SPARK_PUBLIC_DNS (unless Spark has some internal issues). In the same way you can open ports to access Ambari (8080), RM UI (8080), etc. Another approach is to install VNC, login and access all Web UIs using local IPs.

Re: map spark.driver.appUIAddress IP to different IP

Mentor

@Divya Gehlot can you accept the best answer to close this thread?