Reply
Highlighted
Contributor
Posts: 27
Registered: ‎11-05-2018

YARN ACL and Could not determine the proxy server for redirection exception for Spark UI.

Hi All,

 

 

After modifying yarn.admin.acl property from "*" to specific group, when I run spark2-shell from user which is not in this group and trying to access to Spark context Web UI I get error as below:

 

 

javax.servlet.ServletException: Could not determine the proxy server for redirection at org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter.findRedirectUrl(AmIpFilter.java:201) at org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter.doFilter(AmIpFilter.java:141) at org.spark_project.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676) at org.spark_project.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581) at org.spark_project.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180) at org.spark_project.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511) at org.spark_project.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112) at org.spark_project.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) at org.spark_project.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:461) at org.spark_project.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)

 

 

It works only when user is in yarn.admin.acl. I suspected that to see spark ui for own spark context it is not necessary to have yarn admin rights. Am' I right?

 

Thanks for any help.

 

 

Regards, Bart