question Re: YARN Exception: Could not determine the proxy server for redirection in Support Questions

YARN Exception: Could not determine the proxy server for redirection

alex_schaetzle — Mon, 09 Apr 2018 19:52:57 GMT

yarn-site.xml
We are running a cluster with HDP 2.6.3.0-235.
YARN is configured for HA with two Resourcemanagers.

When I run a spark-shell session and want to access the tracking URL of the Spark UI on the server where I started the spark-shell, I get the following error:

HTTP ERROR 500
Problem accessing /. Reason:
    Server Error
Caused by:
javax.servlet.ServletException: Could not determine the proxy server for redirection
	at org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter.findRedirectUrl(AmIpFilter.java:205)
	at org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter.doFilter(AmIpFilter.java:145)
	at org.spark_project.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)
	at org.spark_project.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581)
	at org.spark_project.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
	at org.spark_project.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511)
	at org.spark_project.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
	at org.spark_project.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
	at org.spark_project.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:461)
	at org.spark_project.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)
	at org.spark_project.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
	at org.spark_project.jetty.server.Server.handle(Server.java:524)
	at org.spark_project.jetty.server.HttpChannel.handle(HttpChannel.java:319)
	at org.spark_project.jetty.server.HttpConnection.onFillable(HttpConnection.java:253)
	at org.spark_project.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
	at org.spark_project.jetty.io.FillInterest.fillable(FillInterest.java:95)
	at org.spark_project.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
	at org.spark_project.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
	at org.spark_project.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
	at org.spark_project.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
	at org.spark_project.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
	at org.spark_project.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
	at java.lang.Thread.run(Thread.java:745)

However, the shell is working fine and I can also access the Spark UI following the link to the ApplicationMaster in the Resourcemanager UI.

We didn't have this problem before and I assume it started to exist after configuring YARN for HA (using Ambari wizzard).
Attached you can find the yarn-site.xml (exported with Ambari).

Is there any misconfiguration or does somebody know this problem?

Thx,
Alex

Re: YARN Exception: Could not determine the proxy server for redirection

ssathish — Tue, 10 Apr 2018 03:43:00 GMT

Hi @Alexander Schätzle,

We had a jira for the same issue - https://issues.apache.org/jira/browse/YARN-7269.

The fix went into HDP -2.6.3.0. But I am not sure about the exact build number. May be you can try with HDP-2.6.4.0 repo.

Re: YARN Exception: Could not determine the proxy server for redirection

alex_schaetzle — Tue, 10 Apr 2018 18:21:40 GMT

@ssathish

Ok, thanks for the hint, that seems to be the reason!

How can I find out which is the latest build number of 2.6.3.0? I checked the github repo (https://github.com/hortonworks/hadoop-release/releases) but the numbering is a bit confusing.

I can see that the latest release of 2.6.3 is 2.6.3.36 but which build number is this? And is it possible to upgrade the installed build of 2.6.3 (build 235 in our case) to the latest version of 2.6.3?

Re: YARN Exception: Could not determine the proxy server for redirection

alex_schaetzle — Tue, 10 Apr 2018 18:36:32 GMT

Another question. The release notes of HDP 2.6.3 lists the patch for YARN-7269 as included in HDP 2.6.3. Should this not mean that it is included in the "official" version in the public repo? The build version in the public repo is 235. So if the patch is not included in build number 235, is the information in the release notes than not very confusing or even incorrect?

Re: YARN Exception: Could not determine the proxy server for redirection

alex_schaetzle — Tue, 10 Apr 2018 19:21:56 GMT

@ssathish

I checked the source code of HDP 2.6.3.0-235 and I see that the code changes applied in the patch for https://issues.apache.org/jira/browse/YARN-7269. are already present.

Do you see any other reason why this error then still happens?

Re: YARN Exception: Could not determine the proxy server for redirection

alex_schaetzle — Fri, 20 Apr 2018 15:34:24 GMT

Is there nobody with the same issue?

Re: YARN Exception: Could not determine the proxy server for redirection

sushant_pandey — Wed, 23 May 2018 14:21:47 GMT

I see the same issue when I execute jupyter-console with pyspark kernel. Please see the attach.

error.txt

Re: YARN Exception: Could not determine the proxy server for redirection

alex_schaetzle — Wed, 23 May 2018 17:17:13 GMT

I was not able to fix this issue up to now. The only solution I found currently is to disable Resource Manager HA.

Re: YARN Exception: Could not determine the proxy server for redirection

areshJ — Mon, 01 Jun 2020 20:21:09 GMT

Hi Alex, were you able to get this fixed ? I am getting similar error for hive on tez

javax.servlet.ServletException: Could not determine the proxy server for redirection
	at org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter.findRedirectUrl(AmIpFilter.java:212)
	at org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter.doFilter(AmIpFilter.java:146)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
	at org.apache.hadoop.http.HttpServer2$QuotingInputFilter.doFilter(HttpServer2.java:1619)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
	at org.apache.hadoop.http.NoCacheFilter.doFilter(NoCacheFilter.java:45)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
	at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
	at org.eclipse.jetty.server.Server.handle(Server.java:539)
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:333)
	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108)
	at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
	at java.lang.Thread.run(Thread.java:748)

Re: YARN Exception: Could not determine the proxy server for redirection

cjervis — Mon, 01 Jun 2020 20:58:14 GMT

As this is an older post you would have a better chance of receiving a resolution by starting a new thread. This will also provide the opportunity to provide details specific to your environment that could aid others in providing a more accurate answer to your question.

Re: YARN Exception: Could not determine the proxy server for redirection

jomach — Fri, 05 Jun 2020 11:59:16 GMT

See https://github.com/apache/spark/pull/22164/files