Failed on: Restarting History Server on zld00735

[#1]

I am upgrading HDP 2.3.4 to 2.5.3.40.. but upgrade failed with "Failed on: Restarting History Server on zld00735"

Std err log message

File "/usr/lib/python2.6/site-packages/resource_management/libraries/providers/hdfs_resource.py", line 192, in run_command
    raise Fail(err_msg)
resource_management.core.exceptions.Fail: Execution of 'curl -sS -L -w '%{http_code}' -X PUT --data-binary @/usr/hdp/2.5.3.40-1/hadoop/mapreduce.tar.gz 'http://zld00733.vci.att.com:50070/webhdfs/v1/hdp/apps/2.5.3.40-1/mapreduce/mapreduce.tar.gz?op=CREATE&user.name=hdfs&overwrite=True&permission=444'' returned status_code=500.

Below one is Name node log

2017-05-08 14:42:24,970 ERROR mortbay.log (Slf4jLog.java:warn(87)) - /webhdfs/v1/hdp/apps/2.5.3.40-1/mapreduce/mapreduce.tar.gz
java.lang.IllegalStateException: Form too large206534214>200000
        at org.mortbay.jetty.Request.extractParameters(Request.java:1561)
        at org.mortbay.jetty.Request.getParameterMap(Request.java:870)
        at org.apache.hadoop.http.HttpServer2$QuotingInputFilter$RequestQuoter.getParameterMap(HttpServer2.java:1227)
        at org.apache.hadoop.hdfs.web.AuthFilter.toLowerCase(AuthFilter.java:95)
        at org.apache.hadoop.hdfs.web.AuthFilter.doFilter(AuthFilter.java:82)
        at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
        at org.apache.hadoop.http.HttpServer2$QuotingInputFilter.doFilter(HttpServer2.java:1294)
        at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
        at org.apache.hadoop.http.NoCacheFilter.doFilter(NoCacheFilter.java:45)
        at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
        at org.apache.hadoop.http.NoCacheFilter.doFilter(NoCacheFilter.java:45)
        at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
        at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
        at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
        at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
        at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
        at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
        at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
        at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
        at org.mortbay.jetty.Server.handle(Server.java:326)
        at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
        at org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.java:945)
        at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:756)
        at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
        at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
        at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
        at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)

[#2]

@sivareddy akkala

Looks like you are using ambari 2.4 or prior version.

From ambari 2.5 this bug has been fixed: https://issues.apache.org/jira/browse/AMBARI-18536

.

Explanation Of this error:

That particular curl command is trying to send the request as if it was an HTML form post, using "Content-Type: application/x-www-form-urlencoded". That causes the jetty server to try to slurp the whole request into memory and try to parse the key-value pairs of an HTML form out of it. So as part of that fix the command is slightly changed as [1] so that it will not treat the request like an HTML form post and instead uses "Content-Type: application/octet-stream".

.

[1] "Code Change Detail"

.

[#3 · Accepted solution]

It Works! Thank you so much Jay.

[#4]

@sivareddy akkala

Good to know that it was useful.

It will be great if you can click on the "Accept" link on this thread and mark this thread as "Answered", that will be useful for other users as well.

[#5]

@sivareddy akkala What i meant in my previous comment is that ... you were supposed to click the "Accept" button for the update that resolved the issue (or answered your query). Here you accepted your own update which does not have the answer.