question Re: Failed on: Restarting History Server on zld00735 in Support Questions

Failed on: Restarting History Server on zld00735

siva26akkala — Fri, 16 Sep 2022 11:34:32 GMT

I am upgrading HDP 2.3.4 to 2.5.3.40.. but upgrade failed with "Failed on: Restarting History Server on zld00735"

Std err log message

File "/usr/lib/python2.6/site-packages/resource_management/libraries/providers/hdfs_resource.py", line 192, in run_command
    raise Fail(err_msg)
resource_management.core.exceptions.Fail: Execution of 'curl -sS -L -w '%{http_code}' -X PUT --data-binary @/usr/hdp/2.5.3.40-1/hadoop/mapreduce.tar.gz 'http://zld00733.vci.att.com:50070/webhdfs/v1/hdp/apps/2.5.3.40-1/mapreduce/mapreduce.tar.gz?op=CREATE&user.name=hdfs&overwrite=True&permission=444'' returned status_code=500.

Below one is Name node log

2017-05-08 14:42:24,970 ERROR mortbay.log (Slf4jLog.java:warn(87)) - /webhdfs/v1/hdp/apps/2.5.3.40-1/mapreduce/mapreduce.tar.gz
java.lang.IllegalStateException: Form too large206534214>200000
        at org.mortbay.jetty.Request.extractParameters(Request.java:1561)
        at org.mortbay.jetty.Request.getParameterMap(Request.java:870)
        at org.apache.hadoop.http.HttpServer2$QuotingInputFilter$RequestQuoter.getParameterMap(HttpServer2.java:1227)
        at org.apache.hadoop.hdfs.web.AuthFilter.toLowerCase(AuthFilter.java:95)
        at org.apache.hadoop.hdfs.web.AuthFilter.doFilter(AuthFilter.java:82)
        at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
        at org.apache.hadoop.http.HttpServer2$QuotingInputFilter.doFilter(HttpServer2.java:1294)
        at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
        at org.apache.hadoop.http.NoCacheFilter.doFilter(NoCacheFilter.java:45)
        at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
        at org.apache.hadoop.http.NoCacheFilter.doFilter(NoCacheFilter.java:45)
        at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
        at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
        at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
        at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
        at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
        at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
        at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
        at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
        at org.mortbay.jetty.Server.handle(Server.java:326)
        at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
        at org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.java:945)
        at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:756)
        at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
        at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
        at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
        at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)

Re: Failed on: Restarting History Server on zld00735

jsensharma — Tue, 09 May 2017 11:54:35 GMT

@sivareddy akkala

Looks like you are using ambari 2.4 or prior version.

From ambari 2.5 this bug has been fixed: https://issues.apache.org/jira/browse/AMBARI-18536

Explanation Of this error:

That particular curl command is trying to send the request as if it was an HTML form post, using "Content-Type: application/x-www-form-urlencoded". That causes the jetty server to try to slurp the whole request into memory and try to parse the key-value pairs of an HTML form out of it. So as part of that fix the command is slightly changed as [1] so that it will not treat the request like an HTML form post and instead uses "Content-Type: application/octet-stream".

[1] "Code Change Detail"

Re: Failed on: Restarting History Server on zld00735

siva26akkala — Tue, 09 May 2017 23:31:02 GMT

It Works! Thank you so much Jay.

Re: Failed on: Restarting History Server on zld00735

jsensharma — Tue, 09 May 2017 23:39:56 GMT

@sivareddy akkala

Good to know that it was useful.

It will be great if you can click on the "Accept" link on this thread and mark this thread as "Answered", that will be useful for other users as well.

Re: Failed on: Restarting History Server on zld00735

jsensharma — Wed, 10 May 2017 00:14:15 GMT

@sivareddy akkala What i meant in my previous comment is that ... you were supposed to click the "Accept" button for the update that resolved the issue (or answered your query). Here you accepted your own update which does not have the answer.