Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Spark Thrift job problem HTTP ERROR 500

Highlighted

Spark Thrift job problem HTTP ERROR 500

Hi All,

I running Spark 1 Thrift Server as Proxy.

But it not running as long as I expected.

Every two days, Spark will die likely cronjob. And sometime I can not access to Spark Thrift Server url.

I often get this ERROR.

<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=ISO-8859-1"/>
<title>Error 500 Server Error</title>
</head>
<body><h2>HTTP ERROR 500</h2>
<p>Problem accessing /jobs/. Reason:
<pre>    Server Error</pre></p><h3>Caused by:</h3><pre>java.lang.OutOfMemoryError: Java heap space
	at java.util.Arrays.copyOfRange(Arrays.java:3664)
	at java.lang.String.<init>(String.java:207)
	at java.lang.StringBuilder.toString(StringBuilder.java:407)
	at scala.collection.mutable.StringBuilder.toString(StringBuilder.scala:427)
	at scala.xml.Node.buildString(Node.scala:161)
	at scala.xml.Node.toString(Node.scala:166)
	at org.apache.spark.ui.JettyUtils$$anonfun$htmlResponderToServlet$1.apply(JettyUtils.scala:55)
	at org.apache.spark.ui.JettyUtils$$anonfun$htmlResponderToServlet$1.apply(JettyUtils.scala:55)
	at org.apache.spark.ui.JettyUtils$$anon$1.doGet(JettyUtils.scala:83)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:735)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:848)
	at org.spark-project.jetty.servlet.ServletHolder.handle(ServletHolder.java:684)
	at org.spark-project.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1507)
	at org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter.doFilter(AmIpFilter.java:179)
	at org.spark-project.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1478)
	at org.spark-project.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:499)
	at org.spark-project.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1086)
	at org.spark-project.jetty.servlet.ServletHandler.doScope(ServletHandler.java:427)
	at org.spark-project.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1020)
	at org.spark-project.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
	at org.spark-project.jetty.server.handler.GzipHandler.handle(GzipHandler.java:301)
	at org.spark-project.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:255)
	at org.spark-project.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
	at org.spark-project.jetty.server.Server.handle(Server.java:370)
	at org.spark-project.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:494)
	at org.spark-project.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:973)
	at org.spark-project.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:1035)
	at org.spark-project.jetty.http.HttpParser.parseNext(HttpParser.java:641)
	at org.spark-project.jetty.http.HttpParser.parseAvailable(HttpParser.java:231)
	at org.spark-project.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82)
	at org.spark-project.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:696)
	at org.spark-project.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:53)
</pre>
<hr /><i><small>Powered by Jetty://</small></i><br/>                                                
</body>
</html>

I can see the reason is java.lang.OutOfMemoryError: Java heap space. But how can I fix that ?


3 REPLIES 3

Re: Spark Thrift job problem HTTP ERROR 500

Super Collaborator

Hi @Hoang Le,

Not sure if you hit the bug in your version - https://issues.apache.org/jira/browse/SPARK-14261

if so upgrade may help !!

Re: Spark Thrift job problem HTTP ERROR 500

@bkosaraju

I think my issue not relevant with your link

I am running Spark 1.6.3 on HDP 2.6

Re: Spark Thrift job problem HTTP ERROR 500

Rising Star

Are you using impersonation (ie. hive.server2.enable.doAs=true)? Which version of HDP are you running? Thanks.

Don't have an account?
Coming from Hortonworks? Activate your account here