Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Got 500 Error when trying to view status of a running mapreduce job

avatar
Rising Star

We just launched our cluster with CDH5.3.3 on AWS, and We are currently testing our process on the new cluster.

Our job ran fine, but we ran into issues when we tried to view the actual status detail page of a running job.

We got HTTP 500 error when we either click on the 'Application Master' link on the Resource Manager UI page or by click on the link under 'Child Job Urls' tab when the job is running. We were able to get the job status page after it completed, but not while it was still running. 

Is anyone know how to fix this issue? This is a real issue for us.

The detail error is listed below.

Thanks

 

HTTP ERROR 500

Problem accessing /. Reason:

    Guice configuration errors:

1) Could not find a suitable constructor in com.sun.jersey.guice.spi.container.servlet.GuiceContainer. Classes must have either one (and only one) constructor annotated with @Inject or a zero-argument constructor that is not private.
  at com.sun.jersey.guice.spi.container.servlet.GuiceContainer.class(GuiceContainer.java:108)
  while locating com.sun.jersey.guice.spi.container.servlet.GuiceContainer

1 error

 

Caused by:

com.google.inject.ConfigurationException: Guice configuration errors:

1) Could not find a suitable constructor in com.sun.jersey.guice.spi.container.servlet.GuiceContainer. Classes must have either one (and only one) constructor annotated with @Inject or a zero-argument constructor that is not private.
  at com.sun.jersey.guice.spi.container.servlet.GuiceContainer.class(GuiceContainer.java:108)
  while locating com.sun.jersey.guice.spi.container.servlet.GuiceContainer

1 error
	at com.google.inject.InjectorImpl.getBinding(InjectorImpl.java:113)
	at com.google.inject.InjectorImpl.getBinding(InjectorImpl.java:63)
	at com.google.inject.servlet.FilterDefinition.init(FilterDefinition.java:99)
	at com.google.inject.servlet.ManagedFilterPipeline.initPipeline(ManagedFilterPipeline.java:98)
	at com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:114)
	at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:113)
	at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
	at org.apache.hadoop.http.HttpServer2$QuotingInputFilter.doFilter(HttpServer2.java:1224)
	at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
	at org.apache.hadoop.http.NoCacheFilter.doFilter(NoCacheFilter.java:45)
	at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
	at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
	at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
	at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
	at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:767)
	at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
	at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
	at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
	at org.mortbay.jetty.Server.handle(Server.java:326)
	at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
	at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
	at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
	at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
	at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
	at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
	at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)

 

23 REPLIES 23

avatar
Super Collaborator

A container log is not part of the yarn service logs and will not be affected by any of the yarn settings. The container log looks like a log from an AM and that means that you most likely are looking at a problem of the AM web UI not being able to bind.

The AM web ui will bind to an ephemeral port which can not be limited to a set of ports. Make sure that you allow binding to any port on the NM's from your security groups in AWS.

 

Wilfred

avatar
Explorer

It doesn't look like we have any restrictions on our NodeManager instances.  We only have one outbound security rule in all our groups with the following permissions.

 

Ports: ALL

Protocols: ALL

Source: 10.0.0.0/8

avatar
Explorer

I'm having a similar issue but with a different stack trace. Were you able to resolve this problem? 

avatar
Explorer

We were finally able to resolve this issue.

 

For us, we were using classpath precedence override's to use our own classpath lib jars before cloudera's common lib directory.  This resulted in an older version of guice being loaded for certain actions which caused the errors in the application logs.

 

We resolved this issue by resolving the dependency issue between our projects and cloudera's lib.