Got 500 Error when trying to view status of a running mapreduce job

[#1]

We just launched our cluster with CDH5.3.3 on AWS, and We are currently testing our process on the new cluster.

Our job ran fine, but we ran into issues when we tried to view the actual status detail page of a running job.

We got HTTP 500 error when we either click on the 'Application Master' link on the Resource Manager UI page or by click on the link under 'Child Job Urls' tab when the job is running. We were able to get the job status page after it completed, but not while it was still running. 

Is anyone know how to fix this issue? This is a real issue for us.

The detail error is listed below.

Thanks

HTTP ERROR 500

Problem accessing /. Reason:

    Guice configuration errors:

1) Could not find a suitable constructor in com.sun.jersey.guice.spi.container.servlet.GuiceContainer. Classes must have either one (and only one) constructor annotated with @Inject or a zero-argument constructor that is not private.
  at com.sun.jersey.guice.spi.container.servlet.GuiceContainer.class(GuiceContainer.java:108)
  while locating com.sun.jersey.guice.spi.container.servlet.GuiceContainer

1 error

Caused by:

com.google.inject.ConfigurationException: Guice configuration errors:

1) Could not find a suitable constructor in com.sun.jersey.guice.spi.container.servlet.GuiceContainer. Classes must have either one (and only one) constructor annotated with @Inject or a zero-argument constructor that is not private.
  at com.sun.jersey.guice.spi.container.servlet.GuiceContainer.class(GuiceContainer.java:108)
  while locating com.sun.jersey.guice.spi.container.servlet.GuiceContainer

1 error
	at com.google.inject.InjectorImpl.getBinding(InjectorImpl.java:113)
	at com.google.inject.InjectorImpl.getBinding(InjectorImpl.java:63)
	at com.google.inject.servlet.FilterDefinition.init(FilterDefinition.java:99)
	at com.google.inject.servlet.ManagedFilterPipeline.initPipeline(ManagedFilterPipeline.java:98)
	at com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:114)
	at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:113)
	at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
	at org.apache.hadoop.http.HttpServer2$QuotingInputFilter.doFilter(HttpServer2.java:1224)
	at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
	at org.apache.hadoop.http.NoCacheFilter.doFilter(NoCacheFilter.java:45)
	at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
	at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
	at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
	at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
	at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:767)
	at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
	at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
	at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
	at org.mortbay.jetty.Server.handle(Server.java:326)
	at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
	at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
	at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
	at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
	at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
	at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
	at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)

[#2]

A container log is not part of the yarn service logs and will not be affected by any of the yarn settings. The container log looks like a log from an AM and that means that you most likely are looking at a problem of the AM web UI not being able to bind.

The AM web ui will bind to an ephemeral port which can not be limited to a set of ports. Make sure that you allow binding to any port on the NM's from your security groups in AWS.

Wilfred

[#3]

It doesn't look like we have any restrictions on our NodeManager instances.  We only have one outbound security rule in all our groups with the following permissions.

Ports: ALL

Protocols: ALL

Source: 10.0.0.0/8

[#4]

I'm having a similar issue but with a different stack trace. Were you able to resolve this problem? 

[#5]

We were finally able to resolve this issue.

For us, we were using classpath precedence override's to use our own classpath lib jars before cloudera's common lib directory.  This resulted in an older version of guice being loaded for certain actions which caused the errors in the application logs.

We resolved this issue by resolving the dependency issue between our projects and cloudera's lib.