Support Questions
Find answers, ask questions, and share your expertise

Timeline server becomes unresponsive

New Contributor

Timeline server becomes unresponsive after starting and running ~1 hr. The timeline UI is not accessible.

The jstack command shows 274 of its threads are BLOCKED and 5 threads are IN_NATIVE:

Attaching to process ID 31852, please wait...

Debugger attached successfully.

Server compiler detected.

JVM version is 25.121-b13

Deadlock Detection:

No deadlocks found.

Thread 10453: (state = BLOCKED)

- sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)

- java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=175 (Compiled frame)

- java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt() @bci=1, line=836 (Compiled frame)

- java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(int) @bci=83, line=967 (Compiled frame)

- java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(int) @bci=10, line=1283 (Compiled frame)

- java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock() @bci=5, line=727 (Compiled frame)

- org.apache.hadoop.yarn.server.timeline.LeveldbTimelineStore.put(org.apache.hadoop.yarn.api.records.timeline.TimelineEntities) @bci=7, line=1028 (Compiled frame)

- org.apache.hadoop.yarn.server.timeline.TimelineDataManager.postEntities(org.apache.hadoop.yarn.api.records.timeline.TimelineEntities, org.apache.hadoop.security.UserGroupInformation) @bci=475, line=296 (Compiled frame)

- org.apache.hadoop.yarn.server.timeline.webapp.TimelineWebServices.postEntities(javax.servlet.http.HttpServletRequest, javax.servlet.http.HttpServletResponse, org.apache.hadoop.yarn.api.records.timeline.TimelineEntities) @bci=47, line=259 (Compiled frame)

- sun.reflect.GeneratedMethodAccessor34.invoke(java.lang.Object, java.lang.Object[]) @bci=56 (Compiled frame)

- sun.reflect.DelegatingMethodAccessorImpl.invoke(java.lang.Object, java.lang.Object[]) @bci=6, line=43 (Compiled frame)

- java.lang.reflect.Method.invoke(java.lang.Object, java.lang.Object[]) @bci=56, line=498 (Compiled frame)

- com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(java.lang.reflect.Method, java.lang.Object, java.lang.Object[]) @bci=3, line=60 (Compiled frame)

- com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$TypeOutInvoker._dispatch(java.lang.Object, com.sun.jersey.api.core.HttpContext) @bci=16, line=185 (Compiled frame)

- com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(java.lang.Object, com.sun.jersey.api.core.HttpContext) @bci=3, line=75 (Compiled frame)

- com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(java.lang.CharSequence, java.lang.Object, com.sun.jersey.spi.uri.rules.UriRuleContext) @bci=606, line=288 (Compiled frame)

- com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(java.lang.CharSequence, java.lang.Object, com.sun.jersey.spi.uri.rules.UriRuleContext) @bci=137, line=108 (Compiled frame)

- com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(java.lang.CharSequence, java.lang.Object, com.sun.jersey.spi.uri.rules.UriRuleContext) @bci=242, line=147 (Compiled frame)

- com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(java.lang.CharSequence, java.lang.Object, com.sun.jersey.spi.uri.rules.UriRuleContext) @bci=86, line=84 (Compiled frame)

- com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(com.sun.jersey.server.impl.application.WebApplicationContext, com.sun.jersey.spi.container.ContainerRequest) @bci=100, line=1469 (Compiled frame)

- com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(com.sun.jersey.server.impl.application.WebApplicationContext, com.sun.jersey.spi.container.ContainerRequest, com.sun.jersey.spi.container.ContainerResponse) @bci=19, line=1400 (Compiled frame)

- com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(com.sun.jersey.spi.container.ContainerRequest, com.sun.jersey.spi.container.ContainerResponse) @bci=23, line=1349 (Compiled frame)

- com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(com.sun.jersey.spi.container.ContainerRequest, com.sun.jersey.spi.container.ContainerResponseWriter) @bci=14, line=1339 (Compiled frame)

- com.sun.jersey.spi.container.servlet.WebComponent.service(java.net.URI, java.net.URI, javax.servlet.http.HttpServletRequest, javax.servlet.http.HttpServletResponse) @bci=80, line=416 (Compiled frame)

- com.sun.jersey.spi.container.servlet.ServletContainer.service(java.net.URI, java.net.URI, javax.servlet.http.HttpServletRequest, javax.servlet.http.HttpServletResponse) @bci=9, line=537 (Compiled frame)

- com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(javax.servlet.http.HttpServletRequest, javax.servlet.http.HttpServletResponse, javax.servlet.FilterChain, java.lang.String, java.lang.String, java.lang.String) @bci=310, line=886 (Compiled frame)

- com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(javax.servlet.http.HttpServletRequest, javax.servlet.http.HttpServletResponse, javax.servlet.FilterChain) @bci=92, line=834 (Compiled frame)

- com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(javax.servlet.ServletRequest, javax.servlet.ServletResponse, javax.servlet.FilterChain) @bci=10, line=795 (Compiled frame)

- com.google.inject.servlet.FilterDefinition.doFilter(javax.servlet.ServletRequest, javax.servlet.ServletResponse, com.google.inject.servlet.FilterChainInvocation) @bci=50, line=163 (Compiled frame)

- com.google.inject.servlet.FilterChainInvocation.doFilter(javax.servlet.ServletRequest, javax.servlet.ServletResponse) @bci=34, line=58 (Compiled frame)

- com.google.inject.servlet.ManagedFilterPipeline.dispatch(javax.servlet.ServletRequest, javax.servlet.ServletResponse, javax.servlet.FilterChain) @bci=49, line=118 (Compiled frame)

- com.google.inject.servlet.GuiceFilter.doFilter(javax.servlet.ServletRequest, javax.servlet.ServletResponse, javax.servlet.FilterChain) @bci=57, line=113 (Compiled frame)

- org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(javax.servlet.ServletRequest, javax.servlet.ServletResponse) @bci=53, line=1212 (Compiled frame)

- org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter.doFilter(javax.servlet.ServletRequest, javax.servlet.ServletResponse, javax.servlet.FilterChain) @bci=19, line=96 (Compiled frame)

- org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(javax.servlet.ServletRequest, javax.servlet.ServletResponse) @bci=53, line=1212 (Compiled frame)

- org.apache.hadoop.yarn.server.timeline.webapp.CrossOriginFilter.doFilter(javax.servlet.ServletRequest, javax.servlet.ServletResponse, javax.servlet.FilterChain) @bci=15, line=95 (Compiled frame)

- org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(javax.servlet.ServletRequest, javax.servlet.ServletResponse) @bci=53, line=1212 (Compiled frame)

- org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(javax.servlet.FilterChain, javax.servlet.http.HttpServletRequest, javax.servlet.http.HttpServletResponse) @bci=3, line=622 (Compiled frame)

- org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticationFilter.doFilter(javax.servlet.FilterChain, javax.servlet.http.HttpServletRequest, javax.servlet.http.HttpServletResponse) @bci=231, line=301 (Compiled frame)

- org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(javax.servlet.ServletRequest, javax.servlet.ServletResponse, javax.servlet.FilterChain) @bci=324, line=574 (Compiled frame)

- org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(javax.servlet.ServletRequest, javax.servlet.ServletResponse) @bci=53, line=1212 (Compiled frame)

- org.apache.hadoop.http.HttpServer2$QuotingInputFilter.doFilter(javax.servlet.ServletRequest, javax.servlet.ServletResponse, javax.servlet.FilterChain) @bci=104, line=1296 (Compiled frame)

- org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(javax.servlet.ServletRequest, javax.servlet.ServletResponse) @bci=53, line=1212 (Compiled frame)

- org.apache.hadoop.http.NoCacheFilter.doFilter(javax.servlet.ServletRequest, javax.servlet.ServletResponse, javax.servlet.FilterChain) @bci=58, line=45 (Compiled frame)

- org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(javax.servlet.ServletRequest, javax.servlet.ServletResponse) @bci=53, line=1212 (Compiled frame)

- org.apache.hadoop.http.NoCacheFilter.doFilter(javax.servlet.ServletRequest, javax.servlet.ServletResponse, javax.servlet.FilterChain) @bci=58, line=45 (Compiled frame)

- org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(javax.servlet.ServletRequest, javax.servlet.ServletResponse) @bci=53, line=1212 (Compiled frame)

- org.mortbay.jetty.servlet.ServletHandler.handle(java.lang.String, javax.servlet.http.HttpServletRequest, javax.servlet.http.HttpServletResponse, int) @bci=465, line=399 (Compiled frame)

- org.mortbay.jetty.security.SecurityHandler.handle(java.lang.String, javax.servlet.http.HttpServletRequest, javax.servlet.http.HttpServletResponse, int) @bci=299, line=216 (Compiled frame)

- org.mortbay.jetty.servlet.SessionHandler.handle(java.lang.String, javax.servlet.http.HttpServletRequest, javax.servlet.http.HttpServletResponse, int) @bci=225, line=182 (Compiled frame)

- org.mortbay.jetty.handler.ContextHandler.handle(java.lang.String, javax.servlet.http.HttpServletRequest, javax.servlet.http.HttpServletResponse, int) @bci=721, line=766 (Compiled frame)

- org.mortbay.jetty.webapp.WebAppContext.handle(java.lang.String, javax.servlet.http.HttpServletRequest, javax.servlet.http.HttpServletResponse, int) @bci=53, line=450 (Compiled frame)

- org.mortbay.jetty.handler.ContextHandlerCollection.handle(java.lang.String, javax.servlet.http.HttpServletRequest, javax.servlet.http.HttpServletResponse, int) @bci=373, line=230 (Compiled frame)

- org.mortbay.jetty.handler.HandlerWrapper.handle(java.lang.String, javax.servlet.http.HttpServletRequest, javax.servlet.http.HttpServletResponse, int) @bci=23, line=152 (Compiled frame)

- org.mortbay.jetty.Server.handle(org.mortbay.jetty.HttpConnection) @bci=56, line=322 (Compiled frame)

- org.mortbay.jetty.HttpConnection.handleRequest() @bci=137, line=542 (Compiled frame)

- org.mortbay.jetty.HttpConnection$RequestHandler.content(org.mortbay.io.Buffer) @bci=49, line=945 (Compiled frame)

- org.mortbay.jetty.HttpParser.parseNext() @bci=2872, line=707 (Compiled frame)

- org.mortbay.jetty.HttpParser.parseAvailable() @bci=44, line=218 (Compiled frame)

- org.mortbay.jetty.HttpConnection.handle() @bci=122, line=404 (Compiled frame)

- org.mortbay.io.nio.SelectChannelEndPoint.run() @bci=4, line=410 (Compiled frame)

- org.mortbay.thread.QueuedThreadPool$PoolThread.run() @bci=25, line=582 (Compiled frame)

Thread 10452: (state = BLOCKED)

- sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)

- java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=175 (Compiled frame)

- java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt() @bci=1, line=836 (Compiled frame)

- java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(int) @bci=83, line=967 (Compiled frame)

- java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(int) @bci=10, line=1283 (Compiled frame)

- java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock() @bci=5, line=727 (Compiled frame)

- org.apache.hadoop.yarn.server.timeline.LeveldbTimelineStore.put(org.apache.hadoop.yarn.api.records.timeline.TimelineEntities) @bci=7, line=1028 (Compiled frame)

- org.apache.hadoop.yarn.server.timeline.TimelineDataManager.postEntities(org.apache.hadoop.yarn.api.records.timeline.TimelineEntities, org.apache.hadoop.security.UserGroupInformation) @bci=475, line=296 (Compiled frame)

- org.apache.hadoop.yarn.server.timeline.webapp.TimelineWebServices.postEntities(javax.servlet.http.HttpServletRequest, javax.servlet.http.HttpServletResponse, org.apache.hadoop.yarn.api.records.timeline.TimelineEntities) @bci=47, line=259 (Compiled frame)

- sun.reflect.GeneratedMethodAccessor34.invoke(java.lang.Object, java.lang.Object[]) @bci=56 (Compiled frame)

- sun.reflect.DelegatingMethodAccessorImpl.invoke(java.lang.Object, java.lang.Object[]) @bci=6, line=43 (Compiled frame)

- java.lang.reflect.Method.invoke(java.lang.Object, java.lang.Object[]) @bci=56, line=498 (Compiled frame)

- com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(java.lang.reflect.Method, java.lang.Object, java.lang.Object[]) @bci=3, line=60 (Compiled frame)

- com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$TypeOutInvoker._dispatch(java.lang.Object, com.sun.jersey.api.core.HttpContext) @bci=16, line=185 (Compiled frame)

- com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(java.lang.Object, com.sun.jersey.api.core.HttpContext) @bci=3, line=75 (Compiled frame)

- com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(java.lang.CharSequence, java.lang.Object, com.sun.jersey.spi.uri.rules.UriRuleContext) @bci=606, line=288 (Compiled frame)

- com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(java.lang.CharSequence, java.lang.Object, com.sun.jersey.spi.uri.rules.UriRuleContext) @bci=137, line=108 (Compiled frame)

- com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(java.lang.CharSequence, java.lang.Object, com.sun.jersey.spi.uri.rules.UriRuleContext) @bci=242, line=147 (Compiled frame)

- com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(java.lang.CharSequence, java.lang.Object, com.sun.jersey.spi.uri.rules.UriRuleContext) @bci=86, line=84 (Compiled frame)

- com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(com.sun.jersey.server.impl.application.WebApplicationContext, com.sun.jersey.spi.container.ContainerRequest) @bci=100, line=1469 (Compiled frame)

- com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(com.sun.jersey.server.impl.application.WebApplicationContext, com.sun.jersey.spi.container.ContainerRequest, com.sun.jersey.spi.container.ContainerResponse) @bci=19, line=1400 (Compiled frame)

- com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(com.sun.jersey.spi.container.ContainerRequest, com.sun.jersey.spi.container.ContainerResponse) @bci=23, line=1349 (Compiled frame)

- com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(com.sun.jersey.spi.container.ContainerRequest, com.sun.jersey.spi.container.ContainerResponseWriter) @bci=14, line=1339 (Compiled frame)

- com.sun.jersey.spi.container.servlet.WebComponent.service(java.net.URI, java.net.URI, javax.servlet.http.HttpServletRequest, javax.servlet.http.HttpServletResponse) @bci=80, line=416 (Compiled frame)

- com.sun.jersey.spi.container.servlet.ServletContainer.service(java.net.URI, java.net.URI, javax.servlet.http.HttpServletRequest, javax.servlet.http.HttpServletResponse) @bci=9, line=537 (Compiled frame)

- com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(javax.servlet.http.HttpServletRequest, javax.servlet.http.HttpServletResponse, javax.servlet.FilterChain, java.lang.String, java.lang.String, java.lang.String) @bci=310, line=886 (Compiled frame)

- com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(javax.servlet.http.HttpServletRequest, javax.servlet.http.HttpServletResponse, javax.servlet.FilterChain) @bci=92, line=834 (Compiled frame)

- com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(javax.servlet.ServletRequest, javax.servlet.ServletResponse, javax.servlet.FilterChain) @bci=10, line=795 (Compiled frame)

- com.google.inject.servlet.FilterDefinition.doFilter(javax.servlet.ServletRequest, javax.servlet.ServletResponse, com.google.inject.servlet.FilterChainInvocation) @bci=50, line=163 (Compiled frame)

- com.google.inject.servlet.FilterChainInvocation.doFilter(javax.servlet.ServletRequest, javax.servlet.ServletResponse) @bci=34, line=58 (Compiled frame)

- com.google.inject.servlet.ManagedFilterPipeline.dispatch(javax.servlet.ServletRequest, javax.servlet.ServletResponse, javax.servlet.FilterChain) @bci=49, line=118 (Compiled frame)

- com.google.inject.servlet.GuiceFilter.doFilter(javax.servlet.ServletRequest, javax.servlet.ServletResponse, javax.servlet.FilterChain) @bci=57, line=113 (Compiled frame)

- org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(javax.servlet.ServletRequest, javax.servlet.ServletResponse) @bci=53, line=1212 (Compiled frame)

- org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter.doFilter(javax.servlet.ServletRequest, javax.servlet.ServletResponse, javax.servlet.FilterChain) @bci=19, line=96 (Compiled frame)

- org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(javax.servlet.ServletRequest, javax.servlet.ServletResponse) @bci=53, line=1212 (Compiled frame)

- org.apache.hadoop.yarn.server.timeline.webapp.CrossOriginFilter.doFilter(javax.servlet.ServletRequest, javax.servlet.ServletResponse, javax.servlet.FilterChain) @bci=15, line=95 (Compiled frame)

- org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(javax.servlet.ServletRequest, javax.servlet.ServletResponse) @bci=53, line=1212 (Compiled frame)

- org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(javax.servlet.FilterChain, javax.servlet.http.HttpServletRequest, javax.servlet.http.HttpServletResponse) @bci=3, line=622 (Compiled frame)

- org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticationFilter.doFilter(javax.servlet.FilterChain, javax.servlet.http.HttpServletRequest, javax.servlet.http.HttpServletResponse) @bci=231, line=301 (Compiled frame)

- org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(javax.servlet.ServletRequest, javax.servlet.ServletResponse, javax.servlet.FilterChain) @bci=324, line=574 (Compiled frame)

- org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(javax.servlet.ServletRequest, javax.servlet.ServletResponse) @bci=53, line=1212 (Compiled frame)

- org.apache.hadoop.http.HttpServer2$QuotingInputFilter.doFilter(javax.servlet.ServletRequest, javax.servlet.ServletResponse, javax.servlet.FilterChain) @bci=104, line=1296 (Compiled frame)

- org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(javax.servlet.ServletRequest, javax.servlet.ServletResponse) @bci=53, line=1212 (Compiled frame)

- org.apache.hadoop.http.NoCacheFilter.doFilter(javax.servlet.ServletRequest, javax.servlet.ServletResponse, javax.servlet.FilterChain) @bci=58, line=45 (Compiled frame)

- org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(javax.servlet.ServletRequest, javax.servlet.ServletResponse) @bci=53, line=1212 (Compiled frame)

- org.apache.hadoop.http.NoCacheFilter.doFilter(javax.servlet.ServletRequest, javax.servlet.ServletResponse, javax.servlet.FilterChain) @bci=58, line=45 (Compiled frame)

- org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(javax.servlet.ServletRequest, javax.servlet.ServletResponse) @bci=53, line=1212 (Compiled frame)

- org.mortbay.jetty.servlet.ServletHandler.handle(java.lang.String, javax.servlet.http.HttpServletRequest, javax.servlet.http.HttpServletResponse, int) @bci=465, line=399 (Compiled frame)

- org.mortbay.jetty.security.SecurityHandler.handle(java.lang.String, javax.servlet.http.HttpServletRequest, javax.servlet.http.HttpServletResponse, int) @bci=299, line=216 (Compiled frame)

- org.mortbay.jetty.servlet.SessionHandler.handle(java.lang.String, javax.servlet.http.HttpServletRequest, javax.servlet.http.HttpServletResponse, int) @bci=225, line=182 (Compiled frame)

- org.mortbay.jetty.handler.ContextHandler.handle(java.lang.String, javax.servlet.http.HttpServletRequest, javax.servlet.http.HttpServletResponse, int) @bci=721, line=766 (Compiled frame)

- org.mortbay.jetty.webapp.WebAppContext.handle(java.lang.String, javax.servlet.http.HttpServletRequest, javax.servlet.http.HttpServletResponse, int) @bci=53, line=450 (Compiled frame)

- org.mortbay.jetty.handler.ContextHandlerCollection.handle(java.lang.String, javax.servlet.http.HttpServletRequest, javax.servlet.http.HttpServletResponse, int) @bci=373, line=230 (Compiled frame)

- org.mortbay.jetty.handler.HandlerWrapper.handle(java.lang.String, javax.servlet.http.HttpServletRequest, javax.servlet.http.HttpServletResponse, int) @bci=23, line=152 (Compiled frame)

- org.mortbay.jetty.Server.handle(org.mortbay.jetty.HttpConnection) @bci=56, line=322 (Compiled frame)

- org.mortbay.jetty.HttpConnection.handleRequest() @bci=137, line=542 (Compiled frame)

- org.mortbay.jetty.HttpConnection$RequestHandler.content(org.mortbay.io.Buffer) @bci=49, line=945 (Compiled frame)

- org.mortbay.jetty.HttpParser.parseNext() @bci=2872, line=707 (Compiled frame)

- org.mortbay.jetty.HttpParser.parseAvailable() @bci=44, line=218 (Compiled frame)

- org.mortbay.jetty.HttpConnection.handle() @bci=122, line=404 (Compiled frame)

- org.mortbay.io.nio.SelectChannelEndPoint.run() @bci=4, line=410 (Compiled frame)

- org.mortbay.thread.QueuedThreadPool$PoolThread.run() @bci=25, line=582 (Compiled frame)

1 REPLY 1

New Contributor

@Xin Yang - Were you able resolve this issue? We have encountered a similar issue where the number of timeline server threads jump from ~100 to 300, web requests to port 8188 are not fulfilled and yarn applications (issued by Hive/Tez) hang for hours:


"849734587@qtp-1767230265-1181" #1233 daemon prio=5 os_prio=0 tid=0x00007f58c215d000 nid=0x26d1 waiting on condition [0x00007f588ce6f000]

java.lang.Thread.State: WAITING (parking)

at sun.misc.Unsafe.park(Native Method)

- parking to wait for <0x00000000c197da50> (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)

at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)

at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:967)

at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1283)

at java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:727)

at org.apache.hadoop.yarn.server.timeline.LeveldbTimelineStore.put(LeveldbTimelineStore.java:991)

at org.apache.hadoop.yarn.server.timeline.TimelineDataManager.doPostEntities(TimelineDataManager.java:385)

at org.apache.hadoop.yarn.server.timeline.TimelineDataManager.postEntities(TimelineDataManager.java:317)

at org.apache.hadoop.yarn.server.timeline.webapp.TimelineWebServices.postEntities(TimelineWebServices.java:229)

at sun.reflect.GeneratedMethodAccessor55.invoke(Unknown Source)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:498)

at com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)

at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$TypeOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:185)

at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)

at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:288)

at com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)

at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)

at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)

at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1469)

at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1400)

at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1349)

at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1339)

at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:416)

at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:537)

at com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:886)

at com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:834)

at com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:795)

at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:163)

at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)

at com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:118)

at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:113)

at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)

at org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter.doFilter(StaticUserWebFilter.java:96)

at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)

at org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:644)

at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticationFilter.doFilter(DelegationTokenAuthenticationFilter.java:294)

at org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:592)

at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)

at org.apache.hadoop.security.http.CrossOriginFilter.doFilter(CrossOriginFilter.java:95)

at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)

at org.apache.hadoop.http.HttpServer2$QuotingInputFilter.doFilter(HttpServer2.java:1353)

at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)

at org.apache.hadoop.http.NoCacheFilter.doFilter(NoCacheFilter.java:45)

at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)

at org.apache.hadoop.http.NoCacheFilter.doFilter(NoCacheFilter.java:45)

at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)

at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)

at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)

at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)

at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)

at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)

at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)

at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)

at org.mortbay.jetty.Server.handle(Server.java:326)

at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)

at org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.java:945)

at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:756)

at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)

at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)

at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)

at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)


Take a Tour of the Community
Don't have an account?
Your experience may be limited. Sign in to explore more.