Tez view on Ambari throwing the exception,

[#1]

ERROR [ambari-client-thread-749] ContainerResponse:537 - Mapped exception to response: 500 (Internal Server Error) org.apache.ambari.view.tez.exceptions.TezWebAppException at org.apache.ambari.view.tez.utils.ProxyHelper.getResponse(ProxyHelper.java:79) at org.apache.ambari.view.tez.rest.BaseProxyResource.getData(BaseProxyResource.java:64) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60) at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:205) at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75) at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:302) at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147) at com.sun.jersey.server.impl.uri.rules.SubLocatorRule.accept(SubLocatorRule.java:137) at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147) at com.sun.jersey.server.impl.uri.rules.SubLocatorRule.accept(SubLocatorRule.java:137) at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147) at com.sun.jersey.server.impl.uri.rules.SubLocatorRule.accept(SubLocatorRule.java:137) at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147) at com.sun.jersey.server.impl.uri.rules.SubLocatorRule.accept(SubLocatorRule.java:137) at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147) at com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108) at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147) at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84) at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1542) at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1473) at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1419) at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1409) at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:409) at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:558) at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:733) at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:684) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1507) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330) at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:118) at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:84) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) at org.apache.ambari.server.security.authorization.AmbariAuthorizationFilter.doFilter(AmbariAuthorizationFilter.java:257) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:113) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:103) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:113) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:54) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:45) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) at org.apache.ambari.server.security.authorization.jwt.JwtAuthenticationFilter.doFilter(JwtAuthenticationFilter.java:96) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) at org.springframework.security.web.authentication.www.BasicAuthenticationFilter.doFilter(BasicAuthenticationFilter.java:150) at org.apache.ambari.server.security.authentication.AmbariAuthenticationFilter.doFilter(AmbariAuthenticationFilter.java:88) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) at org.apache.ambari.server.security.authorization.AmbariUserAuthorizationFilter.doFilter(AmbariUserAuthorizationFilter.java:91) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:87) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:192) at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:160) at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:237) at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:167) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1478) at org.apache.ambari.server.api.MethodOverrideFilter.doFilter(MethodOverrideFilter.java:72) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1478) at org.apache.ambari.server.api.AmbariPersistFilter.doFilter(AmbariPersistFilter.java:47) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1478) at org.apache.ambari.server.view.AmbariViewsMDCLoggingFilter.doFilter(AmbariViewsMDCLoggingFilter.java:54) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1478) at org.apache.ambari.server.view.ViewThrottleFilter.doFilter(ViewThrottleFilter.java:161) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1478) at org.apache.ambari.server.security.AbstractSecurityHeaderFilter.doFilter(AbstractSecurityHeaderFilter.java:109) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1478) at org.apache.ambari.server.security.AbstractSecurityHeaderFilter.doFilter(AbstractSecurityHeaderFilter.java:109) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1478) at org.eclipse.jetty.servlets.UserAgentFilter.doFilter(UserAgentFilter.java:82) at org.eclipse.jetty.servlets.GzipFilter.doFilter(GzipFilter.java:294) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1478) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:499) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:557) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1086) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:427) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1020) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135) at org.apache.ambari.server.controller.AmbariHandlerList.processHandlers(AmbariHandlerList.java:212) at org.apache.ambari.server.controller.AmbariHandlerList.processHandlers(AmbariHandlerList.java:201) at org.apache.ambari.server.controller.AmbariHandlerList.handle(AmbariHandlerList.java:150) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116) at org.eclipse.jetty.server.Server.handle(Server.java:370) at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:494) at org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:973) at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:1035) at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:641) at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:231) at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82) at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:696) at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:53) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543) at java.lang.Thread.run(Thread.java:745)

[#2]

could you give some more information? an internal server error can mean many things...

1. Is the view part of Ambari Cluster or is it a Standalone Ambari server for Views?
2. Which version of Ambari and HDP are you using?
3. Did Tez view ever work before? If so, what changed?
4. As which user does ambari-server process run? Check using ambari.properties ambari-server.user
5. As which user are you trying the view? What are the privileges assigned to the user in the view?
6. Are the guidelines defined in Ambari documentation followed correctly?
   1. http://docs.hortonworks.com/HDPDocuments/Ambari-2.2.2.18/bk_ambari-views/content/ch_using_tez_view.h...
7. Is the cluster kerberized?
   1. If views are hosted in a separate server, is the ambari keytab the same in Ambari Server and View server?
8. What configuration method was used to create the view?
   1. Using Local Cluster
   2. Using Remote Cluster
   3. Using Custom Configuration
9. What is the ATS Version used?
   1. If ATS 1.5 is used, in yarn-site is the following set:
   1. yarn.timeline-service.entity-group-fs-store.group-id-plugin-classes = org.apache.tez.dag.history.logging.ats.TimelineCachePluginImpl

[#3]

@Ana Gillan

Thank for you reply.

Yes true internal server error mean many thing, but i cannot found any error other form this. And details as below,

1. View is the part of the ambari, as i am using the Ambari Version2.4.2.0.

2. Ambari Version-2.4.2.0 HDP-2.5.3.0.

3. Yes, Tez view was working perfectly fine before.

4. Ambari-server running with "root" user.

5. Ambari is integrated with AD, so i am using one of the user belong to AD.

Now very strange with issue is two user belongs to same AD and same permission on Ambari, for one user profile tez view got loaded, but for other user profile getting the above exception.

6.

7. Yes cluster is kerberized , and the keytab are available on that server too.

8. Tez view created with the ambari installation.

9. yarn.timeline-service.entity-group-fs-store.group-id-plugin-classes is not assigned with any value.

[#4]

@Rajendra Manjunath

Based on the error you reported and the stackTrace it looks like it is failing to get results from applicationhistory server. So we will need to look at the "timelineserver" server log first of the same timestamp when you noticed the error in Tez View. 500 shows an internal server error that might be logged in the timelineserver log.

Example: If timelineserver is running on host "erie3.example.com" (here: yarn.timeline-service.webapp.address is the address "erie3.example.com:8188" then log location will be.

/var/log/hadoop-yarn/yarn/yarn-yarn-timelineserver-erie3.example.com.log

.

Tez View will make the following kind of proxy call.

http://erie1.example.com:8080/api/v1/views/TEZ/versions/0.7.0.2.5.3.0-136/instances/TEZ_CLUSTER_INST...

Which is actually the following call to the 'timelineserver' server. (Here 'application_1487157748136_0002' is any applicationID).

http://<yarn.timeline-service.webapp.address>:8188/ws/v1/applicationhistory/apps/application_1487157...

So next time when you face this issue then try to access the "ws/v1/applicationhistory/apps" URL directoy on the "timelineserver" to see if it is returning results fine or not?

.

[#5]

@Jay SenSharmaThanks Jay.!!

I checked the timeserver log before and now too, no error on those logs. I am getting only the above exception on ambari-server log not on others.

[#6]

@Rajendra Manjunath

Can you please share details of the exception from the file "/var/log/ambari-server/tez-view/tez-view.log" this file might have some additional details that are not present in amabri-server.log.

[#7]

@Jay SenSharma

i got below mentioned exception on tez-view.log, And to solve it, i tried increasesing the heap of ambari on property file, but it didnt work.

17 Feb 2017 08:38:54,611 ERROR [ambari-client-thread-48382] [TEZ 0.7.0.2.5.3.0-136 TEZ_CLUSTER_INSTANCE] ProxyHelper:78 - Cannot access the url: http://XXXXXXXXXXXXXXXXXX:8188/ws/v1/timeline/TEZ_DAG_ID?limit=11&_=1487317033565 java.net.SocketTimeoutException: Read timed out at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.socketRead(SocketInputStream.java:116) at java.net.SocketInputStream.read(SocketInputStream.java:170) at java.net.SocketInputStream.read(SocketInputStream.java:141) at java.io.BufferedInputStream.fill(BufferedInputStream.java:246) at java.io.BufferedInputStream.read1(BufferedInputStream.java:286) at java.io.BufferedInputStream.read(BufferedInputStream.java:345) at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:704) at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:647) at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1569) at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1474) at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:480)

[#8]

@Rajendra Manjunath

Here we see the cause of error is:

java.net.SocketTimeoutException: Read timed out at java.net.SocketInputStream.socketRead0(Native Method) 

.

It looks like a Timeout issue. Please try to increase the following parameter values in your "/etc/ambari-server/conf/ambari.properties" file:

views.request.read.timeout.millis=60000
views.request.connect.timeout.millis=30000

Then restart ambari-server and then try accessing the Tez View again.

[#9]

@Jay SenSharmaSuggested was the first activity i did, but it didnot worked in my case.