Support Questions
Find answers, ask questions, and share your expertise

Service 'hdfs' check failed-From Ambari

Solved Go to solution

Re: Service 'hdfs' check failed-From Ambari

Mentor

@Kishore Kumar

can you restart the agents and retry these errors may go away?

Re: Service 'hdfs' check failed-From Ambari

Expert Contributor
@Kishore Kumar

As given by @Jay SenSharma and @Geoffrey Shelton Okot the error:

18 Aug 2017 11:08:08,969 ERROR [ambari-client-thread-26] ViewRegistry:930 - Could not find the cluster identified by 2.
18 Aug 2017 11:08:08,970 ERROR [ambari-client-thread-26] ContainerResponse:419 - The RuntimeException could not be mapped to a response, re-throwing to the HTTP container
org.apache.ambari.server.view.IllegalClusterException: Failed to get cluster information associated with this view instance

doesn't look good, can you please follow the official documentation to create the new files view: https://docs.hortonworks.com/HDPDocuments/Ambari-2.5.1.0/bk_ambari-views/content/configuring_your_cl...

And also make sure you are able to reach the cluster from the Ambari host, and the forward and reserve lookup of the cluster (NameNodes) from the Ambari server is going through.

Re: Service 'hdfs' check failed-From Ambari

Mentor

@Kishore Kumar

Have you got the correct host entries in /etc/hosts and passwordless connection set from ambari to the other node.

Re: Service 'hdfs' check failed-From Ambari

Contributor

Based on the article . I added following properties and restated the necessary components .

hadoop.proxyuser.hdp.groups=users

hadoop.proxyuser.hdp.hosts=ambari-server.hostname
Now I see the following error , which close to same as previous

18 Aug 2017 15:31:55,734 ERROR [ambari-client-thread-24] ViewRegistry:930 - Could not find the cluster identified by 2.
18 Aug 2017 15:31:55,736 ERROR [ambari-client-thread-24] ContainerResponse:419 - The RuntimeException could not be mapped to a response, re-throwing to the HTTP container
org.apache.ambari.server.view.IllegalClusterException: Failed to get cluster information associated with this view instance
at org.apache.ambari.server.view.ViewRegistry.getCluster(ViewRegistry.java:931)
at org.apache.ambari.server.view.ViewContextImpl.getCluster(ViewContextImpl.java:370)
at org.apache.ambari.server.view.ViewContextImpl.getPropertyValues(ViewContextImpl.java:437)
at org.apache.ambari.server.view.ViewContextImpl.getProperties(ViewContextImpl.java:171)
at org.apache.ambari.view.commons.hdfs.ViewPropertyHelper.getViewConfigs(ViewPropertyHelper.java:36)
at org.apache.ambari.view.filebrowser.FileBrowserService.getViewConfigs(FileBrowserService.java:52)
at org.apache.ambari.view.filebrowser.FileBrowserService.help(FileBrowserService.java:80)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.sun.jersey.server.impl.uri.rules.SubLocatorRule.dispatch(SubLocatorRule.java:190)
at com.sun.jersey.server.impl.uri.rules.SubLocatorRule.invokeSubLocator(SubLocatorRule.java:183)
at com.sun.jersey.server.impl.uri.rules.SubLocatorRule.accept(SubLocatorRule.java:110)
at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
at com.sun.jersey.server.impl.uri.rules.SubLocatorRule.accept(SubLocatorRule.java:137)
at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
at com.sun.jersey.server.impl.uri.rules.SubLocatorRule.accept(SubLocatorRule.java:137)
at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
at com.sun.jersey.server.impl.uri.rules.SubLocatorRule.accept(SubLocatorRule.java:137)

Caused by: org.apache.ambari.server.ClusterNotFoundException: Cluster not found, clusterId=2
at org.apache.ambari.server.state.cluster.ClustersImpl.getCluster(ClustersImpl.java:290)
at org.apache.ambari.server.view.ViewRegistry.getCluster(ViewRegistry.java:928)
... 102 more

Did the restart for all components as well .

Re: Service 'hdfs' check failed-From Ambari

Contributor

After deleting the File view and recreating again . Clean restart now i can see the "Service checks completed."

But its showing blank page . After looking at the log i see following error

18 Aug 2017 16:02:47,190 ERROR [ambari-client-thread-29] ContainerResponse:537 - Mapped exception to response: 500 (Internal Server Error) org.apache.ambari.view.commons.exceptions.ServiceFormattedException at org.apache.ambari.view.commons.hdfs.FileOperationService.listdir(FileOperationService.java:80) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60) at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:205) at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75) at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:302) at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147) at com.sun.jersey.server.impl.uri.rules.SubLocatorRule.accept(SubLocatorRule.java:137) at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147) at com.sun.jersey.server.impl.uri.rules.SubLocatorRule.accept(SubLocatorRule.java:137) at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147) at com.sun.jersey.server.impl.uri.rules.SubLocatorRule.accept(SubLocatorRule.java:137) at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147) at com.sun.jersey.server.impl.uri.rules.SubLocatorRule.accept(SubLocatorRule.java:137) at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147) at com.sun.jersey.server.impl.uri.rules.SubLocatorRule.accept(SubLocatorRule.java:137) at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147) at com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108) at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147) at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84) at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1542) at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1473) at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1419) at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1409) at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:409) at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:558) at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:733) at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:684) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1507) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330) at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:118) at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:84) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) at org.apache.ambari.server.security.authorization.AmbariAuthorizationFilter.doFilter(AmbariAuthorizationFilter.java:287) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:113) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:103) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:113) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:54) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:45) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) at org.apache.ambari.server.security.authentication.AmbariDelegatingAuthenticationFilter.doFilter(AmbariDelegatingAuthenticationFilter.java:132) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) at org.apache.ambari.server.security.authorization.AmbariUserAuthorizationFilter.doFilter(AmbariUserAuthorizationFilter.java:91) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:87) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:192) at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:160) at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:

Re: Service 'hdfs' check failed-From Ambari

Mentor

Can you check the value in HDFS-->Configs-->Advanced then in the filter paste hadoop.proxyuser

Afer getting the values you will see the pattern,can you delete the one you created previously hadoop.proxyuser.hdp.groups=users

And try this

hadoop.proxyuser.root.groups=users 
hadoop.proxyuser.root.hosts=users 

Can you also run this command and paste the output in here

# cat /etc/ambari-server/conf/ambari.properties | grep ambari-server.user

The user from the above output should replace the xxx below

hadoop.proxyuser.xxx.groups=* 
hadoop.proxyuser.xxx.hosts=* 

And remember to start all the stale configs

Cheers

Re: Service 'hdfs' check failed-From Ambari

Contributor

Here is the output of :

cat /etc/ambari-server/conf/ambari.properties | grep ambari-server.user

ambari-server.user=hdp

After changing the properties and restart of server , still i see following error .

19 Aug 2017 01:51:39,696 ERROR [ambari-metrics-retrieval-service-thread-14] MetricsRetrievalService:492 - Unable to retrieve metrics from http://c9t29418.itcs.hpecorp.net:50070/jmx. Subsequent failures will be suppressed from the log for 20 minutes. java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:589) at sun.net.NetworkClient.doConnect(NetworkClient.java:175) at sun.net.www.http.HttpClient.openServer(HttpClient.java:432) at sun.net.www.http.HttpClient.openServer(HttpClient.java:527) at sun.net.www.http.HttpClient.<init>(HttpClient.java:211) at sun.net.www.http.HttpClient.New(HttpClient.java:308) at sun.net.www.http.HttpClient.New(HttpClient.java:326) at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:1202) at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1138) at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1032) at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:966) at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1546) at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1474) at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:480) at org.apache.ambari.server.controller.internal.URLStreamProvider.processURL(URLStreamProvider.java:218) at org.apache.ambari.server.controller.internal.URLStreamProvider.processURL(URLStreamProvider.java:142) at org.apache.ambari.server.controller.internal.URLStreamProvider.readFrom(URLStreamProvider.java:116) at org.apache.ambari.server.controller.internal.URLStreamProvider.readFrom(URLStreamProvider.java:121) at org.apache.ambari.server.state.services.MetricsRetrievalService$MetricRunnable.run(MetricsRetrievalService.java:438) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745)

Re: Service 'hdfs' check failed-From Ambari

Contributor

I addition to that following error also .

19 Aug 2017 01:56:25,605 ERROR [ambari-client-thread-28] WebHdfsFileSystem:402 - Unable to get HomeDirectory from original File System 19 Aug 2017 01:56:25,615 ERROR [ambari-client-thread-28] ContainerResponse:537 - Mapped exception to response: 500 (Internal Server Error) org.apache.ambari.view.commons.exceptions.ServiceFormattedException at org.apache.ambari.view.commons.hdfs.UserService.homeDir(UserService.java:73) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60) at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:205) at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)

Re: Service 'hdfs' check failed-From Ambari

Contributor

Re: Service 'hdfs' check failed-From Ambari

Mentor

@Kishore Kumar

Can you briefly descript the setup of your cluster?

Does the ambari server have the correct IP/HOST entries in the /etc/hosts

Have you disabled firewall between the cluster nodes?

How is your DNS entry in /etc/resolv.conf

Does the user hdp have a home directory in hdfs ,trun below command as user hdfs

$ hdfs dfs -ls /user

Please revert