Service 'hdfs' check failed-From Ambari

[#1]

Hi Team ,

After installing HDP using the Ambari , from Menu if I click on "File View" .

It get the " Issues detected Service 'hdfs' check failed" . With following exception .

java.lang.NullPointerException
	at org.apache.ambari.server.view.ClusterImpl.getConfigByType(ClusterImpl.java:71)
	at org.apache.ambari.view.utils.hdfs.ConfigurationBuilder.copyPropertiesBySite(ConfigurationBuilder.java:203)
	at org.apache.ambari.view.utils.hdfs.ConfigurationBuilder.buildConfig(ConfigurationBuilder.java:311)
	at org.apache.ambari.view.utils.hdfs.HdfsApi.<init>(HdfsApi.java:68)
	at org.apache.ambari.view.utils.hdfs.HdfsUtil.getHdfsApi(HdfsUtil.java:150)
	at org.apache.ambari.view.utils.hdfs.HdfsUtil.connectToHDFSApi(HdfsUtil.java:138)
	at org.apache.ambari.view.commons.hdfs.HdfsService.hdfsSmokeTest(HdfsService.java:145)
	at org.apache.ambari.view.filebrowser.HelpService.hdfsStatus(HelpService.java:95)

[#2]

@Kishore Kumar

can you restart the agents and retry these errors may go away?

[#3]

@Kishore Kumar

As given by @Jay SenSharma and @Geoffrey Shelton Okot the error:

18 Aug 2017 11:08:08,969 ERROR [ambari-client-thread-26] ViewRegistry:930 - Could not find the cluster identified by 2.
18 Aug 2017 11:08:08,970 ERROR [ambari-client-thread-26] ContainerResponse:419 - The RuntimeException could not be mapped to a response, re-throwing to the HTTP container
org.apache.ambari.server.view.IllegalClusterException: Failed to get cluster information associated with this view instance

doesn't look good, can you please follow the official documentation to create the new files view: https://docs.hortonworks.com/HDPDocuments/Ambari-2.5.1.0/bk_ambari-views/content/configuring_your_cl...

And also make sure you are able to reach the cluster from the Ambari host, and the forward and reserve lookup of the cluster (NameNodes) from the Ambari server is going through.

[#4]

@Kishore Kumar

Have you got the correct host entries in /etc/hosts and passwordless connection set from ambari to the other node.

[#5]

Based on the article . I added following properties and restated the necessary components .

hadoop.proxyuser.hdp.groups=users

hadoop.proxyuser.hdp.hosts=ambari-server.hostname

Now I see the following error , which close to same as previous

18 Aug 2017 15:31:55,734 ERROR [ambari-client-thread-24] ViewRegistry:930 - Could not find the cluster identified by 2.
18 Aug 2017 15:31:55,736 ERROR [ambari-client-thread-24] ContainerResponse:419 - The RuntimeException could not be mapped to a response, re-throwing to the HTTP container
org.apache.ambari.server.view.IllegalClusterException: Failed to get cluster information associated with this view instance
at org.apache.ambari.server.view.ViewRegistry.getCluster(ViewRegistry.java:931)
at org.apache.ambari.server.view.ViewContextImpl.getCluster(ViewContextImpl.java:370)
at org.apache.ambari.server.view.ViewContextImpl.getPropertyValues(ViewContextImpl.java:437)
at org.apache.ambari.server.view.ViewContextImpl.getProperties(ViewContextImpl.java:171)
at org.apache.ambari.view.commons.hdfs.ViewPropertyHelper.getViewConfigs(ViewPropertyHelper.java:36)
at org.apache.ambari.view.filebrowser.FileBrowserService.getViewConfigs(FileBrowserService.java:52)
at org.apache.ambari.view.filebrowser.FileBrowserService.help(FileBrowserService.java:80)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.sun.jersey.server.impl.uri.rules.SubLocatorRule.dispatch(SubLocatorRule.java:190)
at com.sun.jersey.server.impl.uri.rules.SubLocatorRule.invokeSubLocator(SubLocatorRule.java:183)
at com.sun.jersey.server.impl.uri.rules.SubLocatorRule.accept(SubLocatorRule.java:110)
at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
at com.sun.jersey.server.impl.uri.rules.SubLocatorRule.accept(SubLocatorRule.java:137)
at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
at com.sun.jersey.server.impl.uri.rules.SubLocatorRule.accept(SubLocatorRule.java:137)
at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
at com.sun.jersey.server.impl.uri.rules.SubLocatorRule.accept(SubLocatorRule.java:137)

Caused by: org.apache.ambari.server.ClusterNotFoundException: Cluster not found, clusterId=2
at org.apache.ambari.server.state.cluster.ClustersImpl.getCluster(ClustersImpl.java:290)
at org.apache.ambari.server.view.ViewRegistry.getCluster(ViewRegistry.java:928)
... 102 more

@Geoffrey Shelton Okot

Did the restart for all components as well .

[#6]

After deleting the File view and recreating again . Clean restart now i can see the "Service checks completed."

But its showing blank page . After looking at the log i see following error

18 Aug 2017 16:02:47,190 ERROR [ambari-client-thread-29] ContainerResponse:537 - Mapped exception to response: 500 (Internal Server Error) org.apache.ambari.view.commons.exceptions.ServiceFormattedException at org.apache.ambari.view.commons.hdfs.FileOperationService.listdir(FileOperationService.java:80) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60) at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:205) at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75) at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:302) at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147) at com.sun.jersey.server.impl.uri.rules.SubLocatorRule.accept(SubLocatorRule.java:137) at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147) at com.sun.jersey.server.impl.uri.rules.SubLocatorRule.accept(SubLocatorRule.java:137) at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147) at com.sun.jersey.server.impl.uri.rules.SubLocatorRule.accept(SubLocatorRule.java:137) at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147) at com.sun.jersey.server.impl.uri.rules.SubLocatorRule.accept(SubLocatorRule.java:137) at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147) at com.sun.jersey.server.impl.uri.rules.SubLocatorRule.accept(SubLocatorRule.java:137) at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147) at com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108) at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147) at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84) at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1542) at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1473) at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1419) at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1409) at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:409) at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:558) at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:733) at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:684) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1507) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330) at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:118) at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:84) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) at org.apache.ambari.server.security.authorization.AmbariAuthorizationFilter.doFilter(AmbariAuthorizationFilter.java:287) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:113) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:103) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:113) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:54) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:45) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) at org.apache.ambari.server.security.authentication.AmbariDelegatingAuthenticationFilter.doFilter(AmbariDelegatingAuthenticationFilter.java:132) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) at org.apache.ambari.server.security.authorization.AmbariUserAuthorizationFilter.doFilter(AmbariUserAuthorizationFilter.java:91) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:87) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:192) at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:160) at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:

[#7]

Can you check the value in HDFS-->Configs-->Advanced then in the filter paste hadoop.proxyuser

Afer getting the values you will see the pattern,can you delete the one you created previously hadoop.proxyuser.hdp.groups=users

And try this

hadoop.proxyuser.root.groups=users 
hadoop.proxyuser.root.hosts=users 

Can you also run this command and paste the output in here

# cat /etc/ambari-server/conf/ambari.properties | grep ambari-server.user

The user from the above output should replace the xxx below

hadoop.proxyuser.xxx.groups=* 
hadoop.proxyuser.xxx.hosts=* 

And remember to start all the stale configs

Cheers

[#8]

@Kishore Kumar

Can you briefly descript the setup of your cluster?

Does the ambari server have the correct IP/HOST entries in the /etc/hosts

Have you disabled firewall between the cluster nodes?

How is your DNS entry in /etc/resolv.conf

Does the user hdp have a home directory in hdfs ,trun below command as user hdfs

$ hdfs dfs -ls /user

Please revert

[#9]

@Geoffrey Shelton Okot

Here is description of of my cluster :

One node is having the Ambari Server
In Cluster i have 4 nodes .

I have the entries for "nameserver" and the names to resolve .

Yeah i have the correct entries in /etc/hosts .

Regarding the Firewall : Here is the output

service ip6tables status ip6tables: Firewall is not running.

service iptables status IPv4 iptables: Firewall is not running.

When i run following command , this is the output .

# hdfs dfs -ls /user bash: hdfs: command not found

On thing to to add during the cluster setup , i have made changes to some of folder under . I hope this wount make any impact . Please let me know if need some more info

NameNode

/opt/mount1/hdp/hadoop/hdfs/namenode,/opt/mount1/hdp/home/hadoop/hdfs/namenode,/opt/hadoop/hdfs/namenode,/opt/Tidal/hadoop/hdfs/namenode,/opt/mount1/hdp/hadoop/hdfs/namenode,/opt/mount2/hadoop/hdfs/namenode,/opt/mount1/hdp/usr/hadoop/hdfs/namenode,/var/hadoop/hdfs/namenode,/var/crash/hadoop/hdfs/namenode

DataNode

/opt/mount1/hdp/hadoop/hdfs/data,/opt/mount1/hdp/home/hadoop/hdfs/data,/opt/mount1/hdp/Tidal/hadoop/hdfs/data,/opt/mount1/hadoop/hdfs/data,/opt/mount2/hadoop/hdfs/data,/opt/mount1/hdp/usr/hadoop/hdfs/data,/var/hadoop/hdfs/data,/var/crash/hadoop/hdfs/data

[#10]

@Kishore Kumar

Stop hdfs and change the below parameters

NameNode

/var/hadoop/hdfs/namenode 

DataNode

/opt/mount1/hdp/hadoop/hdfs/data,/opt/mount2/hadoop/hdfs/data 

Restart HDFS

To see the user directory while logged on as root

# su - hdfs 
$ hdfs dfs -ls /user 

You should be able to see /user/hdp the above command should work!

Let me know

[#11]

@Geoffrey Shelton Okot

That use dosnt exist in Ambari server node . Its there in rest of Cluster hosts . Here is the output.

drwxrwx--- - ambari-qa hdfs 0 2017-08-18 04:21 /user/ambari-qa

drwxr-xr-x - hcat hdfs 0 2017-08-18 04:18 /user/hcat

drwxr-xr-x - hive hdfs 0 2017-08-18 04:19 /user/hive

Error now .

Failed to transition to undefined

Server status: 500

---

Server Message:

    <small>
      User: hdp is not allowed to impersonate admin
    </small>