Service 'hdfs' check failed-From Ambari

[#1]

Hi Team ,

After installing HDP using the Ambari , from Menu if I click on "File View" .

It get the " Issues detected Service 'hdfs' check failed" . With following exception .

java.lang.NullPointerException
	at org.apache.ambari.server.view.ClusterImpl.getConfigByType(ClusterImpl.java:71)
	at org.apache.ambari.view.utils.hdfs.ConfigurationBuilder.copyPropertiesBySite(ConfigurationBuilder.java:203)
	at org.apache.ambari.view.utils.hdfs.ConfigurationBuilder.buildConfig(ConfigurationBuilder.java:311)
	at org.apache.ambari.view.utils.hdfs.HdfsApi.<init>(HdfsApi.java:68)
	at org.apache.ambari.view.utils.hdfs.HdfsUtil.getHdfsApi(HdfsUtil.java:150)
	at org.apache.ambari.view.utils.hdfs.HdfsUtil.connectToHDFSApi(HdfsUtil.java:138)
	at org.apache.ambari.view.commons.hdfs.HdfsService.hdfsSmokeTest(HdfsService.java:145)
	at org.apache.ambari.view.filebrowser.HelpService.hdfsStatus(HelpService.java:95)

[#2]

@ansharma1 

You can run the following query in Ambari DB

SELECT view_instance_id,resource_id,view_name, cluster_handle,cluster_type FROM viewinstance;

Above query will show that the view which is causing the problem might not be associated with any cluster_handle. (cluster_handle is basically the cluster_id, which you can see in the clusters table).  If cluster_handle for a view is not correctly updated then you might see that kind of message:

org.apache.ambari.server.view.IllegalClusterException: Failed to get cluster information associated with this view instance

If you want to use the same old View to work fine  (instead of creating a new Instance of that view) then you might have to make sure to update the cluster_handle for that view instance is set correctly.

Like

1. Take ambari DB dump (latest dump for backup), As we are going to change the DB manually.

2. Stop ambari-server

3. Run the following queries in the amabri DB. NOTE: Following is just a dummy query the values for 'cluster_handle' and 'view_instance_id' in that query may vary.

UPDATE viewinstance SET cluster_handle = 4 WHERE view_instance_id=3;

[#3]

One thing to add on , the node where my Ambari sever is running is not a part of cluster . Will that lead this issue ?

[#4]

@Kishore Kumar

"root" need to be replaced with the user who is running the ambari server process. For example if you are running ambari server as user "abcd" then the property needs to be set in the Ambari UI-->HDFS --> Configs--> Advanced

hadoop.proxyuser.abcd.groups=*
hadoop.proxyuser.abcd.hosts=*

.

- Regarding your query: "the node where my Ambari sever is running is not a part of cluster . Will that lead this issue ?"

>>> This is nornal scenario, where Ambari is not part of cluster. So it should not cause issue. File View uses "webhdfs" APIs to query hdfs.

.

Is your environment Kerberized? If yes, then you should refer to : https://docs.hortonworks.com/HDPDocuments/Ambari-2.5.1.0/bk_ambari-views/content/Troubleshooting.htm...

- Have you tried creating a new File View instance as mentioned in the previous comment ?

[#5]

@Kishore Kumar

If it still does not work then try the following :

1. Remove the following directory (File View "FILES{1.0.0}" on ambari server. (It will be recreated on next ambari restart)

# rm -rf /var/lib/ambari-server/resources/views/work/FILES\{1.0.0\}/

.

2. Restart ambari server

# ambari-server restart

.

3. Try accessing the view again.

[#6]

Still same . Here is the error

18 Aug 2017 10:34:29,880 WARN [ambari-client-thread-29] ServletHandler:561 - Error Processing URI: /api/v1/views/FILES/versions/1.0.0/instances/AUTO_FILES_INSTANCE/resources/files/user/home - (org.apache.ambari.server.view.IllegalClusterException) Failed to get cluster information associated with this view instance
18 Aug 2017 10:34:30,219 ERROR [ambari-client-thread-25] ViewRegistry:930 - Could not find the cluster identified by 2.
18 Aug 2017 10:34:30,221 ERROR [ambari-client-thread-25] ContainerResponse:419 - The RuntimeException could not be mapped to a response, re-throwing to the HTTP container
org.apache.ambari.server.view.IllegalClusterException: Failed to get cluster information associated with this view instance
at org.apache.ambari.server.view.ViewRegistry.getCluster(ViewRegistry.java:931)
at org.apache.ambari.server.view.ViewContextImpl.getCluster(ViewContextImpl.java:370)
at org.apache.ambari.server.view.ViewContextImpl.getPropertyValues(ViewContextImpl.java:437)
at org.apache.ambari.server.view.ViewContextImpl.getProperties(ViewContextImpl.java:171)
at org.apache.ambari.view.commons.hdfs.ViewPropertyHelper.getViewConfigs(ViewPropertyHelper.java:36)
at org.apache.ambari.view.filebrowser.FileBrowserService.getViewConfigs(FileBrowserService.java:52)
at org.apache.ambari.view.filebrowser.FileBrowserService.help(FileBrowserService.java:80)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)

Caused by: org.apache.ambari.server.ClusterNotFoundException: Cluster not found, clusterId=2
at org.apache.ambari.server.state.cluster.ClustersImpl.getCluster(ClustersImpl.java:290)
at org.apache.ambari.server.view.ViewRegistry.getCluster(ViewRegistry.java:928)
... 102 more
18 Aug 2017 10:34:30,231 WARN [ambari-client-thread-25] ServletHandler:561 - Error Processing URI: /api/v1/views/FILES/versions/1.0.0/instances/AUTO_FILES_INSTANCE/resources/files/help/hdfsStatus - (org.apache.ambari.server.view.IllegalClusterException) Failed to get cluster information associated with this view instance

[#7]

At UI this is the error message

Service checks completed.

HDFS test

Issues detected

Service 'hdfs' check failed:Server Error

[#8]

@@Kishore Kumar

Did you rename or delete the cluster.

[#9]

No I didn't .

[#10]

@Kishore Kumar

Please try creating a new Instance of File View and then test again. (Login to ambari UI as ambari admin and then following the below path)

Ambari UI --> admin (DropDown)-->ManageAmbari-->Views--> FILES -->Create Instance(Button)<br>

Specify the "Instance Name*, Display Name* and Description*" and then save it .. later try accessing that view to see if that works.

[#11]

Still same error

on associated with this view instance
18 Aug 2017 11:08:08,969 ERROR [ambari-client-thread-26] ViewRegistry:930 - Could not find the cluster identified by 2.
18 Aug 2017 11:08:08,970 ERROR [ambari-client-thread-26] ContainerResponse:419 - The RuntimeException could not be mapped to a response, re-throwing to the HTTP container
org.apache.ambari.server.view.IllegalClusterException: Failed to get cluster information associated with this view instance
at org.apache.ambari.server.view.ViewRegistry.getCluster(ViewRegistry.java:931)
at org.apache.ambari.server.view.ViewContextImpl.getCluster(ViewContextImpl.java:370)
at org.apache.ambari.server.view.ViewContextImpl.getPropertyValues(ViewContextImpl.java:437)
at org.apache.ambari.server.view.ViewContextImpl.getProperties(ViewContextImpl.java:171)
at org.apache.ambari.view.commons.hdfs.ViewPropertyHelper.getViewConfigs(ViewPropertyHelper.java:36)
at org.apache.ambari.view.filebrowser.FileBrowserService.getViewConfigs(FileBrowserService.java:52)
at org.apache.ambari.view.filebrowser.FileBrowserService.help(FileBrowserService.java:80)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.sun.jersey.server.impl.uri.rules.SubLocatorRule.dispatch(SubLocatorRule.java:190)
at com.sun.jersey.server.impl.uri.rules.SubLocatorRule.invokeSubLocator(SubLocatorRule.java:183)
at com.sun.jersey.server.impl.uri.rules.SubLocatorRule.accept(SubLocatorRule.java:110)
at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
at com.sun.jersey.server.impl.uri.rules.SubLocatorRule.accept(SubLocatorRule.java:137)

Caused by: org.apache.ambari.server.ClusterNotFoundException: Cluster not found, clusterId=2
at org.apache.ambari.server.state.cluster.ClustersImpl.getCluster(ClustersImpl.java:290)
at org.apache.ambari.server.view.ViewRegistry.getCluster(ViewRegistry.java:928)
... 102 more
18 Aug 2017 11:08:08,972 WARN [ambari-client-thread-26] ServletHandler:561 - Error Processing URI: /api/v1/views/FILES/versions/1.0.0/instances/AUTO_FILES_INSTANCE/resources/files/help/hdfsStatus - (org.apache.ambari.server.view.IllegalClusterException) Failed to get cluster information associated with this view instance