question Re: Service 'hdfs' check failed-From Ambari in Archives of Support Questions (Read Only)

Service 'hdfs' check failed-From Ambari

kishoretiwari — Fri, 18 Aug 2017 12:28:06 GMT

Hi Team ,

After installing HDP using the Ambari , from Menu if I click on "File View" .

It get the " Issues detected Service 'hdfs' check failed" . With following exception .

java.lang.NullPointerException
	at org.apache.ambari.server.view.ClusterImpl.getConfigByType(ClusterImpl.java:71)
	at org.apache.ambari.view.utils.hdfs.ConfigurationBuilder.copyPropertiesBySite(ConfigurationBuilder.java:203)
	at org.apache.ambari.view.utils.hdfs.ConfigurationBuilder.buildConfig(ConfigurationBuilder.java:311)
	at org.apache.ambari.view.utils.hdfs.HdfsApi.<init>(HdfsApi.java:68)
	at org.apache.ambari.view.utils.hdfs.HdfsUtil.getHdfsApi(HdfsUtil.java:150)
	at org.apache.ambari.view.utils.hdfs.HdfsUtil.connectToHDFSApi(HdfsUtil.java:138)
	at org.apache.ambari.view.commons.hdfs.HdfsService.hdfsSmokeTest(HdfsService.java:145)
	at org.apache.ambari.view.filebrowser.HelpService.hdfsStatus(HelpService.java:95)

Re: Service 'hdfs' check failed-From Ambari

jsensharma — Fri, 18 Aug 2017 12:55:26 GMT

@Kishore Kumar

Have you tried restarting ambari once and then tried accessing the view again? If not then please try restarting ambari once.

- If it still does not fix the issue then please provide us the following info:

1. Which version of ambari is it?

2. Is this the default File View instance or you created one. If custom File View instance then please share the details of the view configuration.

3. By any chance do you notice any additional WARN / ERROR in ambari server log?

4. Your HDFS service checks are running fine?

Re: Service 'hdfs' check failed-From Ambari

kishoretiwari — Fri, 18 Aug 2017 13:13:14 GMT

Yes I did restart , it remains same .

1. Which version of ambari is it? : Version2.5.1.0

2. Is this the default File View instance or you created one. If custom File View instance then please share the details of the view configuration. : Its default .

3. By any chance do you notice any additional WARN / ERROR in ambari server log?

Error in log :

18 Aug 2017 06:03:58,495 ERROR [ambari-client-thread-28] ContainerResponse:419 - The RuntimeException could not be mapped to a response, re-throwing to the HTTP container
org.apache.ambari.server.view.IllegalClusterException: Failed to get cluster information associated with this view instance
at org.apache.ambari.server.view.ViewRegistry.getCluster(ViewRegistry.java:931)
at org.apache.ambari.server.view.ViewContextImpl.getCluster(ViewContextImpl.java:370)
at org.apache.ambari.server.view.ViewContextImpl.getPropertyValues(ViewContextImpl.java:437)
at org.apache.ambari.server.view.ViewContextImpl.getProperties(ViewContextImpl.java:171)
at org.apache.ambari.view.commons.hdfs.ViewPropertyHelper.getViewConfigs(ViewPropertyHelper.java:36)
at org.apache.ambari.view.filebrowser.FileBrowserService.getViewConfigs(FileBrowserService.java:52)
at org.apache.ambari.view.filebrowser.FileBrowserService.help(FileBrowserService.java:80)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)

Re: Service 'hdfs' check failed-From Ambari

kishoretiwari — Fri, 18 Aug 2017 16:31:04 GMT

hi Team ,

any update on this ?

Best regards

~Kishore

Re: Service 'hdfs' check failed-From Ambari

Shelton — Fri, 18 Aug 2017 16:36:13 GMT

@Kishore Kumar

Have you set proxyuser in the Ambari UI-->HDFS --> Configs--> Advanced

hadoop.proxyuser.root.groups=*
hadoop.proxyuser.root.hosts=*

if the ambari is being run as root

Re: Service 'hdfs' check failed-From Ambari

jsensharma — Fri, 18 Aug 2017 16:42:15 GMT

@Kishore Kumar

The following error does not looks good:

18 Aug 2017 06:03:58,495 ERROR [ambari-client-thread-28] ContainerResponse:419 - The RuntimeException could not be mapped to a response, re-throwing to the HTTP container
org.apache.ambari.server.view.IllegalClusterException: Failed to get cluster information associated with this view instance

I will suggest you to try creating a new Instance of File View and then test again.

Ambari UI --> admin (Drop Down) --> Manage Ambari --> Views --> FILES --> Create Instance (Button)

Specify the "Instance Name*, Display Name* and Description*" and then try accessing that view to see if that works.

Re: Service 'hdfs' check failed-From Ambari

kishoretiwari — Fri, 18 Aug 2017 16:56:46 GMT

Not its not running as root .

Re: Service 'hdfs' check failed-From Ambari

kishoretiwari — Fri, 18 Aug 2017 16:57:58 GMT

One thing to add on , the node where my Ambari sever is running is not a part of cluster . Will that lead this issue ?

Re: Service 'hdfs' check failed-From Ambari

kishoretiwari — Fri, 18 Aug 2017 16:58:11 GMT

One thing to add on , the node where my Ambari sever is running is not a part of cluster . Will that lead this issue ?

Re: Service 'hdfs' check failed-From Ambari

jsensharma — Fri, 18 Aug 2017 17:01:43 GMT

@Kishore Kumar

"root" need to be replaced with the user who is running the ambari server process. For example if you are running ambari server as user "abcd" then the property needs to be set in the Ambari UI-->HDFS --> Configs--> Advanced

hadoop.proxyuser.abcd.groups=*
hadoop.proxyuser.abcd.hosts=*

- Regarding your query: "the node where my Ambari sever is running is not a part of cluster . Will that lead this issue ?"

>>> This is nornal scenario, where Ambari is not part of cluster. So it should not cause issue. File View uses "webhdfs" APIs to query hdfs.

Is your environment Kerberized? If yes, then you should refer to : https://docs.hortonworks.com/HDPDocuments/Ambari-2.5.1.0/bk_ambari-views/content/Troubleshooting.html

- Have you tried creating a new File View instance as mentioned in the previous comment ?

Re: Service 'hdfs' check failed-From Ambari

jsensharma — Fri, 18 Aug 2017 17:07:05 GMT

@Kishore Kumar

If it still does not work then try the following :

1. Remove the following directory (File View "FILES{1.0.0}" on ambari server. (It will be recreated on next ambari restart)

# rm -rf /var/lib/ambari-server/resources/views/work/FILES\{1.0.0\}/

2. Restart ambari server

# ambari-server restart

3. Try accessing the view again.

Re: Service 'hdfs' check failed-From Ambari

kishoretiwari — Fri, 18 Aug 2017 17:37:30 GMT

Still same . Here is the error

18 Aug 2017 10:34:29,880 WARN [ambari-client-thread-29] ServletHandler:561 - Error Processing URI: /api/v1/views/FILES/versions/1.0.0/instances/AUTO_FILES_INSTANCE/resources/files/user/home - (org.apache.ambari.server.view.IllegalClusterException) Failed to get cluster information associated with this view instance
18 Aug 2017 10:34:30,219 ERROR [ambari-client-thread-25] ViewRegistry:930 - Could not find the cluster identified by 2.
18 Aug 2017 10:34:30,221 ERROR [ambari-client-thread-25] ContainerResponse:419 - The RuntimeException could not be mapped to a response, re-throwing to the HTTP container
org.apache.ambari.server.view.IllegalClusterException: Failed to get cluster information associated with this view instance
at org.apache.ambari.server.view.ViewRegistry.getCluster(ViewRegistry.java:931)
at org.apache.ambari.server.view.ViewContextImpl.getCluster(ViewContextImpl.java:370)
at org.apache.ambari.server.view.ViewContextImpl.getPropertyValues(ViewContextImpl.java:437)
at org.apache.ambari.server.view.ViewContextImpl.getProperties(ViewContextImpl.java:171)
at org.apache.ambari.view.commons.hdfs.ViewPropertyHelper.getViewConfigs(ViewPropertyHelper.java:36)
at org.apache.ambari.view.filebrowser.FileBrowserService.getViewConfigs(FileBrowserService.java:52)
at org.apache.ambari.view.filebrowser.FileBrowserService.help(FileBrowserService.java:80)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)

Caused by: org.apache.ambari.server.ClusterNotFoundException: Cluster not found, clusterId=2
at org.apache.ambari.server.state.cluster.ClustersImpl.getCluster(ClustersImpl.java:290)
at org.apache.ambari.server.view.ViewRegistry.getCluster(ViewRegistry.java:928)
... 102 more
18 Aug 2017 10:34:30,231 WARN [ambari-client-thread-25] ServletHandler:561 - Error Processing URI: /api/v1/views/FILES/versions/1.0.0/instances/AUTO_FILES_INSTANCE/resources/files/help/hdfsStatus - (org.apache.ambari.server.view.IllegalClusterException) Failed to get cluster information associated with this view instance

Re: Service 'hdfs' check failed-From Ambari

Shelton — Fri, 18 Aug 2017 17:39:45 GMT

@Kishore Kumar

Then substitute the root with that user !

Re: Service 'hdfs' check failed-From Ambari

Shelton — Fri, 18 Aug 2017 17:46:49 GMT

@@Kishore Kumar

Did you rename or delete the cluster.

Re: Service 'hdfs' check failed-From Ambari

jsensharma — Fri, 18 Aug 2017 17:49:03 GMT

@Kishore Kumar

Please try creating a new Instance of File View and then test again. (Login to ambari UI as ambari admin and then following the below path)

Ambari UI --> admin (DropDown)-->ManageAmbari-->Views--> FILES -->Create Instance(Button)<br>

Specify the "Instance Name*, Display Name* and Description*" and then save it .. later try accessing that view to see if that works.

Re: Service 'hdfs' check failed-From Ambari

kishoretiwari — Fri, 18 Aug 2017 18:04:36 GMT

No I didn't .

Re: Service 'hdfs' check failed-From Ambari

kishoretiwari — Fri, 18 Aug 2017 18:09:25 GMT

Still same error

on associated with this view instance
18 Aug 2017 11:08:08,969 ERROR [ambari-client-thread-26] ViewRegistry:930 - Could not find the cluster identified by 2.
18 Aug 2017 11:08:08,970 ERROR [ambari-client-thread-26] ContainerResponse:419 - The RuntimeException could not be mapped to a response, re-throwing to the HTTP container
org.apache.ambari.server.view.IllegalClusterException: Failed to get cluster information associated with this view instance
at org.apache.ambari.server.view.ViewRegistry.getCluster(ViewRegistry.java:931)
at org.apache.ambari.server.view.ViewContextImpl.getCluster(ViewContextImpl.java:370)
at org.apache.ambari.server.view.ViewContextImpl.getPropertyValues(ViewContextImpl.java:437)
at org.apache.ambari.server.view.ViewContextImpl.getProperties(ViewContextImpl.java:171)
at org.apache.ambari.view.commons.hdfs.ViewPropertyHelper.getViewConfigs(ViewPropertyHelper.java:36)
at org.apache.ambari.view.filebrowser.FileBrowserService.getViewConfigs(FileBrowserService.java:52)
at org.apache.ambari.view.filebrowser.FileBrowserService.help(FileBrowserService.java:80)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.sun.jersey.server.impl.uri.rules.SubLocatorRule.dispatch(SubLocatorRule.java:190)
at com.sun.jersey.server.impl.uri.rules.SubLocatorRule.invokeSubLocator(SubLocatorRule.java:183)
at com.sun.jersey.server.impl.uri.rules.SubLocatorRule.accept(SubLocatorRule.java:110)
at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
at com.sun.jersey.server.impl.uri.rules.SubLocatorRule.accept(SubLocatorRule.java:137)

Caused by: org.apache.ambari.server.ClusterNotFoundException: Cluster not found, clusterId=2
at org.apache.ambari.server.state.cluster.ClustersImpl.getCluster(ClustersImpl.java:290)
at org.apache.ambari.server.view.ViewRegistry.getCluster(ViewRegistry.java:928)
... 102 more
18 Aug 2017 11:08:08,972 WARN [ambari-client-thread-26] ServletHandler:561 - Error Processing URI: /api/v1/views/FILES/versions/1.0.0/instances/AUTO_FILES_INSTANCE/resources/files/help/hdfsStatus - (org.apache.ambari.server.view.IllegalClusterException) Failed to get cluster information associated with this view instance

Re: Service 'hdfs' check failed-From Ambari

kishoretiwari — Fri, 18 Aug 2017 18:10:43 GMT

At UI this is the error message

Service checks completed.

HDFS test

Issues detected

Service 'hdfs' check failed:Server Error

Re: Service 'hdfs' check failed-From Ambari

Shelton — Fri, 18 Aug 2017 18:19:03 GMT

@Kishore Kumar

can you restart the agents and retry these errors may go away?

Re: Service 'hdfs' check failed-From Ambari

dheer_vijji_rag — Fri, 18 Aug 2017 20:35:04 GMT

@Kishore Kumar

As given by @Jay SenSharma and @Geoffrey Shelton Okot the error:

18 Aug 2017 11:08:08,969 ERROR [ambari-client-thread-26] ViewRegistry:930 - Could not find the cluster identified by 2.
18 Aug 2017 11:08:08,970 ERROR [ambari-client-thread-26] ContainerResponse:419 - The RuntimeException could not be mapped to a response, re-throwing to the HTTP container
org.apache.ambari.server.view.IllegalClusterException: Failed to get cluster information associated with this view instance

doesn't look good, can you please follow the official documentation to create the new files view: https://docs.hortonworks.com/HDPDocuments/Ambari-2.5.1.0/bk_ambari-views/content/configuring_your_cluster_for_files_view.html

And also make sure you are able to reach the cluster from the Ambari host, and the forward and reserve lookup of the cluster (NameNodes) from the Ambari server is going through.