Ambari UI not displaying some Widgets and Actions !!

[#1]

I Deployed a HA Kerberized HDP 2.6 Cluster, using Ambari 2.5.2.0 via its REST API, everething works fine but Ambari UI is not displaying some Widgets and Actions !! I can't either add a new component via its UI.

[#2]

@Amine Benatmane

Have you also upgraded Ambari to 2.5.2 recently?

If yes then have you performed the Ambari Post Upgrade steps.... which includes upgrade AMS i see that the AMS related widgets are not showing data:

Upgrading Ambari Metrics : https://docs.hortonworks.com/HDPDocuments/Ambari-2.5.2.0/bk_ambari-upgrade/content/upgrade_ambari_me...

Please verify the output of the following command is showing all ams binaries version also same as ambari server version:

# rpm -qa | grep ambari

.

[#3]

Hi @Jay SenSharma, No It's a fresh deployment !

[#4]

@Amine Benatmane

If it is fresh installation then please check the NameNode/DataNode logs to see if you find that the "timeline Sink" service is started properly or not? And of from those NameNode/DataNode/Yarn hosts you are able to connect to the AMS collector host port as following:

# cd /var/log/hadoop/hdfs/
# find . -name "*.*" | xargs grep -i "Sink timeline started

.

If iInk timeline is not started then you might be seeing some Connection Refused error in the component (NN/DN/RM/NM logs) logs so please verify of they have the correct AMS host port information mentioned isnide their following file:

# grep 'timeline.collector' /etc/hadoop/conf/hadoop-metrics2.properties 
# grep 'sink.timeline.port' /etc/hadoop/conf/hadoop-metrics2.properties

.

Please try to ping that host & port form the component (NN/DN/RM/NM) hosts like:

# nc -v $AMS_HOST  6188
(OR)
# telnet $AMS_HOST  6188<br>

.

Also please check and share the "/var/log/ambari-metrics-collector/ambari-metrics-collector.log" to se eif there are any WARNING / ERROR mentioned.

.

Also please check if the "/usr/lib/ambari-metrics-hadoop-sink/ambari-metrics-hadoop-sink-with-common*.jar" JAR is loaded by the components like NameNode / DataNode?

# lsof -p `cat /var/run/hadoop/hdfs/hadoop-hdfs-namenode.pid `  | grep metrics
# lsof -p `cat /var/run/hadoop/hdfs/hadoop-hdfs-datanode.pid`  | grep metrics

[#5]

@Jay SenSharma

The "timeline Sink" service is started properly and the Connection to $AMS_HOST 6188 port [tcp/*] succeeded!

No Errors in /var/log/ambari-metrics-collector/ambari-metrics-collector.log yet !

/usr/lib/ambari-metrics-hadoop-sink/ambari-metrics-hadoop-sink-with-common-2.5.2.0.298.jar is loaded correctly by NameNode but not by DataNode !

[#6]

Update:

/var/logs/ambari-server/ambari-server.log shows:

ERROR [ambari-client-thread-27] ReadHandler:102 - Caught a runtime exception executing a query java.lang.NullPointerException at org.apache.ambari.server.controller.internal.ResourceManagerHttpPropertyRequest.getConfigResourceManagerId(ResourceManagerHttpPropertyRequest.java:111) at org.apache.ambari.server.controller.internal.ResourceManagerHttpPropertyRequest.getPort(ResourceManagerHttpPropertyRequest.java:92) at org.apache.ambari.server.controller.internal.ResourceManagerHttpPropertyRequest.getUrl(ResourceManagerHttpPropertyRequest.java:77) at org.apache.ambari.server.controller.internal.HttpPropertyProvider.populateResource(HttpPropertyProvider.java:137) at org.apache.ambari.server.controller.internal.HttpPropertyProvider.populateResources(HttpPropertyProvider.java:121) at org.apache.ambari.server.controller.internal.ClusterControllerImpl.populateResources(ClusterControllerImpl.java:155)

...

31 oct. 2017 12:39:21,312 ERROR [ambari-client-thread-27] ContainerResponse:419 - The RuntimeException could not be mapped to a response, re-throwing to the HTTP container java.lang.NullPointerException at org.apache.ambari.server.controller.internal.ResourceManagerHttpPropertyRequest.getConfigResourceManagerId(ResourceManagerHttpPropertyRequest.java:111) at org.apache.ambari.server.controller.internal.ResourceManagerHttpPropertyRequest.getPort(ResourceManagerHttpPropertyRequest.java:92) at org.apache.ambari.server.controller.internal.ResourceManagerHttpPropertyRequest.getUrl(ResourceManagerHttpPropertyRequest.java:77) at org.apache.ambari.server.controller.internal.HttpPropertyProvider.populateResource(HttpPropertyProvider.java:137) at org.apache.ambari.server.controller.internal.HttpPropertyProvider.populateResources(HttpPropertyProvider.java:121) at org.apache.ambari.server.controller.internal.ClusterControllerImpl.populateResources(ClusterControllerImpl.java:155) at org.apache.ambari.server.api.query.QueryImpl.queryForSubResources(QueryImpl.java:500)

31 oct. 2017 12:39:21,315 WARN [ambari-client-thread-27] ServletHandler:561 - Error Processing URI: /api/v1/clusters/CLUSTER_NAME/components/ - (java.lang.NullPointerException) null

31 oct. 2017 12:39:21,315 WARN [ambari-client-thread-27] ServletHandler:561 - Error Processing URI: /api/v1/clusters/CLUSTER_NAME/components/ - (java.lang.NullPointerException) null

31 oct. 2017 12:39:21,461 INFO [ambari-client-thread-26] QuickLinkVisibilityControllerFactory:44 - No quick link profile is set, will display all quicklinks.

[#7 · Accepted solution]

@Amine Benatmane

I see two errors (Both are NullPointerException) in the logs: (One indicates Yarn Config issue and the other indicates indicates DB inconsistency)

1. This NullPointerException is coming from this API:

https://github.com/apache/ambari/blob/release-2.5.2/ambari-server/src/main/java/org/apache/ambari/se...

Which can happen only if somehow the "hostNameByID" is incorrect or the "hostname" variable is Null. OR if the "yarn.resourcemanager.ha.rm-ids" is not configured properly inside your "yarn-site.xml"

31 oct. 2017 12:39:21,312 ERROR [ambari-client-thread-27] ContainerResponse:419 - The RuntimeException could not be mapped to a response, re-throwing to the HTTP container java.lang.NullPointerException 
     at org.apache.ambari.server.controller.internal.ResourceManagerHttpPropertyRequest.getConfigResourceManagerId(ResourceManagerHttpPropertyRequest.java:111)

.

2. Following "NullPointerException" is more problematic as it indicates a DB Issue (Inconsistency). Which will require a complete review of the DB dump to findout why the conponents API is throwing Null.

31 oct. 2017 12:39:21,315 WARN [ambari-client-thread-27] ServletHandler:561 - Error Processing URI: 
/api/v1/clusters/CLUSTER_NAME/components/ - (java.lang.NullPointerException) null

.

[#8]

@Jay Kumar SenSharma

yarn.resourcemanager.hostname.[RM_ID] was missing in yarn-site config, thanks a lot !