Ambari 2.1.2 null pointer exception after deleting a node - Ambari completely unusable !!

[#1]

Hi,

wired situation...while deleting a node from HDP-cluster I received a popup from Ambari with a very long error message.

But the node has been deleted, at least it disappeared from the node list.

Unfortunately now Ambari (MySQL as backend) is no longer working, it is hanging in the "Loading" state and nothing happens further.

Here is an excerpt from Ambari-server log of the time of deleting the node:

28 Jan 2016 20:14:53,448  INFO [qtp-client-1030] ServiceComponentImpl:626 - Deleting servicecomponenthost for cluster, clusterName=hdpprd, serviceName=SPARK, componentName=SPARK_CLIENT, hostname=b0d0b7k2.com
28 Jan 2016 20:14:53,488  INFO [qtp-client-1030] ServiceComponentImpl:626 - Deleting servicecomponenthost for cluster, clusterName=hdpprd, serviceName=SQOOP, componentName=SQOOP, hostname=b0d0b7k2.com
28 Jan 2016 20:14:53,520  INFO [qtp-client-1030] ServiceComponentImpl:626 - Deleting servicecomponenthost for cluster, clusterName=hdpprd, serviceName=TEZ, componentName=TEZ_CLIENT, hostname=b0d0b7k2.com
28 Jan 2016 20:14:53,840  INFO [qtp-client-1030] ServiceComponentImpl:626 - Deleting servicecomponenthost for cluster, clusterName=hdpprd, serviceName=ZOOKEEPER, componentName=ZOOKEEPER_CLIENT, hostname=b0d0b7k2.com
28 Jan 2016 20:14:53,879  INFO [qtp-client-1030] ServiceComponentImpl:626 - Deleting servicecomponenthost for cluster, clusterName=hdpprd, serviceName=PIG, componentName=PIG, hostname=b0d0b7k2.com
28 Jan 2016 20:14:56,916  INFO [qtp-client-1030] ServiceComponentImpl:626 - Deleting servicecomponenthost for cluster, clusterName=hdpprd, serviceName=YARN, componentName=YARN_CLIENT, hostname=b0d0b7k2.com
28 Jan 2016 20:14:56,920  WARN [qtp-client-1030] ServletHandler:563 - /api/v1/clusters/hdpprd/hosts/b0d0b7k2.com/host_components/YARN_CLIENT
java.lang.NullPointerException
        at org.apache.ambari.server.state.svccomphost.ServiceComponentHostImpl.getStateEntity(ServiceComponentHostImpl.java:1644)
        at org.apache.ambari.server.state.svccomphost.ServiceComponentHostImpl.removeEntities(ServiceComponentHostImpl.java:1411)
        at org.apache.ambari.server.orm.AmbariJpaLocalTxnInterceptor.invoke(AmbariJpaLocalTxnInterceptor.java:68)

...

Here is the current log output from ambari-server after I tried to login (this messages are repeating):

28 Jan 2016 20:14:57,381  WARN [qtp-client-1030] ServletHandler:563 - /api/v1/clusters/hdpprd/hosts
java.lang.NullPointerException
        at org.apache.ambari.server.state.svccomphost.ServiceComponentHostImpl.getStateEntity(ServiceComponentHostImpl.java:1644)

...

28 Jan 2016 20:14:57,413 ERROR [qtp-client-1030] ReadHandler:91 - Caught a runtime exception executing a query
java.lang.NullPointerException
        at org.apache.ambari.server.state.svccomphost.ServiceComponentHostImpl.getStateEntity(ServiceComponentHostImpl.java:1644)

...

28 Jan 2016 20:14:57,900  WARN [qtp-client-1030] ServletHandler:563 - /api/v1/clusters
java.lang.NullPointerException28 Jan 2016 20:14:59,310 ERROR [qtp-client-1030] ReadHandler:91 - Caught a runtime exception executing a query
java.lang.NullPointerException
        at org.apache.ambari.server.state.svccomphost.ServiceComponentHostImpl.getStateEntity(ServiceComponentHostImpl.java:1644)

...

28 Jan 2016 20:15:18,873 ERROR [qtp-client-1030] ReadHandler:91 - Caught a runtime exception executing a query
java.lang.NullPointerException
28 Jan 2016 20:15:18,874  WARN [qtp-client-1030] ServletHandler:563 - /api/v1/clusters/hdpprd/
java.lang.NullPointerException
28 Jan 2016 20:15:19,029 ERROR [qtp-client-1030] ReadHandler:91 - Caught a runtime exception executing a query
java.lang.NullPointerException
28 Jan 2016 20:15:19,030  WARN [qtp-client-1030] ServletHandler:563 - /api/v1/clusters/hdpprd/hosts/b0d02ef2.ldc.uk.zurich.com
java.lang.NullPointerException
28 Jan 2016 20:15:19,054 ERROR [qtp-client-1030] ReadHandler:91 - Caught a runtime exception executing a query
java.lang.NullPointerException
28 Jan 2016 20:15:19,056  WARN [qtp-client-1030] ServletHandler:563 - /api/v1/clusters/hdpprd
java.lang.NullPointerException
28 Jan 2016 20:15:19,079 ERROR [qtp-client-1030] ReadHandler:91 - Caught a runtime exception executing a query
java.lang.NullPointerException
28 Jan 2016 20:15:19,081  WARN [qtp-client-1030] ServletHandler:563 - /api/v1/clusters/hdpprd
java.lang.NullPointerException
28 Jan 2016 20:15:19,090 ERROR [qtp-client-1030] ReadHandler:91 - Caught a runtime exception executing a query
java.lang.NullPointerException
28 Jan 2016 20:15:19,092  WARN [qtp-client-1030] ServletHandler:563 - /api/v1/clusters/hdpprd/hosts
java.lang.NullPointerException
28 Jan 2016 20:15:19,279 ERROR [qtp-client-1030] ReadHandler:91 - Caught a runtime exception executing a query
java.lang.NullPointerException
28 Jan 2016 20:15:19,280  WARN [qtp-client-1030] ServletHandler:563 - /api/v1/clusters/hdpprd/hosts
java.lang.NullPointerException
28 Jan 2016 20:15:19,644 ERROR [qtp-client-1030] ReadHandler:91 - Caught a runtime exception executing a query
java.lang.NullPointerException

Is there something to check directly in MySQL db?!?!

Any hints highly appreciated, Ambari is highly requested to proceed working on that cluster 😉

Thanks in advance...

[#2]

This is the error after providing credentials to login

[#3 · Accepted solution]

@Gerd Koenig

You should open support ticket.

Restart ambari server and check if it helps.

[#4]

Hi @Neeraj Sabharwal ,

thanks...a restart brought back Ambari into life.....right after opening a ticket 😉