REASON: Server not yet listening on http port 8080 after 120 seconds. Exiting. ambari server upgrade 2.7

[#1]

2018-08-02 14:32:51,917 ERROR [main] KerberosChecker:120 - Checksum failed
2018-08-02 14:32:51,918 ERROR [main] AmbariServer:1111 - Failed to run the Ambari Server
org.apache.ambari.server.AmbariException: Ambari Server Kerberos credentials check failed.
Check KDC availability and JAAS configuration in /etc/ambari-server/conf/krb5JAASLogin.conf
        at org.apache.ambari.server.controller.utilities.KerberosChecker.checkJaasConfiguration(KerberosChecker.java:121)
        at org.apache.ambari.server.controller.AmbariServer.main(AmbariServer.java:1102)

[#2 · Accepted solution]

@Sam Red

When you will run the following command then you will find a property in the output as an example "-Djava.security.krb5.conf=/etc/krb5.conf" which ambari is using.

# ps -ef | grep AmbariServer 

Please open that file which is mentioned in the property "java.security.krb5.conf" and findout what is the Address of KDC

# grep -A3 '\[realms\]' /etc/krb5.conf

In the above output whatever KDC hostname you get try to ping the KDC host & Port like following from ambari server host and verify if it is able to connect to port 88 (default port of KDC):

One Ambari Server Host

# ping $KDC_HOST
# nc -v $KDC_HOST 88

If you find any connectivity issue when running the above command from Ambari Server host then you should login in to KDC host and verify if the Firewall is disabled and Port 88 is opened or not?

.

Also find the ambari Keytab and get the principal name

# grep 'keyTab' /etc/ambari-server/conf/krb5JAASLogin.conf
# klist -kte /etc/security/keytabs/ambari.server.keytab

Now verify if ytou are able to get a valid kerberos ticket using the mentioned keytab or not? (this will help in isolating any issue with KDC or ticket retrival)

# kinit -kt /etc/security/keytabs/ambari.server.keytab $AMBARI_PRINCIPAL_FROM_ABOVE_COMMAND

On KDC Host

# netstat -tnlpa | grep 88
# service iptables stop
# service krb5kdc restart

.

[#3]

@Jay Kumar SenSharma

	This is the one error getting when generating keytabs.

KVNO Timestamp           Principal
---- ------------------- ------------------------------------------------------
   ambari-server-ker@HDP.COM   
[root@hdp /]# kinit -kt /etc/security/keytabs/ambari.server.keytab ambari-server-ker@HDP.COM
kinit: Password incorrect while getting initial credentials

[#4]

Also be sure to restrict external access to port 8080, see below:

https://community.hortonworks.com/questions/191898/hdp-261-virus-crytalminer-drwho.html

[#5]

@Jay Kumar SenSharma Never Mind. Changed password for ambari Keytab and restarted ambari. its up and running now.

Thank you.