When the ambari-agent restarts, it loses its heartbeat

[#1]

I have successfully installed all the services,ambari-server and al of ambari-agent No error was reported. When i am trying to restart ambari-agent it is lost heartbeat. I checked Ambari-agent logs and found below

INFO 2018-12-04 10:10:42,629 NetUtil.py:62 - Connecting to https://ds-11.lanqu.com:8440/ca
ERROR 2018-12-04 10:10:42,631 NetUtil.py:88 - EOF occurred in violation of protocol (_ssl.c:579)
ERROR 2018-12-04 10:10:42,631 NetUtil.py:89 - SSLError: Failed to connect. Please check openssl library versions.
Refer to: https://bugzilla.redhat.com/show_bug.cgi?id=1022468 for more details.
WARNING 2018-12-04 10:10:42,631 NetUtil.py:116 - Server at https://ds-11.lanqu.com:8440 is not reachable, sleeping for 10 seconds...

Here is my system environment:

ENV:
OS: CentOS7.3
python: python2.7.5
amabri-server:2.4.1.0
HDP:2.5
java:1.8.0_144

All the solutions I found on the community didn't work

1.change /etc/python/cert-verification.cfg

[https]
verify=disable

2./etc/ambari-agent/conf/ambari-agent.ini

ssl_verify_cert=0
force_https_protocol=PROTOCOL_TLSv1_2

3./etc/ambari-server/conf/ambari.properties

java.home=/usr/java/jdk1.8.0_144/

4.grep -A 1 'jdk.tls.disabledAlgorithms' $JAVA_HOME/jre/lib/security/java.security

#   jdk.tls.disabledAlgorithms=MD5, SSLv3, DSA, RSA keySize < 2048
jdk.tls.disabledAlgorithms=SSLv3, RC4, MD5withRSA, DH keySize < 768, \
    EC keySize < 224

5.Execute this script and the following output

#!/usr/bin/env python
import ssl;
for i in dir(ssl):
        if i.startswith("PROTOCOL"):
                print(i)

output:

PROTOCOL_SSLv23
PROTOCOL_SSLv3
PROTOCOL_TLSv1
PROTOCOL_TLSv1_1
PROTOCOL_TLSv1_2

I have a few questions

I am used oracle jdk，But after the installation is complete,Some nodes have openjdk:

rpm -qa|grep java

java-1.8.0-openjdk-headless-1.8.0.191.b12-0.el7_5.x86_64
java-1.8.0-openjdk-1.8.0.191.b12-0.el7_5.x86_64
mysql-connector-java-5.1.25-3.el7.noarch
python-javapackages-3.4.1-11.el7.noarch
tzdata-java-2018f-2.el7.noarch
javassist-3.16.1-10.el7.noarch
javapackages-tools-3.4.1-11.el7.noarch
javamail-1.4.6-8.el7.noarch

I removed openjdk but it's not worked.

restert ambari-server and agent

So, What the hell is going on here?

[#2 · Accepted solution]

Hi @lei lin,

Can you do following steps and see if this helps you ?

cd /etc/ambari-server/conf/
cp ambari.properties ambari_backup.properties_backup
vi ambari.properties

find the line: security.server.disabled.ciphers :

comment it with #

it should look like :

#security.server.disabled.ciphers=TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384|TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384|TLS_RSA_WITH_AES_256_CBC_SHA256|TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA384|TLS_ECDH_RSA_WITH_AES_256_CBC_SHA384|TLS_DHE_RSA_WITH_AES_256_CBC_SHA256|TLS_DHE_DSS_WITH_AES_256_CBC_SHA256|TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA|TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA|TLS_RSA_WITH_AES_256_CBC_SHA|TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA|TLS_ECDH_RSA_WITH_AES_256_CBC_SHA|TLS_DHE_RSA_WITH_AES_256_CBC_SHA|TLS_DHE_DSS_WITH_AES_256_CBC_SHA|TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256|TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256|TLS_RSA_WITH_AES_128_CBC_SHA256|TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA256|TLS_ECDH_RSA_WITH_AES_128_CBC_SHA256|TLS_DHE_RSA_WITH_AES_128_CBC_SHA256|TLS_DHE_DSS_WITH_AES_128_CBC_SHA256|TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA|TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA|TLS_RSA_WITH_AES_128_CBC_SHA|TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA|TLS_ECDH_RSA_WITH_AES_128_CBC_SHA|TLS_DHE_RSA_WITH_AES_128_CBC_SHA|TLS_DHE_DSS_WITH_AES_128_CBC_SHA|TLS_ECDHE_ECDSA_WITH_3DES_EDE_CBC_SHA|TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA|TLS_ECDH_ECDSA_WITH_3DES_EDE_CBC_SHA|TLS_ECDH_RSA_WITH_3DES_EDE_CBC_SHA|SSL_DHE_RSA_WITH_3DES_EDE_CBC_SHA|SSL_DHE_DSS_WITH_3DES_EDE_CBC_SHA|TLS_EMPTY_RENEGOTIATION_INFO_SCSV|TLS_DH_anon_WITH_AES_256_CBC_SHA256|TLS_ECDH_anon_WITH_AES_256_CBC_SHA|TLS_DH_anon_WITH_AES_256_CBC_SHA|TLS_DH_anon_WITH_AES_128_CBC_SHA256|TLS_ECDH_anon_WITH_AES_128_CBC_SHA|TLS_DH_anon_WITH_AES_128_CBC_SHA|TLS_ECDH_anon_WITH_3DES_EDE_CBC_SHA|SSL_DH_anon_WITH_3DES_EDE_CBC_SHA|SSL_RSA_WITH_DES_CBC_SHA|SSL_DHE_RSA_WITH_DES_CBC_SHA|SSL_DHE_DSS_WITH_DES_CBC_SHA|SSL_DH_anon_WITH_DES_CBC_SHA|SSL_RSA_EXPORT_WITH_DES40_CBC_SHA|SSL_DHE_RSA_EXPORT_WITH_DES40_CBC_SHA|SSL_DHE_DSS_EXPORT_WITH_DES40_CBC_SHA|SSL_DH_anon_EXPORT_WITH_DES40_CBC_SHA|TLS_RSA_WITH_NULL_SHA256|TLS_ECDHE_ECDSA_WITH_NULL_SHA|TLS_ECDHE_RSA_WITH_NULL_SHA|SSL_RSA_WITH_NULL_SHA|TLS_ECDH_ECDSA_WITH_NULL_SHA|TLS_ECDH_RSA_WITH_NULL_SHA|TLS_ECDH_anon_WITH_NULL_SHA|SSL_RSA_WITH_NULL_MD5|TLS_KRB5_WITH_3DES_EDE_CBC_SHA|TLS_KRB5_WITH_3DES_EDE_CBC_MD5|TLS_KRB5_WITH_DES_CBC_SHA|TLS_KRB5_WITH_DES_CBC_MD5|TLS_KRB5_EXPORT_WITH_DES_CBC_40_SHA|TLS_KRB5_EXPORT_WITH_DES_CBC_40_MD5

restart ambari server

see if your ambari-agent is able to connect with ambari-server.

if you dont want to comment every cipher , just want to comment only the specific one.

enable -Djavax.net.debug=all to start up the Ambari Server, look out for ambari-server.out log and find the cipher ambari agent is sending and only remove that cipher from ambari.properties from disabled list.

Please accept this answer if this worked for you.

[#3]

"enable -Djavax.net.debug=all" Is it modified this file /usr/sbin/ambari_server_main.py:

jvm_args = os.getenv('AMBARI_JVM_ARGS', '-Djavax.net.debug=all -Xms512m -Xmx2048m -XX:MaxPermSize=128m')

then restart ambari server,this is ambari-server.out

Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=128m; support was removed in 8.0

Did I make a mistake

[#4]

@Jay Kumar SenSharma @scharan

Why are the installations successful and restart agent successful after registration,but after everything is installed successfully restart agent will fail.

[#5]

Thanks so much,After I comment "security.server.disabled.ciphers",ambari agent was successfully restarted.
So,I want to know how to use "-Djavax.net.debug=all" to output more information.and why please.

[#6]

Hi @lei lin,

CAn you please accept this answer if this helped you.

you need to add -Djavax.net.debug=all /var/lib/ambari-server/ambari-env.sh in export AMBARI_JVM_ARGS=

[#7]

I add -Djavax.net.debug=all /var/lib/ambari-server/ambari-env.sh in export AMBARI_JVM_ARGS=

export AMBARI_JVM_ARGS=$AMBARI_JVM_ARGS'-Xms512m -Xmx2048m -XX:MaxPermSize=128m -Djavax.net.debug=all -Djava.security.auth.login.config=$ROOT/etc/ambari-server/conf/krb5JAASLogin.conf -Djava.security.krb5.conf=/etc/krb5.conf -Djavax.security.auth.useSubjectCredsOnly=false'

Restart the ambari server,There is only some WARN in the ambari-server.log,nothing in ambari-server.out

06 Dec 2018 12:27:26,094  WARN [qtp-ambari-agent-363] nio:720 - javax.net.ssl.SSLHandshakeException: no cipher suites in common

Access the https://192.168.1.81:8440/ca through a browser(192.168.1.81 is ambari server).I use wireshark to capture package,What information can we get from these outputs? ambari server supports TLSv1 protocol...

[#8]

Hi @lei lin,

Can you please login and accept the original answer if this helps you.

and for your last query. you need to add in ambari-env.sh like this :

export AMBARI_JVM_ARGS="$AMBARI_JVM_ARGS -Xms512m -Xmx2048m -XX:MaxPermSize=128m -Djava.security.auth.login.config=$ROOT/etc/ambari-server/conf/krb5JAASLogin.conf -Djava.security.krb5.conf=/etc/krb5.conf -Djavax.security.auth.useSubjectCredsOnly=false -Djavax.net.debug=all"

and restart ambari-server .

you can find the algorithm and debug logs in ambari-server.out. it works for me and as per oracle JDK help. it should has to work for you too : https://docs.oracle.com/javase/7/docs/technotes/guides/security/jsse/ReadDebug.html

[#9]

@lei lin

Can you share agent logs ?

[#10]

What worked for me was setting the hostname of the system!!

I had installed the the HDF cluster to xyz.local.abc
but the hostname kept resetting itself after system restart

so when i set the hostname to above one I was able to start the hearbeats.