Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

When the ambari-agent restarts, it loses its heartbeat

avatar
Explorer

I have successfully installed all the services,ambari-server and al of ambari-agent No error was reported. When i am trying to restart ambari-agent it is lost heartbeat. I checked Ambari-agent logs and found below

INFO 2018-12-04 10:10:42,629 NetUtil.py:62 - Connecting to https://ds-11.lanqu.com:8440/ca
ERROR 2018-12-04 10:10:42,631 NetUtil.py:88 - EOF occurred in violation of protocol (_ssl.c:579)
ERROR 2018-12-04 10:10:42,631 NetUtil.py:89 - SSLError: Failed to connect. Please check openssl library versions.
Refer to: https://bugzilla.redhat.com/show_bug.cgi?id=1022468 for more details.
WARNING 2018-12-04 10:10:42,631 NetUtil.py:116 - Server at https://ds-11.lanqu.com:8440 is not reachable, sleeping for 10 seconds...

Here is my system environment:

ENV:
OS: CentOS7.3
python: python2.7.5
amabri-server:2.4.1.0
HDP:2.5
java:1.8.0_144

All the solutions I found on the community didn't work

1.change /etc/python/cert-verification.cfg

[https]
verify=disable

2./etc/ambari-agent/conf/ambari-agent.ini

ssl_verify_cert=0
force_https_protocol=PROTOCOL_TLSv1_2

3./etc/ambari-server/conf/ambari.properties

java.home=/usr/java/jdk1.8.0_144/

4.grep -A 1 'jdk.tls.disabledAlgorithms' $JAVA_HOME/jre/lib/security/java.security

#   jdk.tls.disabledAlgorithms=MD5, SSLv3, DSA, RSA keySize < 2048
jdk.tls.disabledAlgorithms=SSLv3, RC4, MD5withRSA, DH keySize < 768, \
    EC keySize < 224

5.Execute this script and the following output

#!/usr/bin/env python
import ssl;
for i in dir(ssl):
        if i.startswith("PROTOCOL"):
                print(i)

output:

PROTOCOL_SSLv23
PROTOCOL_SSLv3
PROTOCOL_TLSv1
PROTOCOL_TLSv1_1
PROTOCOL_TLSv1_2

I have a few questions

I am used oracle jdk,But after the installation is complete,Some nodes have openjdk:

rpm -qa|grep java

java-1.8.0-openjdk-headless-1.8.0.191.b12-0.el7_5.x86_64
java-1.8.0-openjdk-1.8.0.191.b12-0.el7_5.x86_64
mysql-connector-java-5.1.25-3.el7.noarch
python-javapackages-3.4.1-11.el7.noarch
tzdata-java-2018f-2.el7.noarch
javassist-3.16.1-10.el7.noarch
javapackages-tools-3.4.1-11.el7.noarch
javamail-1.4.6-8.el7.noarch

I removed openjdk but it's not worked.

restert ambari-server and agent

So, What the hell is going on here?

1 ACCEPTED SOLUTION

avatar

Hi @lei lin,

Can you do following steps and see if this helps you ?

cd /etc/ambari-server/conf/
cp ambari.properties ambari_backup.properties_backup
vi ambari.properties

find the line: security.server.disabled.ciphers :

comment it with #

it should look like :

#security.server.disabled.ciphers=TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384|TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384|TLS_RSA_WITH_AES_256_CBC_SHA256|TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA384|TLS_ECDH_RSA_WITH_AES_256_CBC_SHA384|TLS_DHE_RSA_WITH_AES_256_CBC_SHA256|TLS_DHE_DSS_WITH_AES_256_CBC_SHA256|TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA|TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA|TLS_RSA_WITH_AES_256_CBC_SHA|TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA|TLS_ECDH_RSA_WITH_AES_256_CBC_SHA|TLS_DHE_RSA_WITH_AES_256_CBC_SHA|TLS_DHE_DSS_WITH_AES_256_CBC_SHA|TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256|TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256|TLS_RSA_WITH_AES_128_CBC_SHA256|TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA256|TLS_ECDH_RSA_WITH_AES_128_CBC_SHA256|TLS_DHE_RSA_WITH_AES_128_CBC_SHA256|TLS_DHE_DSS_WITH_AES_128_CBC_SHA256|TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA|TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA|TLS_RSA_WITH_AES_128_CBC_SHA|TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA|TLS_ECDH_RSA_WITH_AES_128_CBC_SHA|TLS_DHE_RSA_WITH_AES_128_CBC_SHA|TLS_DHE_DSS_WITH_AES_128_CBC_SHA|TLS_ECDHE_ECDSA_WITH_3DES_EDE_CBC_SHA|TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA|TLS_ECDH_ECDSA_WITH_3DES_EDE_CBC_SHA|TLS_ECDH_RSA_WITH_3DES_EDE_CBC_SHA|SSL_DHE_RSA_WITH_3DES_EDE_CBC_SHA|SSL_DHE_DSS_WITH_3DES_EDE_CBC_SHA|TLS_EMPTY_RENEGOTIATION_INFO_SCSV|TLS_DH_anon_WITH_AES_256_CBC_SHA256|TLS_ECDH_anon_WITH_AES_256_CBC_SHA|TLS_DH_anon_WITH_AES_256_CBC_SHA|TLS_DH_anon_WITH_AES_128_CBC_SHA256|TLS_ECDH_anon_WITH_AES_128_CBC_SHA|TLS_DH_anon_WITH_AES_128_CBC_SHA|TLS_ECDH_anon_WITH_3DES_EDE_CBC_SHA|SSL_DH_anon_WITH_3DES_EDE_CBC_SHA|SSL_RSA_WITH_DES_CBC_SHA|SSL_DHE_RSA_WITH_DES_CBC_SHA|SSL_DHE_DSS_WITH_DES_CBC_SHA|SSL_DH_anon_WITH_DES_CBC_SHA|SSL_RSA_EXPORT_WITH_DES40_CBC_SHA|SSL_DHE_RSA_EXPORT_WITH_DES40_CBC_SHA|SSL_DHE_DSS_EXPORT_WITH_DES40_CBC_SHA|SSL_DH_anon_EXPORT_WITH_DES40_CBC_SHA|TLS_RSA_WITH_NULL_SHA256|TLS_ECDHE_ECDSA_WITH_NULL_SHA|TLS_ECDHE_RSA_WITH_NULL_SHA|SSL_RSA_WITH_NULL_SHA|TLS_ECDH_ECDSA_WITH_NULL_SHA|TLS_ECDH_RSA_WITH_NULL_SHA|TLS_ECDH_anon_WITH_NULL_SHA|SSL_RSA_WITH_NULL_MD5|TLS_KRB5_WITH_3DES_EDE_CBC_SHA|TLS_KRB5_WITH_3DES_EDE_CBC_MD5|TLS_KRB5_WITH_DES_CBC_SHA|TLS_KRB5_WITH_DES_CBC_MD5|TLS_KRB5_EXPORT_WITH_DES_CBC_40_SHA|TLS_KRB5_EXPORT_WITH_DES_CBC_40_MD5

restart ambari server

see if your ambari-agent is able to connect with ambari-server.

if you dont want to comment every cipher , just want to comment only the specific one.

enable -Djavax.net.debug=all to start up the Ambari Server, look out for ambari-server.out log and find the cipher ambari agent is sending and only remove that cipher from ambari.properties from disabled list.

Please accept this answer if this worked for you.

View solution in original post

9 REPLIES 9

avatar

Hi @lei lin,

Can you do following steps and see if this helps you ?

cd /etc/ambari-server/conf/
cp ambari.properties ambari_backup.properties_backup
vi ambari.properties

find the line: security.server.disabled.ciphers :

comment it with #

it should look like :

#security.server.disabled.ciphers=TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384|TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384|TLS_RSA_WITH_AES_256_CBC_SHA256|TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA384|TLS_ECDH_RSA_WITH_AES_256_CBC_SHA384|TLS_DHE_RSA_WITH_AES_256_CBC_SHA256|TLS_DHE_DSS_WITH_AES_256_CBC_SHA256|TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA|TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA|TLS_RSA_WITH_AES_256_CBC_SHA|TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA|TLS_ECDH_RSA_WITH_AES_256_CBC_SHA|TLS_DHE_RSA_WITH_AES_256_CBC_SHA|TLS_DHE_DSS_WITH_AES_256_CBC_SHA|TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256|TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256|TLS_RSA_WITH_AES_128_CBC_SHA256|TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA256|TLS_ECDH_RSA_WITH_AES_128_CBC_SHA256|TLS_DHE_RSA_WITH_AES_128_CBC_SHA256|TLS_DHE_DSS_WITH_AES_128_CBC_SHA256|TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA|TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA|TLS_RSA_WITH_AES_128_CBC_SHA|TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA|TLS_ECDH_RSA_WITH_AES_128_CBC_SHA|TLS_DHE_RSA_WITH_AES_128_CBC_SHA|TLS_DHE_DSS_WITH_AES_128_CBC_SHA|TLS_ECDHE_ECDSA_WITH_3DES_EDE_CBC_SHA|TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA|TLS_ECDH_ECDSA_WITH_3DES_EDE_CBC_SHA|TLS_ECDH_RSA_WITH_3DES_EDE_CBC_SHA|SSL_DHE_RSA_WITH_3DES_EDE_CBC_SHA|SSL_DHE_DSS_WITH_3DES_EDE_CBC_SHA|TLS_EMPTY_RENEGOTIATION_INFO_SCSV|TLS_DH_anon_WITH_AES_256_CBC_SHA256|TLS_ECDH_anon_WITH_AES_256_CBC_SHA|TLS_DH_anon_WITH_AES_256_CBC_SHA|TLS_DH_anon_WITH_AES_128_CBC_SHA256|TLS_ECDH_anon_WITH_AES_128_CBC_SHA|TLS_DH_anon_WITH_AES_128_CBC_SHA|TLS_ECDH_anon_WITH_3DES_EDE_CBC_SHA|SSL_DH_anon_WITH_3DES_EDE_CBC_SHA|SSL_RSA_WITH_DES_CBC_SHA|SSL_DHE_RSA_WITH_DES_CBC_SHA|SSL_DHE_DSS_WITH_DES_CBC_SHA|SSL_DH_anon_WITH_DES_CBC_SHA|SSL_RSA_EXPORT_WITH_DES40_CBC_SHA|SSL_DHE_RSA_EXPORT_WITH_DES40_CBC_SHA|SSL_DHE_DSS_EXPORT_WITH_DES40_CBC_SHA|SSL_DH_anon_EXPORT_WITH_DES40_CBC_SHA|TLS_RSA_WITH_NULL_SHA256|TLS_ECDHE_ECDSA_WITH_NULL_SHA|TLS_ECDHE_RSA_WITH_NULL_SHA|SSL_RSA_WITH_NULL_SHA|TLS_ECDH_ECDSA_WITH_NULL_SHA|TLS_ECDH_RSA_WITH_NULL_SHA|TLS_ECDH_anon_WITH_NULL_SHA|SSL_RSA_WITH_NULL_MD5|TLS_KRB5_WITH_3DES_EDE_CBC_SHA|TLS_KRB5_WITH_3DES_EDE_CBC_MD5|TLS_KRB5_WITH_DES_CBC_SHA|TLS_KRB5_WITH_DES_CBC_MD5|TLS_KRB5_EXPORT_WITH_DES_CBC_40_SHA|TLS_KRB5_EXPORT_WITH_DES_CBC_40_MD5

restart ambari server

see if your ambari-agent is able to connect with ambari-server.

if you dont want to comment every cipher , just want to comment only the specific one.

enable -Djavax.net.debug=all to start up the Ambari Server, look out for ambari-server.out log and find the cipher ambari agent is sending and only remove that cipher from ambari.properties from disabled list.

Please accept this answer if this worked for you.

avatar
Explorer

"enable -Djavax.net.debug=all" Is it modified this file /usr/sbin/ambari_server_main.py:

jvm_args = os.getenv('AMBARI_JVM_ARGS', '-Djavax.net.debug=all -Xms512m -Xmx2048m -XX:MaxPermSize=128m')

then restart ambari server,this is ambari-server.out

Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=128m; support was removed in 8.0

Did I make a mistake

avatar
Explorer

@Jay Kumar SenSharma @scharan

Why are the installations successful and restart agent successful after registration,but after everything is installed successfully restart agent will fail.

avatar
Explorer

Thanks so much,After I comment "security.server.disabled.ciphers",ambari agent was successfully restarted.
So,I want to know how to use "-Djavax.net.debug=all" to output more information.and why please.

avatar

Hi @lei lin,

CAn you please accept this answer if this helped you.

you need to add -Djavax.net.debug=all /var/lib/ambari-server/ambari-env.sh in export AMBARI_JVM_ARGS=

avatar
Explorer

I add -Djavax.net.debug=all /var/lib/ambari-server/ambari-env.sh in export AMBARI_JVM_ARGS=

export AMBARI_JVM_ARGS=$AMBARI_JVM_ARGS'-Xms512m -Xmx2048m -XX:MaxPermSize=128m -Djavax.net.debug=all -Djava.security.auth.login.config=$ROOT/etc/ambari-server/conf/krb5JAASLogin.conf -Djava.security.krb5.conf=/etc/krb5.conf -Djavax.security.auth.useSubjectCredsOnly=false'

Restart the ambari server,There is only some WARN in the ambari-server.log,nothing in ambari-server.out

06 Dec 2018 12:27:26,094  WARN [qtp-ambari-agent-363] nio:720 - javax.net.ssl.SSLHandshakeException: no cipher suites in common

Access the https://192.168.1.81:8440/ca through a browser(192.168.1.81 is ambari server).I use wireshark to capture package,What information can we get from these outputs? ambari server supports TLSv1 protocol...

94442-15440728831.jpg

avatar

Hi @lei lin,

Can you please login and accept the original answer if this helps you.

and for your last query. you need to add in ambari-env.sh like this :

export AMBARI_JVM_ARGS="$AMBARI_JVM_ARGS -Xms512m -Xmx2048m -XX:MaxPermSize=128m -Djava.security.auth.login.config=$ROOT/etc/ambari-server/conf/krb5JAASLogin.conf -Djava.security.krb5.conf=/etc/krb5.conf -Djavax.security.auth.useSubjectCredsOnly=false -Djavax.net.debug=all"

and restart ambari-server .

you can find the algorithm and debug logs in ambari-server.out. it works for me and as per oracle JDK help. it should has to work for you too : https://docs.oracle.com/javase/7/docs/technotes/guides/security/jsse/ReadDebug.html

avatar
Master Collaborator
@lei lin

Can you share agent logs ?

avatar
Contributor

What worked for me was setting the hostname of the system!!

I had installed the the HDF cluster to xyz.local.abc
but the hostname kept resetting itself after system restart

so when i set the hostname to above one I was able to start the hearbeats.