Heartbeat lost

[#1]

I have few VM in Azure cloud. Upgraded memory on all the VM's and after reboot, looks like all the cluster members are not able to talk to Ambari server. What could be wrong

p.p1 {margin: 0.0px 0.0px 0.0px 0.0px; font: 11.0px Menlo; color: #000000; background-color: #ffffff}
span.s1 {font-variant-ligatures: no-common-ligatures}

INFO 2018-07-11 01:20:43,857 NetUtil.py:70 - Connecting to https://dev1:8440/ca

ERROR 2018-07-11 01:20:43,863 NetUtil.py:96 - EOF occurred in violation of protocol (_ssl.c:579)

ERROR 2018-07-11 01:20:43,863 NetUtil.py:97 - SSLError: Failed to connect. Please check openssl library versions. 

Refer to: https://bugzilla.redhat.com/show_bug.cgi?id=1022468 for more details.

WARNING 2018-07-11 01:20:43,863 NetUtil.py:124 - Server at https://dev1:8440 is not reachable, sleeping for 10 seconds...

INFO 2018-07-11 01:20:53,864 main.py:439 - Connecting to Ambari server at https://dev1:8440 (10.0.0.11)

INFO 2018-07-11 01:20:53,864 NetUtil.py:70 - Connecting to https://dev1:8440/ca

ERROR 2018-07-11 01:20:53,869 NetUtil.py:96 - EOF occurred in violation of protocol (_ssl.c:579)

ERROR 2018-07-11 01:20:53,869 NetUtil.py:97 - SSLError: Failed to connect. Please check openssl library versions. 

Refer to: https://bugzilla.redhat.com/show_bug.cgi?id=1022468 for more details.

WARNING 2018-07-11 01:20:53,869 NetUtil.py:124 - Server at https://dev1:8440 is not reachable, sleeping for 10 seconds...

INFO 2018-07-11 01:21:03,869 NetUtil.py:70 - Connecting to https://dev1:8440/ca

ERROR 2018-07-11 01:21:03,874 NetUtil.py:96 - EOF occurred in violation of protocol (_ssl.c:579)

ERROR 2018-07-11 01:21:03,874 NetUtil.py:97 - SSLError: Failed to connect. Please check openssl library versions. 

Refer to: https://bugzilla.redhat.com/show_bug.cgi?id=1022468 for more details.

WARNING 2018-07-11 01:21:03,874 NetUtil.py:124 - Server at https://dev1:8440 is not reachable, sleeping for 10 seconds...

INFO 2018-07-11 01:21:13,875 NetUtil.py:70 - Connecting to https://dev1:8440/ca

ERROR 2018-07-11 01:21:13,880 NetUtil.py:96 - EOF occurred in violation of protocol (_ssl.c:579)

ERROR 2018-07-11 01:21:13,880 NetUtil.py:97 - SSLError: Failed to connect. Please check openssl library versions. 

Refer to: https://bugzilla.redhat.com/show_bug.cgi?id=1022468 for more details.

WARNING 2018-07-11 01:21:13,880 NetUtil.py:124 - Server at https://dev1:8440 is not reachable, sleeping for 10 seconds...

INFO 2018-07-11 01:21:23,881 NetUtil.py:70 - Connecting to https://dev1:8440/ca

ERROR 2018-07-11 01:21:23,886 NetUtil.py:96 - EOF occurred in violation of protocol (_ssl.c:579)

ERROR 2018-07-11 01:21:23,886 NetUtil.py:97 - SSLError: Failed to connect. Please check openssl library versions. 

Refer to: https://bugzilla.redhat.com/show_bug.cgi?id=1022468 for more details.

WARNING 2018-07-11 01:21:23,886 NetUtil.py:124 - Server at https://dev1:8440 is not reachable, sleeping for 10 seconds.


Refer to: https://bugzilla.redhat.com/show_bug.cgi?id=1022468 for more details.

WARNING 2018-07-11 01:17:13,729 NetUtil.py:124 - Server at https://dev1:8440 is not reachable, sleeping for 10 seconds...

INFO 2018-07-11 01:17:23,730 NetUtil.py:70 - Connecting to https://dev1:8440/ca

ERROR 2018-07-11 01:17:23,734 NetUtil.py:96 - EOF occurred in violation of protocol (_ssl.c:579)

ERROR 2018-07-11 01:17:23,734 NetUtil.py:97 - SSLError: Failed to connect. Please check openssl library versions. 

Refer to: https://bugzilla.redhat.com/show_bug.cgi?id=1022468 for more details.

WARNING 2018-07-11 01:17:23,734 NetUtil.py:124 - Server at https://dev1:8440 is not reachable, sleeping for 10 seconds..

[#2 · Accepted solution]

@Prakash Punj

Please refer to the following article to know the cause and exact resolution for this error which is causing ambari agent to not come up properly:

ERROR 2018-07-11 01:20:43,863 NetUtil.py:96 - EOF occurred in violation of protocol (_ssl.c:579)

https://community.hortonworks.com/articles/188269/javapython-updates-and-ambari-agent-tls-settings.h...

.

[#3]

Thank you @Jay Kumar SenSharma It did fixed the problem but wondering why it was working before and stopped working after a VM restart

[#4]

@Prakash Punj

Was there any yum update command triggered or any rpm update which might have caused upgrade of pythin / jdk binaries?

may be looking at the /var/log/yum.log or installation/update history can help in understanding what changed recently on the vm.