Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Two-way SSL authentication failed when registering ambari agent failure

Two-way SSL authentication failed when registering ambari agent failure

New Contributor

Unable to register ambari-agent. I am getting two way ssl error. Here is the snip of my ambar-agent.ini file

[server]

hostname=namenode1.hadoop.com

url_port=8440

secured_url_port=8441

connect_retry_delay = 10

max_reconnect_retry_delay = 30


[security]

keysdir=/var/lib/ambari-agent/keys

server_crt=ca.crt

passphrase_env_var_name=AMBARI_PASSPHRASE

ssl_verify_cert=0

force_https_protocol=PROTOCOL_TLSv1_2



Error

ERROR 2019-03-19 00:04:10,160 security.py:87 - Two-way SSL authentication failed. Ensure that server and agent certificates were signed by the same CA and restart the agent.

In order to receive a new agent certificate, remove existing certificate file from keys directory. As a workaround you can turn off two-way SSL authentication in server configuration(ambari.properties)

Exiting..

ERROR 2019-03-19 00:04:10,161 Controller.py:212 - Unable to connect to: https://namenode1.hadoop.com:8441/agent/v1/register/namenode1.hadoop.com

Traceback (most recent call last):

File "/usr/lib/python2.6/site-packages/ambari_agent/Controller.py", line 165, in registerWithServer

ret = self.sendRequest(self.registerUrl, data)

File "/usr/lib/python2.6/site-packages/ambari_agent/Controller.py", line 496, in sendRequest

raise IOError('Request to {0} failed due to {1}'.format(url, str(exception)))

IOError: Request to https://namenode1.hadoop.com:8441/agent/v1/register/namenode1.hadoop.com failed due to EOF occurred in violation of protocol (_ssl.c:618)

ERROR 2019-03-19 00:04:10,161 Controller.py:213 - Error:Request to https://namenode1.hadoop.com:8441/agent/v1/register/namenode1.hadoop.com failed due to EOF occurred in violation of protocol (_ssl.c:618)

WARNING 2019-03-19 00:04:10,162 Controller.py:214 - Sleeping for 15 seconds and then trying again


I have tried few option and so far no luck. Any body has seen this issue?


3 REPLIES 3

Re: Two-way SSL authentication failed when registering ambari agent failure

Super Mentor

@Yas Ethio

1. Is this a fresh cluster setup ? Or the Agents were running fine earlier and you started seeing these errors recently?

2. What is the Agent version?

# rpm -qa | grep ambari

3. Have you recently upgrade3d any packages on your agent hosts ? Specially the JDK or the Python libraries? You can verify it by looking the the yum logs

# tail -100f /var/log/yum.log


4. Do you see the "3DES_EDE_CBC" string inside your "$JAVA_HOME/jre/lib/security/java.security" file? On Ambari Server Host?
And if your Operating System in Centos6 (RHEL6) then please refer to the last section in the following doc to

  • Locate the jdk.tls.disabledAlgorithms property and remove the 3DES_EDE_CBC reference

https://community.hortonworks.com/articles/188269/javapython-updates-and-ambari-agent-tls-settings.h...

.

5. Also as we see that you might have enabled the Two Way SSL on your ambari server ? Is it intentional? By default ambari server and agent communication happens on One Way SSL.

# grep 'security.server.two_way_ssl'  /etc/ambari-server/conf/ambari.properties

If you have intentionally enabled 2 way ssl then please check if your Ambari Server certificates are expired by any chance?
Following article and it's comment section will give you more idea in that regard: https://community.hortonworks.com/articles/68799/steps-to-fix-ambari-server-agent-expired-certs.html




.

Re: Two-way SSL authentication failed when registering ambari agent failure

New Contributor

@Jay Kumar SenSharma please see as follows. Thank your quick reply.

1. Ambari agent version

[root@namenode1 ~]# rpm -qa | grep ambari

ambari-agent-2.4.2.0-136.x86_64

ambari-server-2.4.2.0-136.x86_64


2. yum package update list

[root@namenode1 ~]# tail -100f /var/log/yum.log

Mar 16 10:55:04 Installed: net-tools-2.0-0.24.20131004git.el7.x86_64

Mar 16 10:59:49 Installed: 1:perl-parent-0.225-244.el7.noarch

Mar 16 10:59:49 Installed: perl-HTTP-Tiny-0.033-3.el7.noarch

Mar 16 10:59:49 Installed: perl-podlators-2.5.1-3.el7.noarch

Mar 16 10:59:49 Installed: perl-Pod-Perldoc-3.20-4.el7.noarch

Mar 16 10:59:49 Installed: 1:perl-Pod-Escapes-1.04-294.el7_6.noarch

Mar 16 10:59:49 Installed: perl-Encode-2.51-7.el7.x86_64

Mar 16 10:59:49 Installed: perl-Text-ParseWords-3.29-4.el7.noarch

Mar 16 10:59:49 Installed: perl-Pod-Usage-1.63-3.el7.noarch

Mar 16 10:59:50 Installed: 4:perl-libs-5.16.3-294.el7_6.x86_64

Mar 16 10:59:50 Installed: 4:perl-macros-5.16.3-294.el7_6.x86_64

Mar 16 10:59:50 Installed: perl-Storable-2.45-3.el7.x86_64

Mar 16 10:59:50 Installed: perl-Exporter-5.68-3.el7.noarch

Mar 16 10:59:50 Installed: perl-constant-1.27-2.el7.noarch

Mar 16 10:59:50 Installed: perl-Time-Local-1.2300-2.el7.noarch

Mar 16 10:59:50 Installed: perl-Socket-2.010-4.el7.x86_64

Mar 16 10:59:50 Installed: perl-Carp-1.26-244.el7.noarch

Mar 16 10:59:50 Installed: 4:perl-Time-HiRes-1.9725-3.el7.x86_64

Mar 16 10:59:50 Installed: perl-PathTools-3.40-5.el7.x86_64

Mar 16 10:59:50 Installed: perl-Scalar-List-Utils-1.27-248.el7.x86_64

Mar 16 10:59:50 Installed: 1:perl-Pod-Simple-3.28-4.el7.noarch

Mar 16 10:59:50 Installed: perl-File-Temp-0.23.01-3.el7.noarch

Mar 16 10:59:50 Installed: perl-File-Path-2.09-2.el7.noarch

Mar 16 10:59:50 Installed: perl-threads-shared-1.43-6.el7.x86_64

Mar 16 10:59:50 Installed: perl-threads-1.87-4.el7.x86_64

Mar 16 10:59:50 Installed: perl-Filter-1.49-3.el7.x86_64

Mar 16 10:59:50 Installed: perl-Getopt-Long-2.40-3.el7.noarch

Mar 16 10:59:53 Installed: 4:perl-5.16.3-294.el7_6.x86_64

Mar 16 10:59:53 Installed: 2:vim-filesystem-7.4.160-5.el7.x86_64

Mar 16 10:59:56 Installed: 2:vim-common-7.4.160-5.el7.x86_64

Mar 16 10:59:56 Installed: gpm-libs-1.20.7-5.el7.x86_64

Mar 16 10:59:56 Installed: 2:vim-enhanced-7.4.160-5.el7.x86_64

Mar 16 11:18:16 Installed: wget-1.14-18.el7.x86_64

Mar 16 11:18:35 Installed: apr-1.4.8-3.el7_4.1.x86_64

Mar 16 11:18:36 Installed: apr-util-1.5.2-6.el7.x86_64

Mar 16 11:18:36 Installed: httpd-tools-2.4.6-88.el7.centos.x86_64

Mar 16 11:18:36 Installed: mailcap-2.1.41-2.el7.noarch

Mar 16 11:18:37 Installed: httpd-2.4.6-88.el7.centos.x86_64

Mar 16 11:18:54 Installed: autogen-libopts-5.18-5.el7.x86_64

Mar 16 11:18:54 Installed: ntpdate-4.2.6p5-28.el7.centos.x86_64

Mar 16 11:18:55 Installed: ntp-4.2.6p5-28.el7.centos.x86_64

Mar 16 11:18:55 Installed: ntp-perl-4.2.6p5-28.el7.centos.noarch

Mar 16 11:18:55 Installed: ntp-doc-4.2.6p5-28.el7.centos.noarch

Mar 16 11:20:32 Installed: mlocate-0.26-8.el7.x86_64

Mar 18 22:16:16 Installed: python-lxml-3.2.1-4.el7.x86_64

Mar 18 22:16:16 Installed: python-javapackages-3.4.1-11.el7.noarch

Mar 18 22:16:16 Installed: javapackages-tools-3.4.1-11.el7.noarch

Mar 18 22:16:17 Installed: xml-commons-apis-1.4.01-16.el7.noarch

Mar 18 22:16:17 Installed: geronimo-jms-1.1.1-19.el7.noarch

Mar 18 22:16:17 Installed: xml-commons-resolver-1.2-15.el7.noarch

Mar 18 22:16:17 Installed: xalan-j2-2.7.1-23.el7.noarch

Mar 18 22:16:17 Installed: xerces-j2-2.11.0-17.el7_0.noarch

Mar 18 22:16:17 Installed: apache-commons-lang-2.6-15.el7.noarch

Mar 18 22:16:17 Installed: tomcat-servlet-3.0-api-7.0.76-9.el7_6.noarch

Mar 18 22:16:17 Installed: cal10n-0.7.7-4.el7.noarch

Mar 18 22:16:18 Installed: javamail-1.4.6-8.el7.noarch

Mar 18 22:16:18 Installed: log4j-1.2.17-16.el7_4.noarch

Mar 18 22:16:18 Installed: apache-commons-logging-1.1.2-7.el7.noarch

Mar 18 22:16:18 Installed: avalon-logkit-2.1-14.el7.noarch

Mar 18 22:16:18 Installed: avalon-framework-4.3-10.el7.noarch

Mar 18 22:16:18 Installed: javassist-3.16.1-10.el7.noarch

Mar 18 22:16:18 Installed: slf4j-1.7.4-4.el7_4.noarch

Mar 18 22:16:18 Installed: geronimo-jta-1.1.1-17.el7.noarch

Mar 18 22:16:18 Installed: 1:mysql-connector-java-5.1.25-3.el7.noarch

Mar 18 22:26:44 Installed: mysql-community-common-5.7.25-1.el7.x86_64

Mar 18 22:26:45 Installed: mysql-community-libs-5.7.25-1.el7.x86_64

Mar 18 22:26:50 Installed: mysql-community-client-5.7.25-1.el7.x86_64

Mar 18 22:27:24 Installed: mysql-community-server-5.7.25-1.el7.x86_64

Mar 18 22:27:24 Installed: mysql-community-libs-compat-5.7.25-1.el7.x86_64

Mar 18 22:27:24 Erased: 1:mariadb-libs-5.5.60-1.el7_5.x86_64

Mar 18 23:10:34 Installed: postgresql-libs-9.2.24-1.el7_5.x86_64

Mar 18 23:10:37 Installed: postgresql-9.2.24-1.el7_5.x86_64

Mar 18 23:10:38 Installed: postgresql-server-9.2.24-1.el7_5.x86_64

Mar 18 23:11:08 Installed: ambari-server-2.4.2.0-136.x86_64

Mar 18 23:28:05 Installed: ambari-agent-2.4.2.0-136.x86_64


3. python version

[root@namenode1 ~]# python --version

Python 2.7.5


4. Centos OS Version

[root@namenode1 ~]# cat /etc/redhat-release

CentOS Linux release 7.6.1810 (Core)


5. two SSL property has not been able in ambari-properties file.


6. Java version

[root@namenode1 ~]# java -version

java version "1.8.0_144"

Java(TM) SE Runtime Environment (build 1.8.0_144-b01)

Java HotSpot(TM) 64-Bit Server VM (build 25.144-b01, mixed mode)


Please: note this is a version vm

Re: Two-way SSL authentication failed when registering ambari agent failure

New Contributor

Thanks for the quick reply Jay Kumar SenSharma. See below answer to your question.

1. Ambari agent version

[root@namenode1 ~]# rpm -qa | grep ambari

ambari-agent-2.4.2.0-136.x86_64

ambari-server-2.4.2.0-136.x86_64


2. yum package update list

[root@namenode1 ~]# tail -100f /var/log/yum.log

Mar 16 10:55:04 Installed: net-tools-2.0-0.24.20131004git.el7.x86_64

Mar 16 10:59:49 Installed: 1:perl-parent-0.225-244.el7.noarch

Mar 16 10:59:49 Installed: perl-HTTP-Tiny-0.033-3.el7.noarch

Mar 16 10:59:49 Installed: perl-podlators-2.5.1-3.el7.noarch

Mar 16 10:59:49 Installed: perl-Pod-Perldoc-3.20-4.el7.noarch

Mar 16 10:59:49 Installed: 1:perl-Pod-Escapes-1.04-294.el7_6.noarch

Mar 16 10:59:49 Installed: perl-Encode-2.51-7.el7.x86_64

Mar 16 10:59:49 Installed: perl-Text-ParseWords-3.29-4.el7.noarch

Mar 16 10:59:49 Installed: perl-Pod-Usage-1.63-3.el7.noarch

Mar 16 10:59:50 Installed: 4:perl-libs-5.16.3-294.el7_6.x86_64

Mar 16 10:59:50 Installed: 4:perl-macros-5.16.3-294.el7_6.x86_64

Mar 16 10:59:50 Installed: perl-Storable-2.45-3.el7.x86_64

Mar 16 10:59:50 Installed: perl-Exporter-5.68-3.el7.noarch

Mar 16 10:59:50 Installed: perl-constant-1.27-2.el7.noarch

Mar 16 10:59:50 Installed: perl-Time-Local-1.2300-2.el7.noarch

Mar 16 10:59:50 Installed: perl-Socket-2.010-4.el7.x86_64

Mar 16 10:59:50 Installed: perl-Carp-1.26-244.el7.noarch

Mar 16 10:59:50 Installed: 4:perl-Time-HiRes-1.9725-3.el7.x86_64

Mar 16 10:59:50 Installed: perl-PathTools-3.40-5.el7.x86_64

Mar 16 10:59:50 Installed: perl-Scalar-List-Utils-1.27-248.el7.x86_64

Mar 16 10:59:50 Installed: 1:perl-Pod-Simple-3.28-4.el7.noarch

Mar 16 10:59:50 Installed: perl-File-Temp-0.23.01-3.el7.noarch

Mar 16 10:59:50 Installed: perl-File-Path-2.09-2.el7.noarch

Mar 16 10:59:50 Installed: perl-threads-shared-1.43-6.el7.x86_64

Mar 16 10:59:50 Installed: perl-threads-1.87-4.el7.x86_64

Mar 16 10:59:50 Installed: perl-Filter-1.49-3.el7.x86_64

Mar 16 10:59:50 Installed: perl-Getopt-Long-2.40-3.el7.noarch

Mar 16 10:59:53 Installed: 4:perl-5.16.3-294.el7_6.x86_64

Mar 16 10:59:53 Installed: 2:vim-filesystem-7.4.160-5.el7.x86_64

Mar 16 10:59:56 Installed: 2:vim-common-7.4.160-5.el7.x86_64

Mar 16 10:59:56 Installed: gpm-libs-1.20.7-5.el7.x86_64

Mar 16 10:59:56 Installed: 2:vim-enhanced-7.4.160-5.el7.x86_64

Mar 16 11:18:16 Installed: wget-1.14-18.el7.x86_64

Mar 16 11:18:35 Installed: apr-1.4.8-3.el7_4.1.x86_64

Mar 16 11:18:36 Installed: apr-util-1.5.2-6.el7.x86_64

Mar 16 11:18:36 Installed: httpd-tools-2.4.6-88.el7.centos.x86_64

Mar 16 11:18:36 Installed: mailcap-2.1.41-2.el7.noarch

Mar 16 11:18:37 Installed: httpd-2.4.6-88.el7.centos.x86_64

Mar 16 11:18:54 Installed: autogen-libopts-5.18-5.el7.x86_64

Mar 16 11:18:54 Installed: ntpdate-4.2.6p5-28.el7.centos.x86_64

Mar 16 11:18:55 Installed: ntp-4.2.6p5-28.el7.centos.x86_64

Mar 16 11:18:55 Installed: ntp-perl-4.2.6p5-28.el7.centos.noarch

Mar 16 11:18:55 Installed: ntp-doc-4.2.6p5-28.el7.centos.noarch

Mar 16 11:20:32 Installed: mlocate-0.26-8.el7.x86_64

Mar 18 22:16:16 Installed: python-lxml-3.2.1-4.el7.x86_64

Mar 18 22:16:16 Installed: python-javapackages-3.4.1-11.el7.noarch

Mar 18 22:16:16 Installed: javapackages-tools-3.4.1-11.el7.noarch

Mar 18 22:16:17 Installed: xml-commons-apis-1.4.01-16.el7.noarch

Mar 18 22:16:17 Installed: geronimo-jms-1.1.1-19.el7.noarch

Mar 18 22:16:17 Installed: xml-commons-resolver-1.2-15.el7.noarch

Mar 18 22:16:17 Installed: xalan-j2-2.7.1-23.el7.noarch

Mar 18 22:16:17 Installed: xerces-j2-2.11.0-17.el7_0.noarch

Mar 18 22:16:17 Installed: apache-commons-lang-2.6-15.el7.noarch

Mar 18 22:16:17 Installed: tomcat-servlet-3.0-api-7.0.76-9.el7_6.noarch

Mar 18 22:16:17 Installed: cal10n-0.7.7-4.el7.noarch

Mar 18 22:16:18 Installed: javamail-1.4.6-8.el7.noarch

Mar 18 22:16:18 Installed: log4j-1.2.17-16.el7_4.noarch

Mar 18 22:16:18 Installed: apache-commons-logging-1.1.2-7.el7.noarch

Mar 18 22:16:18 Installed: avalon-logkit-2.1-14.el7.noarch

Mar 18 22:16:18 Installed: avalon-framework-4.3-10.el7.noarch

Mar 18 22:16:18 Installed: javassist-3.16.1-10.el7.noarch

Mar 18 22:16:18 Installed: slf4j-1.7.4-4.el7_4.noarch

Mar 18 22:16:18 Installed: geronimo-jta-1.1.1-17.el7.noarch

Mar 18 22:16:18 Installed: 1:mysql-connector-java-5.1.25-3.el7.noarch

Mar 18 22:26:44 Installed: mysql-community-common-5.7.25-1.el7.x86_64

Mar 18 22:26:45 Installed: mysql-community-libs-5.7.25-1.el7.x86_64

Mar 18 22:26:50 Installed: mysql-community-client-5.7.25-1.el7.x86_64

Mar 18 22:27:24 Installed: mysql-community-server-5.7.25-1.el7.x86_64

Mar 18 22:27:24 Installed: mysql-community-libs-compat-5.7.25-1.el7.x86_64

Mar 18 22:27:24 Erased: 1:mariadb-libs-5.5.60-1.el7_5.x86_64

Mar 18 23:10:34 Installed: postgresql-libs-9.2.24-1.el7_5.x86_64

Mar 18 23:10:37 Installed: postgresql-9.2.24-1.el7_5.x86_64

Mar 18 23:10:38 Installed: postgresql-server-9.2.24-1.el7_5.x86_64

Mar 18 23:11:08 Installed: ambari-server-2.4.2.0-136.x86_64

Mar 18 23:28:05 Installed: ambari-agent-2.4.2.0-136.x86_64


3. python version

[root@namenode1 ~]# python --version

Python 2.7.5


4. Centos OS Version

[root@namenode1 ~]# cat /etc/redhat-release

CentOS Linux release 7.6.1810 (Core)


5. two SSL property has not been able in ambari-properties file.


6. Java version

[root@namenode1 ~]# java -version

java version "1.8.0_144"

Java(TM) SE Runtime Environment (build 1.8.0_144-b01)

Java HotSpot(TM) 64-Bit Server VM (build 25.144-b01, mixed mode)


7. java.security

jdk.tls.legacyAlgorithms= \

K_NULL, C_NULL, M_NULL, \

DHE_DSS_EXPORT, DHE_RSA_EXPORT, DH_anon_EXPORT, DH_DSS_EXPORT, \

DH_RSA_EXPORT, RSA_EXPORT, \

DH_anon, ECDH_anon, \

RC4_128, RC4_40, DES_CBC, DES40_CBC




Please: note this is a version vm