Two-way SSL authentication failed when registering ambari agent failure

yadem_ethio · ‎03-19-2019

Unable to register ambari-agent. I am getting two way ssl error. Here is the snip of my ambar-agent.ini file

[server]

hostname=namenode1.hadoop.com

url_port=8440

secured_url_port=8441

connect_retry_delay = 10

max_reconnect_retry_delay = 30

[security]

keysdir=/var/lib/ambari-agent/keys

server_crt=ca.crt

passphrase_env_var_name=AMBARI_PASSPHRASE

ssl_verify_cert=0

force_https_protocol=PROTOCOL_TLSv1_2

Error

ERROR 2019-03-19 00:04:10,160 security.py:87 - Two-way SSL authentication failed. Ensure that server and agent certificates were signed by the same CA and restart the agent.

In order to receive a new agent certificate, remove existing certificate file from keys directory. As a workaround you can turn off two-way SSL authentication in server configuration(ambari.properties)

Exiting..

ERROR 2019-03-19 00:04:10,161 Controller.py:212 - Unable to connect to: https://namenode1.hadoop.com:8441/agent/v1/register/namenode1.hadoop.com

Traceback (most recent call last):

File "/usr/lib/python2.6/site-packages/ambari_agent/Controller.py", line 165, in registerWithServer

ret = self.sendRequest(self.registerUrl, data)

File "/usr/lib/python2.6/site-packages/ambari_agent/Controller.py", line 496, in sendRequest

raise IOError('Request to {0} failed due to {1}'.format(url, str(exception)))

IOError: Request to https://namenode1.hadoop.com:8441/agent/v1/register/namenode1.hadoop.com failed due to EOF occurred in violation of protocol (_ssl.c:618)

ERROR 2019-03-19 00:04:10,161 Controller.py:213 - Error:Request to https://namenode1.hadoop.com:8441/agent/v1/register/namenode1.hadoop.com failed due to EOF occurred in violation of protocol (_ssl.c:618)

WARNING 2019-03-19 00:04:10,162 Controller.py:214 - Sleeping for 15 seconds and then trying again

I have tried few option and so far no luck. Any body has seen this issue?

jsensharma · ‎03-19-2019

@Yas Ethio

1. Is this a fresh cluster setup ? Or the Agents were running fine earlier and you started seeing these errors recently?

2. What is the Agent version?

# rpm -qa | grep ambari

3. Have you recently upgrade3d any packages on your agent hosts ? Specially the JDK or the Python libraries? You can verify it by looking the the yum logs

# tail -100f /var/log/yum.log

4. Do you see the "3DES_EDE_CBC" string inside your "$JAVA_HOME/jre/lib/security/java.security" file? On Ambari Server Host?
And if your Operating System in Centos6 (RHEL6) then please refer to the last section in the following doc to

Locate the jdk.tls.disabledAlgorithms property and remove the 3DES_EDE_CBC reference

https://community.hortonworks.com/articles/188269/javapython-updates-and-ambari-agent-tls-settings.h...

.

5. Also as we see that you might have enabled the Two Way SSL on your ambari server ? Is it intentional? By default ambari server and agent communication happens on One Way SSL.

# grep 'security.server.two_way_ssl'  /etc/ambari-server/conf/ambari.properties

If you have intentionally enabled 2 way ssl then please check if your Ambari Server certificates are expired by any chance?
Following article and it's comment section will give you more idea in that regard: https://community.hortonworks.com/articles/68799/steps-to-fix-ambari-server-agent-expired-certs.html

.

yadem_ethio · ‎03-19-2019

@Jay Kumar SenSharma please see as follows. Thank your quick reply.

1. Ambari agent version

[root@namenode1 ~]# rpm -qa | grep ambari

ambari-agent-2.4.2.0-136.x86_64

ambari-server-2.4.2.0-136.x86_64

2. yum package update list

[root@namenode1 ~]# tail -100f /var/log/yum.log

Mar 16 10:55:04 Installed: net-tools-2.0-0.24.20131004git.el7.x86_64

Mar 16 10:59:49 Installed: 1:perl-parent-0.225-244.el7.noarch

Mar 16 10:59:49 Installed: perl-HTTP-Tiny-0.033-3.el7.noarch

Mar 16 10:59:49 Installed: perl-podlators-2.5.1-3.el7.noarch

Mar 16 10:59:49 Installed: perl-Pod-Perldoc-3.20-4.el7.noarch

Mar 16 10:59:49 Installed: 1:perl-Pod-Escapes-1.04-294.el7_6.noarch

Mar 16 10:59:49 Installed: perl-Encode-2.51-7.el7.x86_64

Mar 16 10:59:49 Installed: perl-Text-ParseWords-3.29-4.el7.noarch

Mar 16 10:59:49 Installed: perl-Pod-Usage-1.63-3.el7.noarch

Mar 16 10:59:50 Installed: 4:perl-libs-5.16.3-294.el7_6.x86_64

Mar 16 10:59:50 Installed: 4:perl-macros-5.16.3-294.el7_6.x86_64

Mar 16 10:59:50 Installed: perl-Storable-2.45-3.el7.x86_64

Mar 16 10:59:50 Installed: perl-Exporter-5.68-3.el7.noarch

Mar 16 10:59:50 Installed: perl-constant-1.27-2.el7.noarch

Mar 16 10:59:50 Installed: perl-Time-Local-1.2300-2.el7.noarch

Mar 16 10:59:50 Installed: perl-Socket-2.010-4.el7.x86_64

Mar 16 10:59:50 Installed: perl-Carp-1.26-244.el7.noarch

Mar 16 10:59:50 Installed: 4:perl-Time-HiRes-1.9725-3.el7.x86_64

Mar 16 10:59:50 Installed: perl-PathTools-3.40-5.el7.x86_64

Mar 16 10:59:50 Installed: perl-Scalar-List-Utils-1.27-248.el7.x86_64

Mar 16 10:59:50 Installed: 1:perl-Pod-Simple-3.28-4.el7.noarch

Mar 16 10:59:50 Installed: perl-File-Temp-0.23.01-3.el7.noarch

Mar 16 10:59:50 Installed: perl-File-Path-2.09-2.el7.noarch

Mar 16 10:59:50 Installed: perl-threads-shared-1.43-6.el7.x86_64

Mar 16 10:59:50 Installed: perl-threads-1.87-4.el7.x86_64

Mar 16 10:59:50 Installed: perl-Filter-1.49-3.el7.x86_64

Mar 16 10:59:50 Installed: perl-Getopt-Long-2.40-3.el7.noarch

Mar 16 10:59:53 Installed: 4:perl-5.16.3-294.el7_6.x86_64

Mar 16 10:59:53 Installed: 2:vim-filesystem-7.4.160-5.el7.x86_64

Mar 16 10:59:56 Installed: 2:vim-common-7.4.160-5.el7.x86_64

Mar 16 10:59:56 Installed: gpm-libs-1.20.7-5.el7.x86_64

Mar 16 10:59:56 Installed: 2:vim-enhanced-7.4.160-5.el7.x86_64

Mar 16 11:18:16 Installed: wget-1.14-18.el7.x86_64

Mar 16 11:18:35 Installed: apr-1.4.8-3.el7_4.1.x86_64

Mar 16 11:18:36 Installed: apr-util-1.5.2-6.el7.x86_64

Mar 16 11:18:36 Installed: httpd-tools-2.4.6-88.el7.centos.x86_64

Mar 16 11:18:36 Installed: mailcap-2.1.41-2.el7.noarch

Mar 16 11:18:37 Installed: httpd-2.4.6-88.el7.centos.x86_64

Mar 16 11:18:54 Installed: autogen-libopts-5.18-5.el7.x86_64

Mar 16 11:18:54 Installed: ntpdate-4.2.6p5-28.el7.centos.x86_64

Mar 16 11:18:55 Installed: ntp-4.2.6p5-28.el7.centos.x86_64

Mar 16 11:18:55 Installed: ntp-perl-4.2.6p5-28.el7.centos.noarch

Mar 16 11:18:55 Installed: ntp-doc-4.2.6p5-28.el7.centos.noarch

Mar 16 11:20:32 Installed: mlocate-0.26-8.el7.x86_64

Mar 18 22:16:16 Installed: python-lxml-3.2.1-4.el7.x86_64

Mar 18 22:16:16 Installed: python-javapackages-3.4.1-11.el7.noarch

Mar 18 22:16:16 Installed: javapackages-tools-3.4.1-11.el7.noarch

Mar 18 22:16:17 Installed: xml-commons-apis-1.4.01-16.el7.noarch

Mar 18 22:16:17 Installed: geronimo-jms-1.1.1-19.el7.noarch

Mar 18 22:16:17 Installed: xml-commons-resolver-1.2-15.el7.noarch

Mar 18 22:16:17 Installed: xalan-j2-2.7.1-23.el7.noarch

Mar 18 22:16:17 Installed: xerces-j2-2.11.0-17.el7_0.noarch

Mar 18 22:16:17 Installed: apache-commons-lang-2.6-15.el7.noarch

Mar 18 22:16:17 Installed: tomcat-servlet-3.0-api-7.0.76-9.el7_6.noarch

Mar 18 22:16:17 Installed: cal10n-0.7.7-4.el7.noarch

Mar 18 22:16:18 Installed: javamail-1.4.6-8.el7.noarch

Mar 18 22:16:18 Installed: log4j-1.2.17-16.el7_4.noarch

Mar 18 22:16:18 Installed: apache-commons-logging-1.1.2-7.el7.noarch

Mar 18 22:16:18 Installed: avalon-logkit-2.1-14.el7.noarch

Mar 18 22:16:18 Installed: avalon-framework-4.3-10.el7.noarch

Mar 18 22:16:18 Installed: javassist-3.16.1-10.el7.noarch

Mar 18 22:16:18 Installed: slf4j-1.7.4-4.el7_4.noarch

Mar 18 22:16:18 Installed: geronimo-jta-1.1.1-17.el7.noarch

Mar 18 22:16:18 Installed: 1:mysql-connector-java-5.1.25-3.el7.noarch

Mar 18 22:26:44 Installed: mysql-community-common-5.7.25-1.el7.x86_64

Mar 18 22:26:45 Installed: mysql-community-libs-5.7.25-1.el7.x86_64

Mar 18 22:26:50 Installed: mysql-community-client-5.7.25-1.el7.x86_64

Mar 18 22:27:24 Installed: mysql-community-server-5.7.25-1.el7.x86_64

Mar 18 22:27:24 Installed: mysql-community-libs-compat-5.7.25-1.el7.x86_64

Mar 18 22:27:24 Erased: 1:mariadb-libs-5.5.60-1.el7_5.x86_64

Mar 18 23:10:34 Installed: postgresql-libs-9.2.24-1.el7_5.x86_64

Mar 18 23:10:37 Installed: postgresql-9.2.24-1.el7_5.x86_64

Mar 18 23:10:38 Installed: postgresql-server-9.2.24-1.el7_5.x86_64

Mar 18 23:11:08 Installed: ambari-server-2.4.2.0-136.x86_64

Mar 18 23:28:05 Installed: ambari-agent-2.4.2.0-136.x86_64

3. python version

[root@namenode1 ~]# python --version

Python 2.7.5

4. Centos OS Version

[root@namenode1 ~]# cat /etc/redhat-release

CentOS Linux release 7.6.1810 (Core)

5. two SSL property has not been able in ambari-properties file.

6. Java version

[root@namenode1 ~]# java -version

java version "1.8.0_144"

Java(TM) SE Runtime Environment (build 1.8.0_144-b01)

Java HotSpot(TM) 64-Bit Server VM (build 25.144-b01, mixed mode)

Please: note this is a version vm

yadem_ethio · ‎03-19-2019

Thanks for the quick reply Jay Kumar SenSharma. See below answer to your question.