Created 03-19-2019 05:29 AM
Unable to register ambari-agent. I am getting two way ssl error. Here is the snip of my ambar-agent.ini file
[server]
hostname=namenode1.hadoop.com
url_port=8440
secured_url_port=8441
connect_retry_delay = 10
max_reconnect_retry_delay = 30
[security]
keysdir=/var/lib/ambari-agent/keys
server_crt=ca.crt
passphrase_env_var_name=AMBARI_PASSPHRASE
ssl_verify_cert=0
force_https_protocol=PROTOCOL_TLSv1_2
Error
ERROR 2019-03-19 00:04:10,160 security.py:87 - Two-way SSL authentication failed. Ensure that server and agent certificates were signed by the same CA and restart the agent.
In order to receive a new agent certificate, remove existing certificate file from keys directory. As a workaround you can turn off two-way SSL authentication in server configuration(ambari.properties)
Exiting..
ERROR 2019-03-19 00:04:10,161 Controller.py:212 - Unable to connect to: https://namenode1.hadoop.com:8441/agent/v1/register/namenode1.hadoop.com
Traceback (most recent call last):
File "/usr/lib/python2.6/site-packages/ambari_agent/Controller.py", line 165, in registerWithServer
ret = self.sendRequest(self.registerUrl, data)
File "/usr/lib/python2.6/site-packages/ambari_agent/Controller.py", line 496, in sendRequest
raise IOError('Request to {0} failed due to {1}'.format(url, str(exception)))
IOError: Request to https://namenode1.hadoop.com:8441/agent/v1/register/namenode1.hadoop.com failed due to EOF occurred in violation of protocol (_ssl.c:618)
ERROR 2019-03-19 00:04:10,161 Controller.py:213 - Error:Request to https://namenode1.hadoop.com:8441/agent/v1/register/namenode1.hadoop.com failed due to EOF occurred in violation of protocol (_ssl.c:618)
WARNING 2019-03-19 00:04:10,162 Controller.py:214 - Sleeping for 15 seconds and then trying again
I have tried few option and so far no luck. Any body has seen this issue?
Created 03-19-2019 05:37 AM
1. Is this a fresh cluster setup ? Or the Agents were running fine earlier and you started seeing these errors recently?
2. What is the Agent version?
# rpm -qa | grep ambari
3. Have you recently upgrade3d any packages on your agent hosts ? Specially the JDK or the Python libraries? You can verify it by looking the the yum logs
# tail -100f /var/log/yum.log
4. Do you see the "3DES_EDE_CBC" string inside your "$JAVA_HOME/jre/lib/security/java.security" file? On Ambari Server Host?
And if your Operating System in Centos6 (RHEL6) then please refer to the last section in the following doc to
.
5. Also as we see that you might have enabled the Two Way SSL on your ambari server ? Is it intentional? By default ambari server and agent communication happens on One Way SSL.
# grep 'security.server.two_way_ssl' /etc/ambari-server/conf/ambari.properties
If you have intentionally enabled 2 way ssl then please check if your Ambari Server certificates are expired by any chance?
Following article and it's comment section will give you more idea in that regard: https://community.hortonworks.com/articles/68799/steps-to-fix-ambari-server-agent-expired-certs.html
.
Created 03-19-2019 10:23 PM
@Jay Kumar SenSharma please see as follows. Thank your quick reply.
1. Ambari agent version
[root@namenode1 ~]# rpm -qa | grep ambari
ambari-agent-2.4.2.0-136.x86_64
ambari-server-2.4.2.0-136.x86_64
2. yum package update list
[root@namenode1 ~]# tail -100f /var/log/yum.log
Mar 16 10:55:04 Installed: net-tools-2.0-0.24.20131004git.el7.x86_64
Mar 16 10:59:49 Installed: 1:perl-parent-0.225-244.el7.noarch
Mar 16 10:59:49 Installed: perl-HTTP-Tiny-0.033-3.el7.noarch
Mar 16 10:59:49 Installed: perl-podlators-2.5.1-3.el7.noarch
Mar 16 10:59:49 Installed: perl-Pod-Perldoc-3.20-4.el7.noarch
Mar 16 10:59:49 Installed: 1:perl-Pod-Escapes-1.04-294.el7_6.noarch
Mar 16 10:59:49 Installed: perl-Encode-2.51-7.el7.x86_64
Mar 16 10:59:49 Installed: perl-Text-ParseWords-3.29-4.el7.noarch
Mar 16 10:59:49 Installed: perl-Pod-Usage-1.63-3.el7.noarch
Mar 16 10:59:50 Installed: 4:perl-libs-5.16.3-294.el7_6.x86_64
Mar 16 10:59:50 Installed: 4:perl-macros-5.16.3-294.el7_6.x86_64
Mar 16 10:59:50 Installed: perl-Storable-2.45-3.el7.x86_64
Mar 16 10:59:50 Installed: perl-Exporter-5.68-3.el7.noarch
Mar 16 10:59:50 Installed: perl-constant-1.27-2.el7.noarch
Mar 16 10:59:50 Installed: perl-Time-Local-1.2300-2.el7.noarch
Mar 16 10:59:50 Installed: perl-Socket-2.010-4.el7.x86_64
Mar 16 10:59:50 Installed: perl-Carp-1.26-244.el7.noarch
Mar 16 10:59:50 Installed: 4:perl-Time-HiRes-1.9725-3.el7.x86_64
Mar 16 10:59:50 Installed: perl-PathTools-3.40-5.el7.x86_64
Mar 16 10:59:50 Installed: perl-Scalar-List-Utils-1.27-248.el7.x86_64
Mar 16 10:59:50 Installed: 1:perl-Pod-Simple-3.28-4.el7.noarch
Mar 16 10:59:50 Installed: perl-File-Temp-0.23.01-3.el7.noarch
Mar 16 10:59:50 Installed: perl-File-Path-2.09-2.el7.noarch
Mar 16 10:59:50 Installed: perl-threads-shared-1.43-6.el7.x86_64
Mar 16 10:59:50 Installed: perl-threads-1.87-4.el7.x86_64
Mar 16 10:59:50 Installed: perl-Filter-1.49-3.el7.x86_64
Mar 16 10:59:50 Installed: perl-Getopt-Long-2.40-3.el7.noarch
Mar 16 10:59:53 Installed: 4:perl-5.16.3-294.el7_6.x86_64
Mar 16 10:59:53 Installed: 2:vim-filesystem-7.4.160-5.el7.x86_64
Mar 16 10:59:56 Installed: 2:vim-common-7.4.160-5.el7.x86_64
Mar 16 10:59:56 Installed: gpm-libs-1.20.7-5.el7.x86_64
Mar 16 10:59:56 Installed: 2:vim-enhanced-7.4.160-5.el7.x86_64
Mar 16 11:18:16 Installed: wget-1.14-18.el7.x86_64
Mar 16 11:18:35 Installed: apr-1.4.8-3.el7_4.1.x86_64
Mar 16 11:18:36 Installed: apr-util-1.5.2-6.el7.x86_64
Mar 16 11:18:36 Installed: httpd-tools-2.4.6-88.el7.centos.x86_64
Mar 16 11:18:36 Installed: mailcap-2.1.41-2.el7.noarch
Mar 16 11:18:37 Installed: httpd-2.4.6-88.el7.centos.x86_64
Mar 16 11:18:54 Installed: autogen-libopts-5.18-5.el7.x86_64
Mar 16 11:18:54 Installed: ntpdate-4.2.6p5-28.el7.centos.x86_64
Mar 16 11:18:55 Installed: ntp-4.2.6p5-28.el7.centos.x86_64
Mar 16 11:18:55 Installed: ntp-perl-4.2.6p5-28.el7.centos.noarch
Mar 16 11:18:55 Installed: ntp-doc-4.2.6p5-28.el7.centos.noarch
Mar 16 11:20:32 Installed: mlocate-0.26-8.el7.x86_64
Mar 18 22:16:16 Installed: python-lxml-3.2.1-4.el7.x86_64
Mar 18 22:16:16 Installed: python-javapackages-3.4.1-11.el7.noarch
Mar 18 22:16:16 Installed: javapackages-tools-3.4.1-11.el7.noarch
Mar 18 22:16:17 Installed: xml-commons-apis-1.4.01-16.el7.noarch
Mar 18 22:16:17 Installed: geronimo-jms-1.1.1-19.el7.noarch
Mar 18 22:16:17 Installed: xml-commons-resolver-1.2-15.el7.noarch
Mar 18 22:16:17 Installed: xalan-j2-2.7.1-23.el7.noarch
Mar 18 22:16:17 Installed: xerces-j2-2.11.0-17.el7_0.noarch
Mar 18 22:16:17 Installed: apache-commons-lang-2.6-15.el7.noarch
Mar 18 22:16:17 Installed: tomcat-servlet-3.0-api-7.0.76-9.el7_6.noarch
Mar 18 22:16:17 Installed: cal10n-0.7.7-4.el7.noarch
Mar 18 22:16:18 Installed: javamail-1.4.6-8.el7.noarch
Mar 18 22:16:18 Installed: log4j-1.2.17-16.el7_4.noarch
Mar 18 22:16:18 Installed: apache-commons-logging-1.1.2-7.el7.noarch
Mar 18 22:16:18 Installed: avalon-logkit-2.1-14.el7.noarch
Mar 18 22:16:18 Installed: avalon-framework-4.3-10.el7.noarch
Mar 18 22:16:18 Installed: javassist-3.16.1-10.el7.noarch
Mar 18 22:16:18 Installed: slf4j-1.7.4-4.el7_4.noarch
Mar 18 22:16:18 Installed: geronimo-jta-1.1.1-17.el7.noarch
Mar 18 22:16:18 Installed: 1:mysql-connector-java-5.1.25-3.el7.noarch
Mar 18 22:26:44 Installed: mysql-community-common-5.7.25-1.el7.x86_64
Mar 18 22:26:45 Installed: mysql-community-libs-5.7.25-1.el7.x86_64
Mar 18 22:26:50 Installed: mysql-community-client-5.7.25-1.el7.x86_64
Mar 18 22:27:24 Installed: mysql-community-server-5.7.25-1.el7.x86_64
Mar 18 22:27:24 Installed: mysql-community-libs-compat-5.7.25-1.el7.x86_64
Mar 18 22:27:24 Erased: 1:mariadb-libs-5.5.60-1.el7_5.x86_64
Mar 18 23:10:34 Installed: postgresql-libs-9.2.24-1.el7_5.x86_64
Mar 18 23:10:37 Installed: postgresql-9.2.24-1.el7_5.x86_64
Mar 18 23:10:38 Installed: postgresql-server-9.2.24-1.el7_5.x86_64
Mar 18 23:11:08 Installed: ambari-server-2.4.2.0-136.x86_64
Mar 18 23:28:05 Installed: ambari-agent-2.4.2.0-136.x86_64
3. python version
[root@namenode1 ~]# python --version
Python 2.7.5
4. Centos OS Version
[root@namenode1 ~]# cat /etc/redhat-release
CentOS Linux release 7.6.1810 (Core)
5. two SSL property has not been able in ambari-properties file.
6. Java version
[root@namenode1 ~]# java -version
java version "1.8.0_144"
Java(TM) SE Runtime Environment (build 1.8.0_144-b01)
Java HotSpot(TM) 64-Bit Server VM (build 25.144-b01, mixed mode)
Please: note this is a version vm
Created 03-19-2019 10:27 PM
Thanks for the quick reply Jay Kumar SenSharma. See below answer to your question.
1. Ambari agent version
[root@namenode1 ~]# rpm -qa | grep ambari
ambari-agent-2.4.2.0-136.x86_64
ambari-server-2.4.2.0-136.x86_64
2. yum package update list
[root@namenode1 ~]# tail -100f /var/log/yum.log
Mar 16 10:55:04 Installed: net-tools-2.0-0.24.20131004git.el7.x86_64
Mar 16 10:59:49 Installed: 1:perl-parent-0.225-244.el7.noarch
Mar 16 10:59:49 Installed: perl-HTTP-Tiny-0.033-3.el7.noarch
Mar 16 10:59:49 Installed: perl-podlators-2.5.1-3.el7.noarch
Mar 16 10:59:49 Installed: perl-Pod-Perldoc-3.20-4.el7.noarch
Mar 16 10:59:49 Installed: 1:perl-Pod-Escapes-1.04-294.el7_6.noarch
Mar 16 10:59:49 Installed: perl-Encode-2.51-7.el7.x86_64
Mar 16 10:59:49 Installed: perl-Text-ParseWords-3.29-4.el7.noarch
Mar 16 10:59:49 Installed: perl-Pod-Usage-1.63-3.el7.noarch
Mar 16 10:59:50 Installed: 4:perl-libs-5.16.3-294.el7_6.x86_64
Mar 16 10:59:50 Installed: 4:perl-macros-5.16.3-294.el7_6.x86_64
Mar 16 10:59:50 Installed: perl-Storable-2.45-3.el7.x86_64
Mar 16 10:59:50 Installed: perl-Exporter-5.68-3.el7.noarch
Mar 16 10:59:50 Installed: perl-constant-1.27-2.el7.noarch
Mar 16 10:59:50 Installed: perl-Time-Local-1.2300-2.el7.noarch
Mar 16 10:59:50 Installed: perl-Socket-2.010-4.el7.x86_64
Mar 16 10:59:50 Installed: perl-Carp-1.26-244.el7.noarch
Mar 16 10:59:50 Installed: 4:perl-Time-HiRes-1.9725-3.el7.x86_64
Mar 16 10:59:50 Installed: perl-PathTools-3.40-5.el7.x86_64
Mar 16 10:59:50 Installed: perl-Scalar-List-Utils-1.27-248.el7.x86_64
Mar 16 10:59:50 Installed: 1:perl-Pod-Simple-3.28-4.el7.noarch
Mar 16 10:59:50 Installed: perl-File-Temp-0.23.01-3.el7.noarch
Mar 16 10:59:50 Installed: perl-File-Path-2.09-2.el7.noarch
Mar 16 10:59:50 Installed: perl-threads-shared-1.43-6.el7.x86_64
Mar 16 10:59:50 Installed: perl-threads-1.87-4.el7.x86_64
Mar 16 10:59:50 Installed: perl-Filter-1.49-3.el7.x86_64
Mar 16 10:59:50 Installed: perl-Getopt-Long-2.40-3.el7.noarch
Mar 16 10:59:53 Installed: 4:perl-5.16.3-294.el7_6.x86_64
Mar 16 10:59:53 Installed: 2:vim-filesystem-7.4.160-5.el7.x86_64
Mar 16 10:59:56 Installed: 2:vim-common-7.4.160-5.el7.x86_64
Mar 16 10:59:56 Installed: gpm-libs-1.20.7-5.el7.x86_64
Mar 16 10:59:56 Installed: 2:vim-enhanced-7.4.160-5.el7.x86_64
Mar 16 11:18:16 Installed: wget-1.14-18.el7.x86_64
Mar 16 11:18:35 Installed: apr-1.4.8-3.el7_4.1.x86_64
Mar 16 11:18:36 Installed: apr-util-1.5.2-6.el7.x86_64
Mar 16 11:18:36 Installed: httpd-tools-2.4.6-88.el7.centos.x86_64
Mar 16 11:18:36 Installed: mailcap-2.1.41-2.el7.noarch
Mar 16 11:18:37 Installed: httpd-2.4.6-88.el7.centos.x86_64
Mar 16 11:18:54 Installed: autogen-libopts-5.18-5.el7.x86_64
Mar 16 11:18:54 Installed: ntpdate-4.2.6p5-28.el7.centos.x86_64
Mar 16 11:18:55 Installed: ntp-4.2.6p5-28.el7.centos.x86_64
Mar 16 11:18:55 Installed: ntp-perl-4.2.6p5-28.el7.centos.noarch
Mar 16 11:18:55 Installed: ntp-doc-4.2.6p5-28.el7.centos.noarch
Mar 16 11:20:32 Installed: mlocate-0.26-8.el7.x86_64
Mar 18 22:16:16 Installed: python-lxml-3.2.1-4.el7.x86_64
Mar 18 22:16:16 Installed: python-javapackages-3.4.1-11.el7.noarch
Mar 18 22:16:16 Installed: javapackages-tools-3.4.1-11.el7.noarch
Mar 18 22:16:17 Installed: xml-commons-apis-1.4.01-16.el7.noarch
Mar 18 22:16:17 Installed: geronimo-jms-1.1.1-19.el7.noarch
Mar 18 22:16:17 Installed: xml-commons-resolver-1.2-15.el7.noarch
Mar 18 22:16:17 Installed: xalan-j2-2.7.1-23.el7.noarch
Mar 18 22:16:17 Installed: xerces-j2-2.11.0-17.el7_0.noarch
Mar 18 22:16:17 Installed: apache-commons-lang-2.6-15.el7.noarch
Mar 18 22:16:17 Installed: tomcat-servlet-3.0-api-7.0.76-9.el7_6.noarch
Mar 18 22:16:17 Installed: cal10n-0.7.7-4.el7.noarch
Mar 18 22:16:18 Installed: javamail-1.4.6-8.el7.noarch
Mar 18 22:16:18 Installed: log4j-1.2.17-16.el7_4.noarch
Mar 18 22:16:18 Installed: apache-commons-logging-1.1.2-7.el7.noarch
Mar 18 22:16:18 Installed: avalon-logkit-2.1-14.el7.noarch
Mar 18 22:16:18 Installed: avalon-framework-4.3-10.el7.noarch
Mar 18 22:16:18 Installed: javassist-3.16.1-10.el7.noarch
Mar 18 22:16:18 Installed: slf4j-1.7.4-4.el7_4.noarch
Mar 18 22:16:18 Installed: geronimo-jta-1.1.1-17.el7.noarch
Mar 18 22:16:18 Installed: 1:mysql-connector-java-5.1.25-3.el7.noarch
Mar 18 22:26:44 Installed: mysql-community-common-5.7.25-1.el7.x86_64
Mar 18 22:26:45 Installed: mysql-community-libs-5.7.25-1.el7.x86_64
Mar 18 22:26:50 Installed: mysql-community-client-5.7.25-1.el7.x86_64
Mar 18 22:27:24 Installed: mysql-community-server-5.7.25-1.el7.x86_64
Mar 18 22:27:24 Installed: mysql-community-libs-compat-5.7.25-1.el7.x86_64
Mar 18 22:27:24 Erased: 1:mariadb-libs-5.5.60-1.el7_5.x86_64
Mar 18 23:10:34 Installed: postgresql-libs-9.2.24-1.el7_5.x86_64
Mar 18 23:10:37 Installed: postgresql-9.2.24-1.el7_5.x86_64
Mar 18 23:10:38 Installed: postgresql-server-9.2.24-1.el7_5.x86_64
Mar 18 23:11:08 Installed: ambari-server-2.4.2.0-136.x86_64
Mar 18 23:28:05 Installed: ambari-agent-2.4.2.0-136.x86_64
3. python version
[root@namenode1 ~]# python --version
Python 2.7.5
4. Centos OS Version
[root@namenode1 ~]# cat /etc/redhat-release
CentOS Linux release 7.6.1810 (Core)
5. two SSL property has not been able in ambari-properties file.
6. Java version
[root@namenode1 ~]# java -version
java version "1.8.0_144"
Java(TM) SE Runtime Environment (build 1.8.0_144-b01)
Java HotSpot(TM) 64-Bit Server VM (build 25.144-b01, mixed mode)
7. java.security
jdk.tls.legacyAlgorithms= \
K_NULL, C_NULL, M_NULL, \
DHE_DSS_EXPORT, DHE_RSA_EXPORT, DH_anon_EXPORT, DH_DSS_EXPORT, \
DH_RSA_EXPORT, RSA_EXPORT, \
DH_anon, ECDH_anon, \
RC4_128, RC4_40, DES_CBC, DES40_CBC
Please: note this is a version vm