Registration to ambari server via the ambari agent fail

[#1]

Hello Ambari Gurus,

I am installing the HDP 2.6 stack and I face the below issue.

Ambari agents don't get registered with the ambari server . When looking at the agent log it shows the following error:

IOError: Request to https://ls5387v7.wdf.sap.corp:8441/agent/v1/register/ls5387v8.wdf.sap.corp failed due to EOF occurred in violation of protocol (_ssl.c:661)

ERROR 2017-04-21 23:19:51,277 Controller.py:227 - Error:Request to https://ls5387v7.wdf.sap.corp:8441/agent/v1/register/ls5387v8.wdf.sap.corp failed due to EOF occurred in violation of protocol (_ssl.c:661) WARNING 2017-04-21 23:19:51,277 Controller.py:228 - Sleeping for 25 seconds and then trying again

Registration log for the host shows the following:-

========================== Creating target directory... ========================== Command start time 2017-04-21 23:16:17 Connection to ls5387v8.wdf.sap.corp closed. SSH command execution finished host=ls5387v8.wdf.sap.corp, exitcode=0 Command end time 2017-04-21 23:16:17 ========================== Copying ambari sudo script... ========================== Command start time 2017-04-21 23:16:17 scp /var/lib/ambari-server/ambari-sudo.sh host=ls5387v8.wdf.sap.corp, exitcode=0 Command end time 2017-04-21 23:16:17 ========================== Copying common functions script... ========================== Command start time 2017-04-21 23:16:17 scp /usr/lib/python2.6/site-packages/ambari_commons host=ls5387v8.wdf.sap.corp, exitcode=0 Command end time 2017-04-21 23:16:18 ========================== Copying create-python-wrap script... ========================== Command start time 2017-04-21 23:16:18 scp /var/lib/ambari-server/create-python-wrap.sh host=ls5387v8.wdf.sap.corp, exitcode=0 Command end time 2017-04-21 23:16:18 ========================== Copying OS type check script... ========================== Command start time 2017-04-21 23:16:18 scp /usr/lib/python2.6/site-packages/ambari_server/os_check_type.py host=ls5387v8.wdf.sap.corp, exitcode=0 Command end time 2017-04-21 23:16:18 ========================== Running create-python-wrap script... ========================== Command start time 2017-04-21 23:16:18 Connection to ls5387v8.wdf.sap.corp closed. SSH command execution finished host=ls5387v8.wdf.sap.corp, exitcode=0 Command end time 2017-04-21 23:16:19 ========================== Running OS type check... ========================== Command start time 2017-04-21 23:16:19 Cluster primary/cluster OS family is suse12 and local/current OS family is suse12 Connection to ls5387v8.wdf.sap.corp closed. SSH command execution finished host=ls5387v8.wdf.sap.corp, exitcode=0 Command end time 2017-04-21 23:16:19 ========================== Checking 'sudo' package on remote host... ========================== Command start time 2017-04-21 23:16:19 Connection to ls5387v8.wdf.sap.corp closed. SSH command execution finished host=ls5387v8.wdf.sap.corp, exitcode=0 Command end time 2017-04-21 23:16:19 ========================== Copying repo file to 'tmp' folder... ========================== Command start time 2017-04-21 23:16:19 scp /etc/zypp/repos.d/ambari.repo host=ls5387v8.wdf.sap.corp, exitcode=0 Command end time 2017-04-21 23:16:20 ========================== Moving file to repo dir... ========================== Command start time 2017-04-21 23:16:20 Connection to ls5387v8.wdf.sap.corp closed. SSH command execution finished host=ls5387v8.wdf.sap.corp, exitcode=0 Command end time 2017-04-21 23:16:20 ========================== Changing permissions for ambari.repo... ========================== Command start time 2017-04-21 23:16:20 Connection to ls5387v8.wdf.sap.corp closed. SSH command execution finished host=ls5387v8.wdf.sap.corp, exitcode=0 Command end time 2017-04-21 23:16:20 ========================== Copying setup script file... ========================== Command start time 2017-04-21 23:16:20 scp /usr/lib/python2.6/site-packages/ambari_server/setupAgent.py host=ls5387v8.wdf.sap.corp, exitcode=0 Command end time 2017-04-21 23:16:21 ========================== Running setup agent script... ========================== Command start time 2017-04-21 23:16:21 ("ERROR 2017-04-21 23:16:43,609 Controller.py:227 - Error:Request to https://ls5387v7.wdf.sap.corp:8441/agent/v1/register/ls5387v8.wdf.sap.corp failed due to EOF occurred in violation of protocol (_ssl.c:661) WARNING 2017-04-21 23:16:43,610 Controller.py:228 - Sleeping for 13 seconds and then trying again INFO 2017-04-21 23:16:54,945 main.py:286 - Agent not going to die gracefully, going to execute kill -9 INFO 2017-04-21 23:16:54,974 ExitHelper.py:56 - Performing cleanup before exiting... INFO 2017-04-21 23:16:55,838 main.py:145 - loglevel=logging.INFO INFO 2017-04-21 23:16:55,838 main.py:145 - loglevel=logging.INFO INFO 2017-04-21 23:16:55,838 main.py:145 - loglevel=logging.INFO INFO 2017-04-21 23:16:55,841 DataCleaner.py:39 - Data cleanup thread started INFO 2017-04-21 23:16:55,843 DataCleaner.py:120 - Data cleanup started INFO 2017-04-21 23:16:55,843 DataCleaner.py:122 - Data cleanup finished INFO 2017-04-21 23:16:56,003 PingPortListener.py:50 - Ping port listener started on port: 8670 INFO 2017-04-21 23:16:56,011 main.py:436 - Connecting to Ambari server at https://ls5387v7.wdf.sap.corp:8440 (10.21.24.138) INFO 2017-04-21 23:16:56,011 NetUtil.py:67 - Connecting to https://ls5387v7.wdf.sap.corp:8440/ca INFO 2017-04-21 23:16:56,148 main.py:446 - Connected to Ambari server ls5387v7.wdf.sap.corp INFO 2017-04-21 23:16:56,150 threadpool.py:58 - Started thread pool with 3 core threads and 20 maximum threads WARNING 2017-04-21 23:16:56,150 AlertSchedulerHandler.py:280 - [AlertScheduler] /var/lib/ambari-agent/cache/alerts/definitions.json not found or invalid. No alerts will be scheduled until registration occurs. INFO 2017-04-21 23:16:56,151 AlertSchedulerHandler.py:175 - [AlertScheduler] Starting <ambari_agent.apscheduler.scheduler.Scheduler object at 0x7f3865d85cd0>; currently running: False INFO 2017-04-21 23:16:58,171 hostname.py:98 - Read public hostname 'ls5387v8.wdf.sap.corp' using socket.getfqdn() INFO 2017-04-21 23:16:58,224 Hardware.py:174 - Some mount points were ignored: /dev/shm, /run, /sys/fs/cgroup INFO 2017-04-21 23:16:58,295 Facter.py:202 - Directory: '/etc/resource_overrides' does not exist - it won't be used for gathering system resources. ", None) ("ERROR 2017-04-21 23:16:43,609 Controller.py:227 - Error:Request to https://ls5387v7.wdf.sap.corp:8441/agent/v1/register/ls5387v8.wdf.sap.corp failed due to EOF occurred in violation of protocol (_ssl.c:661) WARNING 2017-04-21 23:16:43,610 Controller.py:228 - Sleeping for 13 seconds and then trying again INFO 2017-04-21 23:16:54,945 main.py:286 - Agent not going to die gracefully, going to execute kill -9 INFO 2017-04-21 23:16:54,974 ExitHelper.py:56 - Performing cleanup before exiting... INFO 2017-04-21 23:16:55,838 main.py:145 - loglevel=logging.INFO INFO 2017-04-21 23:16:55,838 main.py:145 - loglevel=logging.INFO INFO 2017-04-21 23:16:55,838 main.py:145 - loglevel=logging.INFO INFO 2017-04-21 23:16:55,841 DataCleaner.py:39 - Data cleanup thread started INFO 2017-04-21 23:16:55,843 DataCleaner.py:120 - Data cleanup started INFO 2017-04-21 23:16:55,843 DataCleaner.py:122 - Data cleanup finished INFO 2017-04-21 23:16:56,003 PingPortListener.py:50 - Ping port listener started on port: 8670 INFO 2017-04-21 23:16:56,011 main.py:436 - Connecting to Ambari server at https://ls5387v7.wdf.sap.corp:8440 (10.21.24.138) INFO 2017-04-21 23:16:56,011 NetUtil.py:67 - Connecting to https://ls5387v7.wdf.sap.corp:8440/ca INFO 2017-04-21 23:16:56,148 main.py:446 - Connected to Ambari server ls5387v7.wdf.sap.corp INFO 2017-04-21 23:16:56,150 threadpool.py:58 - Started thread pool with 3 core threads and 20 maximum threads WARNING 2017-04-21 23:16:56,150 AlertSchedulerHandler.py:280 - [AlertScheduler] /var/lib/ambari-agent/cache/alerts/definitions.json not found or invalid. No alerts will be scheduled until registration occurs. INFO 2017-04-21 23:16:56,151 AlertSchedulerHandler.py:175 - [AlertScheduler] Starting <ambari_agent.apscheduler.scheduler.Scheduler object at 0x7f3865d85cd0>; currently running: False INFO 2017-04-21 23:16:58,171 hostname.py:98 - Read public hostname 'ls5387v8.wdf.sap.corp' using socket.getfqdn() INFO 2017-04-21 23:16:58,224 Hardware.py:174 - Some mount points were ignored: /dev/shm, /run, /sys/fs/cgroup INFO 2017-04-21 23:16:58,295 Facter.py:202 - Directory: '/etc/resource_overrides' does not exist - it won't be used for gathering system resources. ", None) Connection to ls5387v8.wdf.sap.corp closed. SSH command execution finished host=ls5387v8.wdf.sap.corp, exitcode=0 Command end time 2017-04-21 23:16:58 Registering with the server... Registration with the server failed. OK Licensed under the Apache License, Version 2.0. See third-party tools/resources that Ambari uses and their respective authors

[#2]

@ amarnathreddy pappu

no ambari sever is not configured for 2 way ssl as the required parameter is not enabled.

yes the ambari agent shows exactly what you have mentioned.

The command you mentioned shows:-

CONNECTED(00000003) depth=0 C = AU, ST = Some-State, O = Internet Widgits Pty Ltd verify error:num=18:self signed certificate verify return:1 depth=0 C = AU, ST = Some-State, O = Internet Widgits Pty Ltd verify return:1 --- Certificate chain 0 s:/C=AU/ST=Some-State/O=Internet Widgits Pty Ltd i:/C=AU/ST=Some-State/O=Internet Widgits Pty Ltd --- Server certificate -----BEGIN CERTIFICATE----- MIIFpTCCA42gAwIBAgIBATANBgkqhkiG9w0BAQsFADBFMQswCQYDVQQGEwJBVTET MBEGA1UECAwKU29tZS1TdGF0ZTEhMB8GA1UECgwYSW50ZXJuZXQgV2lkZ2l0cyBQ dHkgTHRkMB4XDTE3MDQyMTE2MTgwOVoXDTE4MDQyMTE2MTgwOVowRTELMAkGA1UE BhMCQVUxEzARBgNVBAgMClNvbWUtU3RhdGUxITAfBgNVBAoMGEludGVybmV0IFdp ZGdpdHMgUHR5IEx0ZDCCAiIwDQYJKoZIhvcNAQEBBQADggIPADCCAgoCggIBANWQ xlofKWsaR+FtclgHw2Z8fwFNESPdc2Q6l5OTXAkrA4E8gbYBeMySIS4wZIqCrvnt OmfKZxwGYD/D8YzzGTCBMjY93F/hO9UK5kQGMJp+G4261u9jG+8FfoVF8zFaYr53 +g7YR+l+CfR4to0ZqjYugjWPU02UUabpw3uMpM8HvCYnkyfhhl0qurleC7bll44g RptALAPwb4FLwmABhygbLAZV4gKHn0ONPhPON6zV2VA9iudUOZl4wi+jQGjjb5TX SiBqE3Kd9W0ND7t61pER+sla9ASH5OVWZEMVIjnQNIDJ5PHudpA34MiItoR/JaaP kicUCtoGx8OoCxNMofSB5kLFXH+fcuk7zZlQeeeLFn1qMzDWGBNrKfQKzCJchE6P OhBArBPk6hZOFLzeqNbYiyD/w7bnXdg7qUwkE+hyu6c0UmdMdqCsmoME/0dAVJOD poqcuq5DyyQmLluFwRKZ0zlUEkPvK9Ey4l5E18gc+JvcfTlSrNoHYJ/hqRQYMU8B VRMupECYm6pvqT1CZEHM996gGbrWXjLsgtdGPX1VM0uRwtlGePpvMY6W/HtQoket XWywiJsaDQWucIxxAh/0JbIiXm5v+bUlj7fYnSOk2i9HI/x/oZh+3zQY6VjLSucd s2eJH8u4bLazbY3rYB6wCkevtdiZ+IiDqxCOSOxZAgMBAAGjgZ8wgZwwHQYDVR0O BBYEFK9z9r1rnK9uDkiZD6jWnTCHxWPdMG0GA1UdIwRmMGSAFK9z9r1rnK9uDkiZ D6jWnTCHxWPdoUmkRzBFMQswCQYDVQQGEwJBVTETMBEGA1UECAwKU29tZS1TdGF0 ZTEhMB8GA1UECgwYSW50ZXJuZXQgV2lkZ2l0cyBQdHkgTHRkggEBMAwGA1UdEwQF MAMBAf8wDQYJKoZIhvcNAQELBQADggIBAMZgMZPsqgRWU8nWGMbQl6kPrjo758Yw QMDD+O1B0pD57BZqcDEAHAmP0v1Am6DcGyRvWzwhBzRoT8VeNJKdyROQGhMXPWbC /E5kvBX6VxaetII9VgyOIUjizC/HKdS24PVu8sK6y7h0CNmmtUJj4P25SaOY7g2y -----END CERTIFICATE----- subject=/C=AU/ST=Some-State/O=Internet Widgits Pty Ltd issuer=/C=AU/ST=Some-State/O=Internet Widgits Pty Ltd --- No client certificate CA names sent --- SSL handshake has read 2257 bytes and written 455 bytes --- New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-GCM-SHA384 Server public key is 4096 bit Secure Renegotiation IS supported Compression: NONE Expansion: NONE SSL-Session: Protocol : TLSv1.2 Cipher : ECDHE-RSA-AES256-GCM-SHA384 Session-ID: 58FE6DA17EFA5278E0381D826F3E7E7E3F6558A6D4683964ACFDF4B4C63AD632 Session-ID-ctx: Master-Key: C0EEC8877A651977C8F5B6FCC78B4FD977DDA0A7BF06203DE433D04EC4B45A1788F8802B7F47AF58C210C321DD9BD225 Key-Arg : None PSK identity: None PSK identity hint: None SRP username: None Start Time: 1493069217 Timeout : 300 (sec) Verify return code: 18 (self signed certificate) --- closed

[#3 · Accepted solution]

We have over come the problem by adding

following option to security section in ambari-agent.ini in all the hosts in the cluster:

[security] force_https_protocol=PROTOCOL_TLSv1_2

[#4]

This worked for me. Thanks @Anish Gupta

[#5]

Same problem here when adding an extra node (after initial install) with HDP 2.6 and WITHOUT enabling two way ssl in Ambari.

And also same solution, adding the following property to /etc/ambari-agent/conf/ambari-agent.ini after install and registration failure and restarting the process:

force_https_protocol=PROTOCOL_TLSv1_2

[#6]

Guys, I'm installing my first ambari cluster and I'm at this page...

when you use the wizard system install the agents on the clients and the .ini file comes from the server (I think).

where to modify it before get pushed from the server?

thanks!

[#7]

read alsto about a limitation of Python..