question Re: Error in Step 3 Ambari: automatically register hosts in Support Questions

Error in Step 3 Ambari: automatically register hosts

jloormoreira — Fri, 31 Jul 2020 17:27:54 GMT

Hi everyone.

Before anything I'm going to describe the environment where I'm trying to install the Cluster on Amabri:

Static hostname: sandbox-hdp.hortonworks.com
Icon name: computer-vm
Chassis: vm
Machine ID: d8323e61d06a489f97b6175ebe8ff9c7
Boot ID: a0b2c8c30d5345f796a51dc754e807e9
Virtualization: vmware
Operating System: CentOS Linux 7 (Core)
CPE OS Name: cpe:/o:centos:centos:7
Kernel: Linux 3.10.0-1127.13.1.el7.x86_64
Architecture: x86-64

Amabri Agent Version:

ambari-agent.x86_64 2.7.4.0-118 @Ambari-2.7.4.0

Ambari Server Version:

ambari-server.x86_64 2.7.4.0-118 @Ambari-2.7.4.0

Ambari Agent ini file:

[server]
hostname=sandbox-hdp.hortonworks.com
url_port=8440
secured_url_port=8441
connect_retry_delay=10
max_reconnect_retry_delay=30

[agent]

....

public_hostname_script=/var/lib/ambari-agent/hostname.sh
hostname_script=/var/lib/ambari-agent/hostname.sh

[security]

....
force_https_protocol=PROTOCOL_TLSv1_2

/var/lib/ambari-agent/hostname.sh

#!/bin/bash
#hostname -f <===== I've tried using this
echo 'sandbox-hdp.hortonworks.com'

/etc/sysconfig/network:

NETWORKING=yes
NETWORKING_IPV6=yes
HOSTNAME=sandbox-hdp.hortonworks.com

/etc/hosts

127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.100.183 sandbox-hdp.hortonworks.com sandbox-hdp

I think thats all the config information about the environment. Amabri Sever and Agents work perfect, i can access to Ambari UI with the user admin and password admin; even there I created the ssh key for the server without problem and using WinSCP I can access to any folder of the server because I use the root user. I need to mention that everything is in one server, only one machine has ther server and the agent installed.

Until there no problem at all; but when I want to create a cluster using the Wizard of Ambari; in the step 3 is where i got the problem:

1. In the /var/log/ambari-agent/ambari-agent.log file always i got this message:

WARNING 2020-07-29 21:37:12,212 hostname.py:90 - Execution of '/var/lib/ambari-agent/hostname.sh' returned 126. /bin/sh: /var/lib/ambari-agent/hostname.sh: Permission denied.

2. At the same log file, got not problem with the secured ports:

INFO 2020-07-29 21:37:11,597 main.py:155 - loglevel=logging.INFO
INFO 2020-07-29 21:37:11,599 Hardware.py:68 - Initializing host system information.
INFO 2020-07-29 21:37:11,605 Hardware.py:188 - Some mount points were ignored: /dev, /dev/shm, /run, /sys/fs/cgroup, /boot, /home, /run/user/0
INFO 2020-07-29 21:37:11,617 Facter.py:202 - Directory: '/etc/resource_overrides' does not exist - it won't be used for gathering system resources.
INFO 2020-07-29 21:37:11,620 Hardware.py:73 - Host system information: {'kernel': 'Linux', 'domain': 'hortonworks.com', 'physicalprocessorcount': 4, 'kernelrelease': '3.10.0-1127.13.1.el7.x86_64', 'uptime_days': '0', 'memorytotal': 12137760, 'swapfree': '5.87 GB', 'memorysize': 12137760, 'osfamily': 'redhat', 'swapsize': '5.87 GB', 'processorcount': 4, 'netmask': '255.255.255.0', 'timezone': '-05', 'hardwareisa': 'x86_64', 'memoryfree': 10153096, 'operatingsystem': 'centos', 'kernelmajversion': '3.10', 'kernelversion': '3.10.0', 'macaddress': '00:0C:29:01:4A:A8', 'operatingsystemrelease': '7.8.2003', 'ipaddress': '192.168.100.183', 'hostname': 'sandbox-hdp', 'uptime_hours': '20', 'fqdn': 'sandbox-hdp.hortonworks.com', 'id': 'root', 'architecture': 'x86_64', 'selinux': False, 'mounts': [{'available': '45682904', 'used': '6720296', 'percent': '13%', 'device': '/dev/mapper/centos-root', 'mountpoint': '/', 'type': 'xfs', 'size': '52403200'}], 'hardwaremodel': 'x86_64', 'uptime_seconds': '74504', 'interfaces': 'ens192,lo'}
WARNING 2020-07-29 21:37:11,621 shell.py:826 - can not switch user for RUN_COMMAND.
WARNING 2020-07-29 21:37:11,625 shell.py:826 - can not switch user for RUN_COMMAND.
WARNING 2020-07-29 21:37:11,729 shell.py:826 - can not switch user for RUN_COMMAND.
INFO 2020-07-29 21:37:11,734 main.py:308 - Agent died gracefully, exiting.
INFO 2020-07-29 21:37:11,734 ExitHelper.py:57 - Performing cleanup before exiting...
INFO 2020-07-29 21:37:11,734 AlertSchedulerHandler.py:159 - [AlertScheduler] Stopped the alert scheduler.
INFO 2020-07-29 21:37:11,735 AlertSchedulerHandler.py:159 - [AlertScheduler] Stopped the alert scheduler.
INFO 2020-07-29 21:37:11,985 main.py:155 - loglevel=logging.INFO
INFO 2020-07-29 21:37:11,988 Hardware.py:68 - Initializing host system information.
INFO 2020-07-29 21:37:11,992 Hardware.py:188 - Some mount points were ignored: /dev, /dev/shm, /run, /sys/fs/cgroup, /boot, /home, /run/user/0
INFO 2020-07-29 21:37:12,004 Facter.py:202 - Directory: '/etc/resource_overrides' does not exist - it won't be used for gathering system resources.
INFO 2020-07-29 21:37:12,007 Hardware.py:73 - Host system information: {'kernel': 'Linux', 'domain': 'hortonworks.com', 'physicalprocessorcount': 4, 'kernelrelease': '3.10.0-1127.13.1.el7.x86_64', 'uptime_days': '0', 'memorytotal': 12137760, 'swapfree': '5.87 GB', 'memorysize': 12137760, 'osfamily': 'redhat', 'swapsize': '5.87 GB', 'processorcount': 4, 'netmask': '255.255.255.0', 'timezone': '-05', 'hardwareisa': 'x86_64', 'memoryfree': 10175244, 'operatingsystem': 'centos', 'kernelmajversion': '3.10', 'kernelversion': '3.10.0', 'macaddress': '00:0C:29:01:4A:A8', 'operatingsystemrelease': '7.8.2003', 'ipaddress': '192.168.100.183', 'hostname': 'sandbox-hdp', 'uptime_hours': '20', 'fqdn': 'sandbox-hdp.hortonworks.com', 'id': 'root', 'architecture': 'x86_64', 'selinux': False, 'mounts': [{'available': '45687172', 'used': '6716028', 'percent': '13%', 'device': '/dev/mapper/centos-root', 'mountpoint': '/', 'type': 'xfs', 'size': '52403200'}], 'hardwaremodel': 'x86_64', 'uptime_seconds': '74504', 'interfaces': 'ens192,lo'}
INFO 2020-07-29 21:37:12,009 DataCleaner.py:39 - Data cleanup thread started
INFO 2020-07-29 21:37:12,009 DataCleaner.py:120 - Data cleanup started
INFO 2020-07-29 21:37:12,010 DataCleaner.py:122 - Data cleanup finished
WARNING 2020-07-29 21:37:12,011 hostname.py:63 - Unexpected error while retrieving hostname: '(<type 'exceptions.OSError'>, OSError(13, 'Permission denied'), <traceback object at 0x7f4ff6f46fc8>)', defaulting to socket.getfqdn()
INFO 2020-07-29 21:37:12,011 hostname.py:64 - Read hostname 'sandbox-hdp.hortonworks.com' using socket.getfqdn().
INFO 2020-07-29 21:37:12,014 PingPortListener.py:50 - Ping port listener started on port: 8670
INFO 2020-07-29 21:37:12,015 main.py:191 - Newloglevel=logging.DEBUG
INFO 2020-07-29 21:37:12,016 main.py:481 - Connecting to Ambari server at https://sandbox-hdp.hortonworks.com:8440 (192.168.100.183)
DEBUG 2020-07-29 21:37:12,016 NetUtil.py:104 - Trying to connect to https://sandbox-hdp.hortonworks.com:8440
INFO 2020-07-29 21:37:12,016 NetUtil.py:61 - Connecting to https://sandbox-hdp.hortonworks.com:8440/ca
DEBUG 2020-07-29 21:37:12,065 NetUtil.py:81 - GET https://sandbox-hdp.hortonworks.com:8440/ca -> 200, body:
INFO 2020-07-29 21:37:12,066 main.py:491 - Connected to Ambari server sandbox-hdp.hortonworks.com
INFO 2020-07-29 21:37:12,066 AlertSchedulerHandler.py:149 - [AlertScheduler] Starting <ambari_agent.apscheduler.scheduler.Scheduler object at 0x7f4ff6f3c8d0>; currently running: False
INFO 2020-07-29 21:37:12,067 NetUtil.py:61 - Connecting to https://sandbox-hdp.hortonworks.com:8440/connection_info
DEBUG 2020-07-29 21:37:12,116 NetUtil.py:81 - GET https://sandbox-hdp.hortonworks.com:8440/connection_info -> 200, body: {"security.server.two_way_ssl":"true"}
DEBUG 2020-07-29 21:37:12,117 security.py:56 - Server two-way SSL authentication required: True
INFO 2020-07-29 21:37:12,117 security.py:59 - Server require two-way SSL authentication. Use it instead of one-way...
INFO 2020-07-29 21:37:12,117 security.py:61 - Connecting to wss://sandbox-hdp.hortonworks.com:8441/agent/stomp/v1
INFO 2020-07-29 21:37:12,117 security.py:224 - Server certicate exists, ok
INFO 2020-07-29 21:37:12,117 security.py:232 - Agent key exists, ok
INFO 2020-07-29 21:37:12,117 security.py:240 - Agent certificate exists, ok
INFO 2020-07-29 21:37:12,169 transport.py:329 - Starting receiver loop
DEBUG 2020-07-29 21:37:12,169 transport.py:269 - Sending frame: ['STOMP', '\n', 'accept-version:1.2\n', 'host:0\n', '\n', '\x00']

3. At the same log file, I always get problem with long data at the time that Ambari is trying to register the host:

WARNING 2020-07-29 21:37:12,212 hostname.py:90 - Execution of '/var/lib/ambari-agent/hostname.sh' returned 126. /bin/sh: /var/lib/ambari-agent/hostname.sh: Permission denied

INFO 2020-07-29 21:37:12,220 hostname.py:103 - Read public hostname '<!doctype html public "-//w3c//dtd xhtml 1.0 transitional//en" "http://www.w3.org/tr/xhtml1/dtd/xhtml1-transitional.dtd">^M
<html xmlns="http://www.w3.org/1999/xhtml">^M
<head>^M
<meta content="text/html; charset=utf-8" http-equiv="content-type" />^M
<meta content="no-cache" http-equiv="pragma" />^M
<title>waiting...</title>^M
<script type="text/javascript">^M
var pagename = '/';^M
top.location.replace(pagename);^M
</script>^M
</head>^M
<body> </body>^M
</html>^M
' from http://169.254.169.254/latest/meta-data/public-hostname
INFO 2020-07-29 21:37:12,220 HeartbeatThread.py:127 - Sending registration request

....

DEBUG 2020-07-29 21:37:12,227 transport.py:189 - Received frame: 'MESSAGE', headers={'content-length': '648', 'destination': '/user/', 'message-id': '1d065200-6399-190c-1707-13715074fb75-50', 'content-type': 'application/json;charset=UTF-8', 'correlationId': '0', 'subscription': 'sub'}, body='{"status":"OK","exitstatus":1,"log":"Exception [EclipseLink-4002] (Eclipse Persistence Services - 2.6.2.v20151217-774c696): org.eclipse.persistence.exceptions.DatabaseException\\nInternal Exception: com.mysql.cj.jdbc.exceptions.MysqlDataTruncation: Data truncation: Data too long for column \'public_host_name\' at row 1\\nError Code: 1406\\nCall: UPDATE hosts SET ipv4 = ?, ipv6 = ?, cpu_count = ?, ph_cpu_count = ?, total_mem = ?, os_arch = ?, os_type = ?, host_attributes = ?, public_host_name = ? WHERE (host_id = ?)\\n\\tbind => [10 parameters bound]\\nQuery: UpdateObjectQuery(org.apache.ambari.server.orm.entities.HostStateEntity@8b6494c6)","id":-1}'
INFO 2020-07-29 21:37:12,227 __init__.py:82 - Event from server at /user/ (correlation_id=0): {u'status': u'OK', u'exitstatus': 1, u'log': u"Exception [EclipseLink-4002] (Eclipse Persistence Services - 2.6.2.v20151217-774c696): org.eclipse.persistence.exceptions.DatabaseException\nInternal Exception: com.mysql.cj.jdbc.exceptions.MysqlDataTruncation: Data truncation: Data too long for column 'public_host_name' at row 1\nError Code: 1406\nCall: UPDATE hosts SET ipv4 = ?, ipv6 = ?, cpu_count = ?, ph_cpu_count = ?, total_mem = ?, os_arch = ?, os_type = ?, host_attributes = ?, public_host_name = ? WHERE (host_id = ?)\n\tbind => [10 parameters bound]\nQuery: UpdateObjectQuery(org.apache.ambari.server.orm.entities.HostStateEntity@8b6494c6)", u'id': -1}
INFO 2020-07-29 21:37:12,228 HeartbeatThread.py:132 - Registration response received
DEBUG 2020-07-29 21:37:12,229 HeartbeatThread.py:133 - Registration response is {u'status': u'OK', u'exitstatus': 1, u'log': u"Exception [EclipseLink-4002] (Eclipse Persistence Services - 2.6.2.v20151217-774c696): org.eclipse.persistence.exceptions.DatabaseException\nInternal Exception: com.mysql.cj.jdbc.exceptions.MysqlDataTruncation: Data truncation: Data too long for column 'public_host_name' at row 1\nError Code: 1406\nCall: UPDATE hosts SET ipv4 = ?, ipv6 = ?, cpu_count = ?, ph_cpu_count = ?, total_mem = ?, os_arch = ?, os_type = ?, host_attributes = ?, public_host_name = ? WHERE (host_id = ?)\n\tbind => [10 parameters bound]\nQuery: UpdateObjectQuery(org.apache.ambari.server.orm.entities.HostStateEntity@8b6494c6)", u'id': -1}
ERROR 2020-07-29 21:37:12,229 HeartbeatThread.py:108 - Exception in HeartbeatThread. Re-running the registration

4. And at the Ambari UI i got the same message:

I've done any advise that i'v found in Internet but nothing makes me pass throu Step 3, and I think is because the error shown in the point 1 that i wrote.

One more thing, the reason why I'm writting this, is because I'm Hadoop as the central process on my thesis for my Master Degree; and i've been stock in this problem from 1 month ago and i don't know what to do.

Re: Error in Step 3 Ambari: automatically register hosts

SagarKanani — Fri, 07 Aug 2020 09:25:58 GMT

can you try removing this : force_https_protocol=PROTOCOL_TLSv1_2.
This was true for previous versions of Ambari but probably not for the new versions.
Another try is to try without SSL so as to understand if the communication is broken or the SSL config is a problem.

Hope this helps.

Re: Error in Step 3 Ambari: automatically register hosts

stevenmatison — Fri, 07 Aug 2020 11:47:44 GMT

@jloormoreira

A couple suggestions:

If you are using the hdp sandbox, you should be be able to spin up the vm and have ambari and services installed out of the box. No need to run cluster install wizard. This is the HDP Sandbox: https://www.cloudera.com/downloads/hortonworks-sandbox/hdp.html?utm_source=mktg-tutorial
If you are installing a cluster yourself, you need to back up and complete the documentated steps to install ambari-server & ambari-agent and setup ambari-server. Install as root user and do not do anything for "SSL" until you have base install completed. Complete host mapping for your fqdn (/etc/hosts) in your machine and in the VM. Make sure these are right. Make sure the VM hostname is right and persisting. Next, in the VM complete password less auth steps for the node by creating ssh keys (ssh-keygen) and adding that ~/.ssh/id_rsa.pub key to root ~/.ssh/authorized_keys. Be sure to do the initial login ssh root@yourfqdn and click Y the first time. Now you can go to ambari to install cluster. During cluster install use the id_rsa (not id_rsa.pub) for register host. Once these basics are completed the rest of the install should go fine. My last bit of advice is, if the size of the vm you are using is limited do no try to install all ambari services. It will take a 16-32gb instance to run whole stack and even this is almost too much for single node. In the 8-16gb range you will have issues trying to install everything so recommend installing only basics (yarn,hdfs,ambari metrics) + the other components you need.

Re: Error in Step 3 Ambari: automatically register hosts

jloormoreira — Sat, 08 Aug 2020 04:28:09 GMT

Hi everyone...

I have to thanks to all of you that pass throu this post, specially for the two guys that response to the problem.

It was like 10 times that i recreated everything, from the virtual machine to the wizard on Ambari, but always got the same error: The point 3 on my first post; always got a HTML as a response when i was creating the cluster.

And i felt like something else was missing in the tutorial that Cloudera has in its web portal: https://docs.cloudera.com/HDPDocuments/Ambari-2.7.5.0/bk_ambari-installation/content/install-mysql.html.

Finally i found out a post somewhere in the internet; where some guy explain that is necesary to give excecution permision to the files hostname.sh and public_host_name.sh that is/are necesaries in the Ambari's setting ini file if you aren't doing manualy creation of the hosts; and after that voilà.

I think, somehow Cloudera needs to put this point into its tutorial; because i found like a million post asking for the same problem, even here in cloudera: https://community.cloudera.com/t5/Support-Questions/I-am-not-able-to-register-the-host-Error-registration-with/td-p/237818

I hope that with this post everyone else that experience the same problem, use this advice and make possible the creation of the cluster.

Re: Error in Step 3 Ambari: automatically register hosts

jloormoreira — Sat, 08 Aug 2020 04:29:43 GMT

Thanks for your point and if you got time, please read the solution that i found out somewhere in the internet.

Re: Error in Step 3 Ambari: automatically register hosts

jloormoreira — Sat, 08 Aug 2020 04:31:34 GMT

Thanks for your point and if you got time, please read the solution that i found out somewhere in the internet.