Error in Step 3 Ambari: automatically register hosts
Hi everyone.
Before anything I'm going to describe the environment where I'm trying to install the Cluster on Amabri:
Static hostname: sandbox-hdp.hortonworks.com
Icon name: computer-vm
Chassis: vm
Machine ID: d8323e61d06a489f97b6175ebe8ff9c7
Boot ID: a0b2c8c30d5345f796a51dc754e807e9
Virtualization: vmware
Operating System: CentOS Linux 7 (Core)
CPE OS Name: cpe:/o:centos:centos:7
Kernel: Linux 3.10.0-1127.13.1.el7.x86_64
Architecture: x86-64
Amabri Agent Version:
ambari-agent.x86_64 2.7.4.0-118
Ambari Server Version:
ambari-server.x86_64 2.7.4.0-118
Ambari Agent ini file:
[server]
hostname=sandbox-hdp.hortonworks.com
url_port=8440
secured_url_port=8441
connect_retry_delay=10
max_reconnect_retry_delay=30
[agent]
....
public_hostname_script=/var/lib/ambari-agent/hostname.sh
hostname_script=/var/lib/ambari-agent/hostname.sh
[security]
....
force_https_protocol=PROTOCOL_TLSv1_2
/var/lib/ambari-agent/hostname.sh
#!/bin/bash
#hostname -f <===== I've tried using this
echo 'sandbox-hdp.hortonworks.com'
/etc/sysconfig/network:
NETWORKING=yes
NETWORKING_IPV6=yes
HOSTNAME=sandbox-hdp.hortonworks.com
/etc/hosts
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.100.183 sandbox-hdp.hortonworks.com sandbox-hdp
I think thats all the config information about the environment. Amabri Sever and Agents work perfect, i can access to Ambari UI with the user admin and password admin; even there I created the ssh key for the server without problem and using WinSCP I can access to any folder of the server because I use the root user. I need to mention that everything is in one server, only one machine has ther server and the agent installed.
Until there no problem at all; but when I want to create a cluster using the Wizard of Ambari; in the step 3 is where i got the problem:
1. In the /var/log/ambari-agent/ambari-agent.log file always i got this message:
WARNING 2020-07-29 21:37:12,212 hostname.py:90 - Execution of '/var/lib/ambari-agent/hostname.sh' returned 126. /bin/sh: /var/lib/ambari-agent/hostname.sh: Permission denied.
2. At the same log file, got not problem with the secured ports:
INFO 2020-07-29 21:37:11,597 main.py:155 - loglevel=logging.INFO
INFO 2020-07-29 21:37:11,599 Hardware.py:68 - Initializing host system information.
INFO 2020-07-29 21:37:11,605 Hardware.py:188 - Some mount points were ignored: /dev, /dev/shm, /run, /sys/fs/cgroup, /boot, /home, /run/user/0
INFO 2020-07-29 21:37:11,617 Facter.py:202 - Directory: '/etc/resource_overrides' does not exist - it won't be used for gathering system resources.
INFO 2020-07-29 21:37:11,620 Hardware.py:73 - Host system information: {'kernel': 'Linux', 'domain': 'hortonworks.com', 'physicalprocessorcount': 4, 'kernelrelease': '3.10.0-1127.13.1.el7.x86_64', 'uptime_days': '0', 'memorytotal': 12137760, 'swapfree': '5.87 GB', 'memorysize': 12137760, 'osfamily': 'redhat', 'swapsize': '5.87 GB', 'processorcount': 4, 'netmask': '255.255.255.0', 'timezone': '-05', 'hardwareisa': 'x86_64', 'memoryfree': 10153096, 'operatingsystem': 'centos', 'kernelmajversion': '3.10', 'kernelversion': '3.10.0', 'macaddress': '00:0C:29:01:4A:A8', 'operatingsystemrelease': '7.8.2003', 'ipaddress': '192.168.100.183', 'hostname': 'sandbox-hdp', 'uptime_hours': '20', 'fqdn': 'sandbox-hdp.hortonworks.com', 'id': 'root', 'architecture': 'x86_64', 'selinux': False, 'mounts': [{'available': '45682904', 'used': '6720296', 'percent': '13%', 'device': '/dev/mapper/centos-root', 'mountpoint': '/', 'type': 'xfs', 'size': '52403200'}], 'hardwaremodel': 'x86_64', 'uptime_seconds': '74504', 'interfaces': 'ens192,lo'}
WARNING 2020-07-29 21:37:11,621 shell.py:826 - can not switch user for RUN_COMMAND.
WARNING 2020-07-29 21:37:11,625 shell.py:826 - can not switch user for RUN_COMMAND.
WARNING 2020-07-29 21:37:11,729 shell.py:826 - can not switch user for RUN_COMMAND.
INFO 2020-07-29 21:37:11,734 main.py:308 - Agent died gracefully, exiting.
INFO 2020-07-29 21:37:11,734 ExitHelper.py:57 - Performing cleanup before exiting...
INFO 2020-07-29 21:37:11,734 AlertSchedulerHandler.py:159 - [AlertScheduler] Stopped the alert scheduler.
INFO 2020-07-29 21:37:11,735 AlertSchedulerHandler.py:159 - [AlertScheduler] Stopped the alert scheduler.
INFO 2020-07-29 21:37:11,985 main.py:155 - loglevel=logging.INFO
INFO 2020-07-29 21:37:11,988 Hardware.py:68 - Initializing host system information.
INFO 2020-07-29 21:37:11,992 Hardware.py:188 - Some mount points were ignored: /dev, /dev/shm, /run, /sys/fs/cgroup, /boot, /home, /run/user/0
INFO 2020-07-29 21:37:12,004 Facter.py:202 - Directory: '/etc/resource_overrides' does not exist - it won't be used for gathering system resources.
INFO 2020-07-29 21:37:12,007 Hardware.py:73 - Host system information: {'kernel': 'Linux', 'domain': 'hortonworks.com', 'physicalprocessorcount': 4, 'kernelrelease': '3.10.0-1127.13.1.el7.x86_64', 'uptime_days': '0', 'memorytotal': 12137760, 'swapfree': '5.87 GB', 'memorysize': 12137760, 'osfamily': 'redhat', 'swapsize': '5.87 GB', 'processorcount': 4, 'netmask': '255.255.255.0', 'timezone': '-05', 'hardwareisa': 'x86_64', 'memoryfree': 10175244, 'operatingsystem': 'centos', 'kernelmajversion': '3.10', 'kernelversion': '3.10.0', 'macaddress': '00:0C:29:01:4A:A8', 'operatingsystemrelease': '7.8.2003', 'ipaddress': '192.168.100.183', 'hostname': 'sandbox-hdp', 'uptime_hours': '20', 'fqdn': 'sandbox-hdp.hortonworks.com', 'id': 'root', 'architecture': 'x86_64', 'selinux': False, 'mounts': [{'available': '45687172', 'used': '6716028', 'percent': '13%', 'device': '/dev/mapper/centos-root', 'mountpoint': '/', 'type': 'xfs', 'size': '52403200'}], 'hardwaremodel': 'x86_64', 'uptime_seconds': '74504', 'interfaces': 'ens192,lo'}
INFO 2020-07-29 21:37:12,009 DataCleaner.py:39 - Data cleanup thread started
INFO 2020-07-29 21:37:12,009 DataCleaner.py:120 - Data cleanup started
INFO 2020-07-29 21:37:12,010 DataCleaner.py:122 - Data cleanup finished
WARNING 2020-07-29 21:37:12,011 hostname.py:63 - Unexpected error while retrieving hostname: '(<type 'exceptions.OSError'>, OSError(13, 'Permission denied'), <traceback object at 0x7f4ff6f46fc8>)', defaulting to socket.getfqdn()
INFO 2020-07-29 21:37:12,011 hostname.py:64 - Read hostname 'sandbox-hdp.hortonworks.com' using socket.getfqdn().
INFO 2020-07-29 21:37:12,014 PingPortListener.py:50 - Ping port listener started on port: 8670
INFO 2020-07-29 21:37:12,015 main.py:191 - Newloglevel=logging.DEBUG
INFO 2020-07-29 21:37:12,016 main.py:481 - Connecting to Ambari server at https://sandbox-hdp.hortonworks.com:8440 (192.168.100.183)
DEBUG 2020-07-29 21:37:12,016 NetUtil.py:104 - Trying to connect to https://sandbox-hdp.hortonworks.com:8440
INFO 2020-07-29 21:37:12,016 NetUtil.py:61 - Connecting to https://sandbox-hdp.hortonworks.com:8440/ca
DEBUG 2020-07-29 21:37:12,065 NetUtil.py:81 - GET https://sandbox-hdp.hortonworks.com:8440/ca -> 200, body:
INFO 2020-07-29 21:37:12,066 main.py:491 - Connected to Ambari server sandbox-hdp.hortonworks.com
INFO 2020-07-29 21:37:12,066 AlertSchedulerHandler.py:149 - [AlertScheduler] Starting <ambari_agent.apscheduler.scheduler.Scheduler object at 0x7f4ff6f3c8d0>; currently running: False
INFO 2020-07-29 21:37:12,067 NetUtil.py:61 - Connecting to https://sandbox-hdp.hortonworks.com:8440/connection_info
DEBUG 2020-07-29 21:37:12,116 NetUtil.py:81 - GET https://sandbox-hdp.hortonworks.com:8440/connection_info -> 200, body: {"security.server.two_way_ssl":"true"}
DEBUG 2020-07-29 21:37:12,117 security.py:56 - Server two-way SSL authentication required: True
INFO 2020-07-29 21:37:12,117 security.py:59 - Server require two-way SSL authentication. Use it instead of one-way...
INFO 2020-07-29 21:37:12,117 security.py:61 - Connecting to wss://sandbox-hdp.hortonworks.com:8441/agent/stomp/v1
INFO 2020-07-29 21:37:12,117 security.py:224 - Server certicate exists, ok
INFO 2020-07-29 21:37:12,117 security.py:232 - Agent key exists, ok
INFO 2020-07-29 21:37:12,117 security.py:240 - Agent certificate exists, ok
INFO 2020-07-29 21:37:12,169 transport.py:329 - Starting receiver loop
DEBUG 2020-07-29 21:37:12,169 transport.py:269 - Sending frame: ['STOMP', '\n', 'accept-version:1.2\n', 'host:0\n', '\n', '\x00']
3. At the same log file, I always get problem with long data at the time that Ambari is trying to register the host:
WARNING 2020-07-29 21:37:12,212 hostname.py:90 - Execution of '/var/lib/ambari-agent/hostname.sh' returned 126. /bin/sh: /var/lib/ambari-agent/hostname.sh: Permission denied
INFO 2020-07-29 21:37:12,220 hostname.py:103 - Read public hostname '<!doctype html public "-//w3c//dtd xhtml 1.0 transitional//en" "http://www.w3.org/tr/xhtml1/dtd/xhtml1-transitional.dtd">^M
<html xmlns="http://www.w3.org/1999/xhtml">^M
<head>^M
<meta content="text/html; charset=utf-8" http-equiv="content-type" />^M
<meta content="no-cache" http-equiv="pragma" />^M
<title>waiting...</title>^M
<script type="text/javascript">^M
var pagename = '/';^M
top.location.replace(pagename);^M
</script>^M
</head>^M
<body> </body>^M
</html>^M
' from http://169.254.169.254/latest/meta-data/public-hostname
INFO 2020-07-29 21:37:12,220 HeartbeatThread.py:127 - Sending registration request
....
DEBUG 2020-07-29 21:37:12,227 transport.py:189 - Received frame: 'MESSAGE', headers={'content-length': '648', 'destination': '/user/', 'message-id': '1d065200-6399-190c-1707-13715074fb75-50', 'content-type': 'application/json;charset=UTF-8', 'correlationId': '0', 'subscription': 'sub'}, body='{"status":"OK","exitstatus":1,"log":"Exception [EclipseLink-4002] (Eclipse Persistence Services - 2.6.2.v20151217-774c696): org.eclipse.persistence.exceptions.DatabaseException\\nInternal Exception: com.mysql.cj.jdbc.exceptions.MysqlDataTruncation: Data truncation: Data too long for column \'public_host_name\' at row 1\\nError Code: 1406\\nCall: UPDATE hosts SET ipv4 = ?, ipv6 = ?, cpu_count = ?, ph_cpu_count = ?, total_mem = ?, os_arch = ?, os_type = ?, host_attributes = ?, public_host_name = ? WHERE (host_id = ?)\\n\\tbind => [10 parameters bound]\\nQuery: UpdateObjectQuery(org.apache.ambari.server.orm.entities.HostStateEntity@8b6494c6)","id":-1}'
INFO 2020-07-29 21:37:12,227 __init__.py:82 - Event from server at /user/ (correlation_id=0): {u'status': u'OK', u'exitstatus': 1, u'log': u"Exception [EclipseLink-4002] (Eclipse Persistence Services - 2.6.2.v20151217-774c696): org.eclipse.persistence.exceptions.DatabaseException\nInternal Exception: com.mysql.cj.jdbc.exceptions.MysqlDataTruncation: Data truncation: Data too long for column 'public_host_name' at row 1\nError Code: 1406\nCall: UPDATE hosts SET ipv4 = ?, ipv6 = ?, cpu_count = ?, ph_cpu_count = ?, total_mem = ?, os_arch = ?, os_type = ?, host_attributes = ?, public_host_name = ? WHERE (host_id = ?)\n\tbind => [10 parameters bound]\nQuery: UpdateObjectQuery(org.apache.ambari.server.orm.entities.HostStateEntity@8b6494c6)", u'id': -1}
INFO 2020-07-29 21:37:12,228 HeartbeatThread.py:132 - Registration response received
DEBUG 2020-07-29 21:37:12,229 HeartbeatThread.py:133 - Registration response is {u'status': u'OK', u'exitstatus': 1, u'log': u"Exception [EclipseLink-4002] (Eclipse Persistence Services - 2.6.2.v20151217-774c696): org.eclipse.persistence.exceptions.DatabaseException\nInternal Exception: com.mysql.cj.jdbc.exceptions.MysqlDataTruncation: Data truncation: Data too long for column 'public_host_name' at row 1\nError Code: 1406\nCall: UPDATE hosts SET ipv4 = ?, ipv6 = ?, cpu_count = ?, ph_cpu_count = ?, total_mem = ?, os_arch = ?, os_type = ?, host_attributes = ?, public_host_name = ? WHERE (host_id = ?)\n\tbind => [10 parameters bound]\nQuery: UpdateObjectQuery(org.apache.ambari.server.orm.entities.HostStateEntity@8b6494c6)", u'id': -1}
ERROR 2020-07-29 21:37:12,229 HeartbeatThread.py:108 - Exception in HeartbeatThread. Re-running the registration
4. And at the Ambari UI i got the same message:
I've done any advise that i'v found in Internet but nothing makes me pass throu Step 3, and I think is because the error shown in the point 1 that i wrote.
One more thing, the reason why I'm writting this, is because I'm Hadoop as the central process on my thesis for my Master Degree; and i've been stock in this problem from 1 month ago and i don't know what to do.
