Thanks @Jay Kumar SenSharma. Question: On other nodes, do we have to do anything after installing new version of JAVA. Just by putting the JAVA in the same path as Ambari-server, is that going to be ok..? ... View more

Thank you @Jay Kumar SenSharma It did fixed the problem but wondering why it was working before and stopped working after a VM restart ... View more

@Jay Kumar SenSharma Thank you. Not sure why is this setup like that. security.server.two_way_ssl = true I disabled it and restarted ambari-server. On the host machine I added property "force_https_protocol=PROTOCOL_TLSv1_2" and then I tried registering again BUT this time it did recognized and started registering but then in the review process I got the error below and registration process is stuck at "PREPARING TO DEPLOY: 14 of 14 task COMPLETED" Should I re-image the machine to SUSE LINUX 12 SP1 or SP2 ( per document HDP only supports till SP2) An internal system exception occurred: Trying to map host to cluster where stack does not support host's os type, clusterName=hadoop, clusterStackId=HDP-2.4, hostname=ds.asotc.com, hostOsFamily=suse12 ... View more

@Jay Kumar SenSharma Looks like it got updated now ds:/var/log/ambari-agent # tail -50 /var/log/ambari-agent/ambari-agent.log INFO 2018-04-24 10:55:27,940 security.py:55 - Server require two-way SSL authentication. Use it instead of one-way... INFO 2018-04-24 10:55:27,941 security.py:179 - Server certicate not exists, downloading INFO 2018-04-24 10:55:27,941 security.py:202 - Downloading server cert from https://ambari.asotc.com:8440/cert/ca/ ERROR 2018-04-24 10:55:27,949 Controller.py:226 - Unable to connect to: https://ambari.asotc.com:8441/agent/v1/register/ds.asotc.com Traceback (most recent call last): File "/usr/lib/python2.6/site-packages/ambari_agent/Controller.py", line 175, in registerWithServer ret = self.sendRequest(self.registerUrl, data) File "/usr/lib/python2.6/site-packages/ambari_agent/Controller.py", line 545, in sendRequest raise IOError('Request to {0} failed due to {1}'.format(url, str(exception))) IOError: Request to https://ambari.asotc.com:8441/agent/v1/register/ds.asotc.com failed due to <urlopen error EOF occurred in violation of protocol (_ssl.c:661)> ERROR 2018-04-24 10:55:27,949 Controller.py:227 - Error:Request to https://ambari.asotc.com:8441/agent/v1/register/ds.asotc.com failed due to <urlopen error EOF occurred in violation of protocol (_ssl.c:661)> WARNING 2018-04-24 10:55:27,949 Controller.py:228 - Sleeping for 26 seconds and then trying again DEBUG 2018-04-24 10:55:54,143 HostCheckReportFileHandler.py:126 - Host check report at /var/lib/ambari-agent/data/hostcheck.result DEBUG 2018-04-24 10:55:54,144 HostCheckReportFileHandler.py:177 - Removing old host check file at /var/lib/ambari-agent/data/hostcheck.result DEBUG 2018-04-24 10:55:54,145 HostCheckReportFileHandler.py:182 - Creating host check file at /var/lib/ambari-agent/data/hostcheck.result INFO 2018-04-24 10:55:54,158 Controller.py:170 - Registering with ds.asotc.com (172.16.1.67) (agent='{"hardwareProfile": {"kernel": "Linux", "domain": "asotc.com", "physicalprocessorcount": 1, "kernelrelease": "4.4.73-7-default", "uptime_days": "0", "memorytotal": 8068504, "swapfree": "2.00 GB", "memorysize": 8068504, "osfamily": "suse", "swapsize": "2.00 GB", "processorcount": 1, "netmask": "255.255.255.0", "timezone": "EST", "hardwareisa": "x86_64", "memoryfree": 6838720, "operatingsystem": "sles", "kernelmajversion": "4.4", "kernelversion": "4.4.73", "macaddress": "00:15:5D:01:C8:19", "operatingsystemrelease": "12", "ipaddress": "172.16.1.67", "hostname": "ds", "uptime_hours": "1", "fqdn": "ds.asotc.com", "id": "root", "architecture": "x86_64", "selinux": false, "mounts": [{"available": "4023372", "used": "0", "percent": "0%", "device": "devtmpfs", "mountpoint": "/dev", "type": "devtmpfs", "size": "4023372"}, {"available": "55906100", "used": "5387708", "percent": "9%", "device": "/dev/mapper/system-root", "mountpoint": "/", "type": "btrfs", "size": "62914560"}, {"available": "55906100", "used": "5387708", "percent": "9%", "device": "/dev/mapper/system-root", "mountpoint": "/.snapshots", "type": "btrfs", "size": "62914560"}, {"available": "55906100", "used": "5387708", "percent": "9%", "device": "/dev/mapper/system-root", "mountpoint": "/var/tmp", "type": "btrfs", "size": "62914560"}, {"available": "55906100", "used": "5387708", "percent": "9%", "device": "/dev/mapper/system-root", "mountpoint": "/home", "type": "btrfs", "size": "62914560"}, {"available": "55906100", "used": "5387708", "percent": "9%", "device": "/dev/mapper/system-root", "mountpoint": "/var/lib/mysql", "type": "btrfs", "size": "62914560"}, {"available": "55906100", "used": "5387708", "percent": "9%", "device": "/dev/mapper/system-root", "mountpoint": "/boot/grub2/x86_64-efi", "type": "btrfs", "size": "62914560"}, {"available": "55906100", "used": "5387708", "percent": "9%", "device": "/dev/mapper/system-root", "mountpoint": "/opt", "type": "btrfs", "size": "62914560"}, {"available": "55906100", "used": "5387708", "percent": "9%", "device": "/dev/mapper/system-root", "mountpoint": "/tmp", "type": "btrfs", "size": "62914560"}, {"available": "55906100", "used": "5387708", "percent": "9%", "device": "/dev/mapper/system-root", "mountpoint": "/srv", "type": "btrfs", "size": "62914560"}, {"available": "55906100", "used": "5387708", "percent": "9%", "device": "/dev/mapper/system-root", "mountpoint": "/var/spool", "type": "btrfs", "size": "62914560"}, {"available": "55906100", "used": "5387708", "percent": "9%", "device": "/dev/mapper/system-root", "mountpoint": "/usr/local", "type": "btrfs", "size": "62914560"}, {"available": "55906100", "used": "5387708", "percent": "9%", "device": "/dev/mapper/system-root", "mountpoint": "/var/lib/libvirt/images", "type": "btrfs", "size": "62914560"}, {"available": "55906100", "used": "5387708", "percent": "9%", "device": "/dev/mapper/system-root", "mountpoint": "/var/lib/mailman", "type": "btrfs", "size": "62914560"}, {"available": "55906100", "used": "5387708", "percent": "9%", "device": "/dev/mapper/system-root", "mountpoint": "/var/lib/machines", "type": "btrfs", "size": "62914560"}, {"available": "55906100", "used": "5387708", "percent": "9%", "device": "/dev/mapper/system-root", "mountpoint": "/var/log", "type": "btrfs", "size": "62914560"}, {"available": "55906100", "used": "5387708", "percent": "9%", "device": "/dev/mapper/system-root", "mountpoint": "/var/crash", "type": "btrfs", "size": "62914560"}, {"available": "55906100", "used": "5387708", "percent": "9%", "device": "/dev/mapper/system-root", "mountpoint": "/var/lib/pgsql", "type": "btrfs", "size": "62914560"}, {"available": "55906100", "used": "5387708", "percent": "9%", "device": "/dev/mapper/system-root", "mountpoint": "/boot/grub2/i386-pc", "type": "btrfs", "size": "62914560"}, {"available": "55906100", "used": "5387708", "percent": "9%", "device": "/dev/mapper/system-root", "mountpoint": "/var/lib/mariadb", "type": "btrfs", "size": "62914560"}, {"available": "55906100", "used": "5387708", "percent": "9%", "device": "/dev/mapper/system-root", "mountpoint": "/var/lib/named", "type": "btrfs", "size": "62914560"}, {"available": "55906100", "used": "5387708", "percent": "9%", "device": "/dev/mapper/system-root", "mountpoint": "/var/opt", "type": "btrfs", "size": "62914560"}, {"available": "55906100", "used": "5387708", "percent": "9%", "device": "/dev/mapper/system-root", "mountpoint": "/var/cache", "type": "btrfs", "size": "62914560"}], "hardwaremodel": "x86_64", "uptime_seconds": "6573", "interfaces": "eth0,lo"}, "currentPingPort": 8670, "prefix": "/var/lib/ambari-agent/data", "agentVersion": "2.5.1.0", "agentEnv": {"transparentHugePage": "", "hostHealth": {"agentTimeStampAtReporting": 1524581754145, "activeJavaProcs": [], "liveServices": [{"status": "Healthy", "name": "ntpd or ntp", "desc": ""}]}, "reverseLookup": true, "alternatives": [], "hasUnlimitedJcePolicy": null, "umask": "18", "firewallName": "rcSuSEfirewall2", "stackFoldersAndFiles": [], "existingUsers": [], "firewallRunning": false}, "timestamp": 1524581753950, "hostname": "ds.asotc.com", "responseId": -1, "publicHostname": "ds.asotc.com"}') INFO 2018-04-24 10:55:54,162 NetUtil.py:70 - Connecting to https://ambari.asotc.com:8440/connection_info DEBUG 2018-04-24 10:55:54,335 NetUtil.py:90 - GET https://ambari.asotc.com:8440/connection_info -> 200, body: {"security.server.two_way_ssl":"true"} DEBUG 2018-04-24 10:55:54,336 security.py:52 - Server two-way SSL authentication required: True INFO 2018-04-24 10:55:54,336 security.py:55 - Server require two-way SSL authentication. Use it instead of one-way... INFO 2018-04-24 10:55:54,337 security.py:179 - Server certicate not exists, downloading INFO 2018-04-24 10:55:54,337 security.py:202 - Downloading server cert from https://ambari.asotc.com:8440/cert/ca/ ERROR 2018-04-24 10:55:54,345 Controller.py:226 - Unable to connect to: https://ambari.asotc.com:8441/agent/v1/register/ds.asotc.com Traceback (most recent call last): File "/usr/lib/python2.6/site-packages/ambari_agent/Controller.py", line 175, in registerWithServer ret = self.sendRequest(self.registerUrl, data) File "/usr/lib/python2.6/site-packages/ambari_agent/Controller.py", line 545, in sendRequest raise IOError('Request to {0} failed due to {1}'.format(url, str(exception))) IOError: Request to https://ambari.asotc.com:8441/agent/v1/register/ds.asotc.com failed due to <urlopen error EOF occurred in violation of protocol (_ssl.c:661)> ERROR 2018-04-24 10:55:54,345 Controller.py:227 - Error:Request to https://ambari.asotc.com:8441/agent/v1/register/ds.asotc.com failed due to <urlopen error EOF occurred in violation of protocol (_ssl.c:661)> WARNING 2018-04-24 10:55:54,345 Controller.py:228 - Sleeping for 25 seconds and then trying again DEBUG 2018-04-24 10:56:19,552 HostCheckReportFileHandler.py:126 - Host check report at /var/lib/ambari-agent/data/hostcheck.result DEBUG 2018-04-24 10:56:19,553 HostCheckReportFileHandler.py:177 - Removing old host check file at /var/lib/ambari-agent/data/hostcheck.result DEBUG 2018-04-24 10:56:19,553 HostCheckReportFileHandler.py:182 - Creating host check file at /var/lib/ambari-agent/data/hostcheck.result INFO 2018-04-24 10:56:19,558 Controller.py:170 - Registering with ds.asotc.com (172.16.1.67) (agent='{"hardwareProfile": {"kernel": "Linux", "domain": "asotc.com", "physicalprocessorcount": 1, "kernelrelease": "4.4.73-7-default", "uptime_days": "0", "memorytotal": 8068504, "swapfree": "2.00 GB", "memorysize": 8068504, "osfamily": "suse", "swapsize": "2.00 GB", "processorcount": 1, "netmask": "255.255.255.0", "timezone": "EST", "hardwareisa": "x86_64", "memoryfree": 6838720, "operatingsystem": "sles", "kernelmajversion": "4.4", "kernelversion": "4.4.73", "macaddress": "00:15:5D:01:C8:19", "operatingsystemrelease": "12", "ipaddress": "172.16.1.67", "hostname": "ds", "uptime_hours": "1", "fqdn": "ds.asotc.com", "id": "root", "architecture": "x86_64", "selinux": false, "mounts": [{"available": "4023372", "used": "0", "percent": "0%", "device": "devtmpfs", "mountpoint": "/dev", "type": "devtmpfs", "size": "4023372"}, {"available": "55906100", "used": "5387708", "percent": "9%", "device": "/dev/mapper/system-root", "mountpoint": "/", "type": "btrfs", "size": "62914560"}, {"available": "55906100", "used": "5387708", "percent": "9%", "device": "/dev/mapper/system-root", "mountpoint": "/.snapshots", "type": "btrfs", "size": "62914560"}, {"available": "55906100", "used": "5387708", "percent": "9%", "device": "/dev/mapper/system-root", "mountpoint": "/var/tmp", "type": "btrfs", "size": "62914560"}, {"available": "55906100", "used": "5387708", "percent": "9%", "device": "/dev/mapper/system-root", "mountpoint": "/home", "type": "btrfs", "size": "62914560"}, {"available": "55906100", "used": "5387708", "percent": "9%", "device": "/dev/mapper/system-root", "mountpoint": "/var/lib/mysql", "type": "btrfs", "size": "62914560"}, {"available": "55906100", "used": "5387708", "percent": "9%", "device": "/dev/mapper/system-root", "mountpoint": "/boot/grub2/x86_64-efi", "type": "btrfs", "size": "62914560"}, {"available": "55906100", "used": "5387708", "percent": "9%", "device": "/dev/mapper/system-root", "mountpoint": "/opt", "type": "btrfs", "size": "62914560"}, {"available": "55906100", "used": "5387708", "percent": "9%", "device": "/dev/mapper/system-root", "mountpoint": "/tmp", "type": "btrfs", "size": "62914560"}, {"available": "55906100", "used": "5387708", "percent": "9%", "device": "/dev/mapper/system-root", "mountpoint": "/srv", "type": "btrfs", "size": "62914560"}, {"available": "55906100", "used": "5387708", "percent": "9%", "device": "/dev/mapper/system-root", "mountpoint": "/var/spool", "type": "btrfs", "size": "62914560"}, {"available": "55906100", "used": "5387708", "percent": "9%", "device": "/dev/mapper/system-root", "mountpoint": "/usr/local", "type": "btrfs", "size": "62914560"}, {"available": "55906100", "used": "5387708", "percent": "9%", "device": "/dev/mapper/system-root", "mountpoint": "/var/lib/libvirt/images", "type": "btrfs", "size": "62914560"}, {"available": "55906100", "used": "5387708", "percent": "9%", "device": "/dev/mapper/system-root", "mountpoint": "/var/lib/mailman", "type": "btrfs", "size": "62914560"}, {"available": "55906100", "used": "5387708", "percent": "9%", "device": "/dev/mapper/system-root", "mountpoint": "/var/lib/machines", "type": "btrfs", "size": "62914560"}, {"available": "55906100", "used": "5387708", "percent": "9%", "device": "/dev/mapper/system-root", "mountpoint": "/var/log", "type": "btrfs", "size": "62914560"}, {"available": "55906100", "used": "5387708", "percent": "9%", "device": "/dev/mapper/system-root", "mountpoint": "/var/crash", "type": "btrfs", "size": "62914560"}, {"available": "55906100", "used": "5387708", "percent": "9%", "device": "/dev/mapper/system-root", "mountpoint": "/var/lib/pgsql", "type": "btrfs", "size": "62914560"}, {"available": "55906100", "used": "5387708", "percent": "9%", "device": "/dev/mapper/system-root", "mountpoint": "/boot/grub2/i386-pc", "type": "btrfs", "size": "62914560"}, {"available": "55906100", "used": "5387708", "percent": "9%", "device": "/dev/mapper/system-root", "mountpoint": "/var/lib/mariadb", "type": "btrfs", "size": "62914560"}, {"available": "55906100", "used": "5387708", "percent": "9%", "device": "/dev/mapper/system-root", "mountpoint": "/var/lib/named", "type": "btrfs", "size": "62914560"}, {"available": "55906100", "used": "5387708", "percent": "9%", "device": "/dev/mapper/system-root", "mountpoint": "/var/opt", "type": "btrfs", "size": "62914560"}, {"available": "55906100", "used": "5387708", "percent": "9%", "device": "/dev/mapper/system-root", "mountpoint": "/var/cache", "type": "btrfs", "size": "62914560"}], "hardwaremodel": "x86_64", "uptime_seconds": "6573", "interfaces": "eth0,lo"}, "currentPingPort": 8670, "prefix": "/var/lib/ambari-agent/data", "agentVersion": "2.5.1.0", "agentEnv": {"transparentHugePage": "", "hostHealth": {"agentTimeStampAtReporting": 1524581779553, "activeJavaProcs": [], "liveServices": [{"status": "Healthy", "name": "ntpd or ntp", "desc": ""}]}, "reverseLookup": true, "alternatives": [], "hasUnlimitedJcePolicy": null, "umask": "18", "firewallName": "rcSuSEfirewall2", "stackFoldersAndFiles": [], "existingUsers": [], "firewallRunning": false}, "timestamp": 1524581779358, "hostname": "ds.asotc.com", "responseId": -1, "publicHostname": "ds.asotc.com"}') INFO 2018-04-24 10:56:19,561 NetUtil.py:70 - Connecting to https://ambari.asotc.com:8440/connection_info DEBUG 2018-04-24 10:56:19,683 NetUtil.py:90 - GET https://ambari.asotc.com:8440/connection_info -> 200, body: {"security.server.two_way_ssl":"true"} DEBUG 2018-04-24 10:56:19,684 security.py:52 - Server two-way SSL authentication required: True INFO 2018-04-24 10:56:19,684 security.py:55 - Server require two-way SSL authentication. Use it instead of one-way... INFO 2018-04-24 10:56:19,684 security.py:179 - Server certicate not exists, downloading INFO 2018-04-24 10:56:19,684 security.py:202 - Downloading server cert from https://ambari.asotc.com:8440/cert/ca/ ERROR 2018-04-24 10:56:19,693 Controller.py:226 - Unable to connect to: https://ambari.asotc.com:8441/agent/v1/register/ds.asotc.com Traceback (most recent call last): File "/usr/lib/python2.6/site-packages/ambari_agent/Controller.py", line 175, in registerWithServer ret = self.sendRequest(self.registerUrl, data) File "/usr/lib/python2.6/site-packages/ambari_agent/Controller.py", line 545, in sendRequest raise IOError('Request to {0} failed due to {1}'.format(url, str(exception))) IOError: Request to https://ambari.asotc.com:8441/agent/v1/register/ds.asotc.com failed due to <urlopen error EOF occurred in violation of protocol (_ssl.c:661)> ERROR 2018-04-24 10:56:19,693 Controller.py:227 - Error:Request to https://ambari.asotc.com:8441/agent/v1/register/ds.asotc.com failed due to <urlopen error EOF occurred in violation of protocol (_ssl.c:661)> WARNING 2018-04-24 10:56:19,694 Controller.py:228 - Sleeping for 28 seconds and then trying again ... View more

@Jay Kumar SenSharma time stamp are changing but file is not getting updated. This is weird: ds:/var/log/ambari-agent # ls -l total 2112 -rw-r--r-- 1 root root 2158132 Apr 24 10:47 ambari-agent.log -rw-r--r-- 1 root root 0 Apr 24 10:47 ambari-agent.out ... View more

@Jay Kumar SenSharma Changed the INI to Debug and restarted the ambari-agent. Reason of why I am having SUSE LINUX - Our cluster is all Centos 7 but we have a new requirement to move DATA from SAP to hadoop and we need BODS ( SAP tool) to be installed on a machine on the hadoop cluster. So we are deploying a new machine in the cluster with SUSE LINUX as BODS requires SUSE LINUX 12 SP13. That's what SAP group are telling me. Event after changing the loglevel to debug, it's still not showing the detail [server] hostname=ambari.asotc.com url_port=8440 secured_url_port=8441 connect_retry_delay=10 max_reconnect_retry_delay=30 [agent] logdir=/var/log/ambari-agent piddir=/var/run/ambari-agent prefix=/var/lib/ambari-agent/data ;loglevel=(DEBUG/INFO) loglevel=DEBUG data_cleanup_interval=86400 data_cleanup_max_age=2592000 data_cleanup_max_size_MB = 100 ping_port=8670 cache_dir=/var/lib/ambari-agent/cache tolerate_download_failures=true run_as_user=root LOG OUTPUT: NFO 2018-04-24 09:23:26,242 main.py:145 - loglevel=logging.INFO INFO 2018-04-24 09:23:26,242 main.py:145 - loglevel=logging.INFO INFO 2018-04-24 09:23:26,242 main.py:145 - loglevel=logging.INFO INFO 2018-04-24 09:23:26,246 DataCleaner.py:39 - Data cleanup thread started INFO 2018-04-24 09:23:26,257 DataCleaner.py:120 - Data cleanup started INFO 2018-04-24 09:23:26,258 DataCleaner.py:122 - Data cleanup finished INFO 2018-04-24 09:23:26,356 PingPortListener.py:50 - Ping port listener started on port: 8670 INFO 2018-04-24 09:23:26,358 main.py:437 - Connecting to Ambari server at https://ambari.asotc.com:8440 (172.16.1.15) INFO 2018-04-24 09:23:26,359 NetUtil.py:70 - Connecting to https://ambari.asotc.com:8440/ca INFO 2018-04-24 09:23:26,759 main.py:447 - Connected to Ambari server ambari.asotc.com INFO 2018-04-24 09:23:26,760 hostname.py:67 - agent:hostname_script configuration not defined thus read hostname 'ds.asotc.com' using socket.getfqdn(). INFO 2018-04-24 09:23:26,761 threadpool.py:58 - Started thread pool with 3 core threads and 20 maximum threads WARNING 2018-04-24 09:23:26,762 AlertSchedulerHandler.py:280 - [AlertScheduler] /var/lib/ambari-agent/cache/alerts/definitions.json not found or invalid. No alerts will be scheduled until registration occurs. INFO 2018-04-24 09:23:26,762 AlertSchedulerHandler.py:175 - [AlertScheduler] Starting <ambari_agent.apscheduler.scheduler.Scheduler object at 0x7f196b693c50>; currently running: False INFO 2018-04-24 09:23:26,950 hostname.py:106 - Read public hostname 'ds.asotc.com' using socket.getfqdn() INFO 2018-04-24 09:23:26,951 Hardware.py:48 - Initializing host system information. INFO 2018-04-24 09:23:26,979 Hardware.py:176 - Some mount points were ignored: /dev/shm, /run, /sys/fs/cgroup, /run/user/0 INFO 2018-04-24 09:23:27,010 hostname.py:67 - agent:hostname_script configuration not defined thus read hostname 'ds.asotc.com' using socket.getfqdn(). INFO 2018-04-24 09:23:27,039 Facter.py:202 - Directory: '/etc/resource_overrides' does not exist - it won't be used for gathering system resources. ... View more