Member since
05-08-2018
12
Posts
0
Kudos Received
0
Solutions
08-13-2018
07:55 AM
We have set up a HDP cluster hosted on Azure VM's. Now we want to integrate Azure AD/ Domain services to authenticate the users for accessing the cluster using their corporate credentials. How can we achieve this.
... View more
Labels:
07-17-2018
11:19 AM
@pdarvasi I was following the instruction provided by Hortonworks: Launch Cloudbreak - Cloudbreak on Azure Marketplace But with link provide by you, I was able to create cloudbreak 2.7.0. Thank you.
... View more
07-16-2018
09:05 AM
I need to create cloudbreak V2.7.0 for hortonworks data platform in Azure, but when I am creating it, I didn't find any options for the cloudbreak version and after the creation I'm getting V 2.4.0. Please suggest how to create cloudbreak 2.7.0.
... View more
Labels:
06-14-2018
09:32 AM
@Jay Kumar SenSharma Thank you for the response. I have run the script threaddump_cpu_with_cmd.sh to collect threaddump. Attached are the two files which are generated by this script: top-highcpu.txt jcmd-threaddump.txt
... View more
06-13-2018
08:52 AM
After launching hive command, it is taking more than 2-3 hours to start the Hive CLI. The screen will have below output for about 2-3 hours and then Hive CLI will start: [hdfs@hdp-m0 ~]$ hive
log4j:WARN No such property [maxFileSize] in org.apache.log4j.DailyRollingFileAppender.
Logging initialized using configuration in file:/etc/hive/2.6.4.5-2/0/hive-log4j.properties
... View more
05-23-2018
09:06 AM
I have created aHDP cluster 2.6.4.5-2 using cloudbreak 2.4.0 in Azure. I am able to successfully login to Ambari UI and most of the services are running fine, but I'm unable to open any of the quick links of all the services from Ambari UI. Below is the error I'm getting: 504 Gateway Timeout Server error - server 137.117.246.41 is unreachable at this moment.
Please retry the request or contact your administrator.
... View more
Labels:
05-15-2018
11:13 AM
@Jay Kumar SenSharma Thank you for you suggestion of checking ambari-agent.log. The connection to port 8440 was refused. I enabled that port and now the registration is successful.
... View more
05-14-2018
11:36 AM
@Geoffrey Shelton Okot I have configured all of the above steps, except one point. In the network configuration file i.e. /etc/sysconfig/networt, we have to modify the below properties but I have Ubuntu machine and it does not have the "sysconfig" directory. NETWORKING=yes
NETWORKING_IPV6=yes
HOSTNAME=fully.qualified.domain.name How to handle this step in Ubuntu. Also I noticed that in the ambari host registration log, below warning ,message is there: WARNING 2018-05-11 08:32:37,715 AlertSchedulerHandler.py:280 - [AlertScheduler] /var/lib/ambari-agent/cache/alerts/definitions.json not found or invalid. No alerts will be scheduled until registration occurs. Is this impacting the host registration?
... View more
05-14-2018
07:07 AM
@Geoffrey Shelton Okot Below is the content of /etc/hosts,it's same in all three hosts: 127.0.0.1 localhost
# The following lines are desirable for IPv6 capable hosts
::1 ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
ff02::3 ip6-allhosts
137.xxx.xxx.xx6 HDPLZTESTVM.xxxxxxxxxx.ax.internal.cloudapp.net HDPLZTESTVM
137.xxx.xxx.xx0 HDPLZVMSN1.xxxxxxxxxx.ax.internal.cloudapp.net HDPLZVMSN1
137.xxx.xxx.xx2 HDPLZVMSN2.xxxxxxxxxx.ax.internal.cloudapp.net HDPLZVMSN2
Below is the entry for server in /etc/ambari-agent/conf/ambari-agent.ini file in all the three hosts: [server]
hostname=HDPLZTESTVM.xxxxxxxxxx.ax.internal.cloudapp.net
url_port=8440
secured_url_port=8441 HDPLZTESTVM is the host where I have installed the ambari-server and also I have installed ambari-agent on all the three hosts. I'm launching the ambari cluster install wizard after starting the ambari-server and ambari-agent, but still the registration is failing for all the host.
... View more
05-11-2018
10:22 AM
I'm trying to create a new cluster with 3 nodes. I have followed all the steps given in the link: https://docs.hortonworks.com/HDPDocuments/Ambari-2.5.1.0/bk_ambari-installation/content/ch_Getting_Ready.html but all the host registration got failed. HDPLZTESTVM is the host where I have installed ambari-server and below is the registration log for this host: ==========================
Creating target directory...
==========================
Command start time 2018-05-11 08:32:01
Connection to hdplztestvm.xxxxxxxxxx.ax.internal.cloudapp.net closed.
SSH command execution finished
host=hdplztestvm.xxxxxxxxxx.ax.internal.cloudapp.net, exitcode=0
Command end time 2018-05-11 08:32:01
==========================
Copying ambari sudo script...
==========================
Command start time 2018-05-11 08:32:01
scp /var/lib/ambari-server/ambari-sudo.sh
host=hdplztestvm.xxxxxxxxxx.ax.internal.cloudapp.net, exitcode=0
Command end time 2018-05-11 08:32:02
==========================
Copying common functions script...
==========================
Command start time 2018-05-11 08:32:02
scp /usr/lib/python2.6/site-packages/ambari_commons
host=hdplztestvm.xxxxxxxxxx.ax.internal.cloudapp.net, exitcode=0
Command end time 2018-05-11 08:32:02
==========================
Copying create-python-wrap script...
==========================
Command start time 2018-05-11 08:32:02
scp /var/lib/ambari-server/create-python-wrap.sh
host=hdplztestvm.xxxxxxxxxx.ax.internal.cloudapp.net, exitcode=0
Command end time 2018-05-11 08:32:02
==========================
Copying OS type check script...
==========================
Command start time 2018-05-11 08:32:02
scp /usr/lib/python2.6/site-packages/ambari_server/os_check_type.py
host=hdplztestvm.xxxxxxxxxx.ax.internal.cloudapp.net, exitcode=0
Command end time 2018-05-11 08:32:02
==========================
Running create-python-wrap script...
==========================
Command start time 2018-05-11 08:32:02
Connection to hdplztestvm.xxxxxxxxxx.ax.internal.cloudapp.net closed.
SSH command execution finished
host=hdplztestvm.xxxxxxxxxx.ax.internal.cloudapp.net, exitcode=0
Command end time 2018-05-11 08:32:03
==========================
Running OS type check...
==========================
Command start time 2018-05-11 08:32:03
Cluster primary/cluster OS family is ubuntu16 and local/current OS family is ubuntu16
Connection to hdplztestvm.xxxxxxxxxx.ax.internal.cloudapp.net closed.
SSH command execution finished
host=hdplztestvm.xxxxxxxxxx.ax.internal.cloudapp.net, exitcode=0
Command end time 2018-05-11 08:32:03
==========================
Checking 'sudo' package on remote host...
==========================
Command start time 2018-05-11 08:32:03
Connection to hdplztestvm.xxxxxxxxxx.ax.internal.cloudapp.net closed.
SSH command execution finished
host=hdplztestvm.xxxxxxxxxx.ax.internal.cloudapp.net, exitcode=0
Command end time 2018-05-11 08:32:03
==========================
Copying repo file to 'tmp' folder...
==========================
Command start time 2018-05-11 08:32:03
scp /etc/apt/sources.list.d/ambari.list
host=hdplztestvm.xxxxxxxxxx.ax.internal.cloudapp.net, exitcode=0
Command end time 2018-05-11 08:32:04
==========================
Moving file to repo dir...
==========================
Command start time 2018-05-11 08:32:04
Connection to hdplztestvm.xxxxxxxxxx.ax.internal.cloudapp.net closed.
SSH command execution finished
host=hdplztestvm.xxxxxxxxxx.ax.internal.cloudapp.net, exitcode=0
Command end time 2018-05-11 08:32:04
==========================
Changing permissions for ambari.repo...
==========================
Command start time 2018-05-11 08:32:04
Connection to hdplztestvm.xxxxxxxxxx.ax.internal.cloudapp.net closed.
SSH command execution finished
host=hdplztestvm.xxxxxxxxxx.ax.internal.cloudapp.net, exitcode=0
Command end time 2018-05-11 08:32:04
==========================
Update apt cache of repository...
==========================
Command start time 2018-05-11 08:32:04
0% [Working]
Hit:1 http://ppa.launchpad.net/webupd8team/java/ubuntu xenial InRelease
0% [Waiting for headers]
0% [1 InRelease gpgv 17.5 kB] [Waiting for headers]
Hit:2 http://public-repo-1.hortonworks.com/ambari/ubuntu16/2.x/updates/2.5.1.0 Ambari InRelease
0% [1 InRelease gpgv 17.5 kB]
0% [Working]
0% [2 InRelease gpgv 7,394 B]
20% [Working]
Reading package lists... 0%
Reading package lists... 0%
Reading package lists... 0%
Reading package lists... 0%
Reading package lists... 1%
Reading package lists... 1%
Reading package lists... 1%
Reading package lists... 2%
Reading package lists... 2%
Reading package lists... Done
W: http://public-repo-1.hortonworks.com/ambari/ubuntu16/2.x/updates/2.5.1.0/dists/Ambari/InRelease: Signature by key DF52ED4F7A3A5882C0994C66B9733A7A07513CAD uses weak digest algorithm (SHA1)
Connection to hdplztestvm.xxxxxxxxxx.ax.internal.cloudapp.net closed.
SSH command execution finished
host=hdplztestvm.xxxxxxxxxx.ax.internal.cloudapp.net, exitcode=0
Command end time 2018-05-11 08:32:04
==========================
Copying setup script file...
==========================
Command start time 2018-05-11 08:32:04
scp /usr/lib/python2.6/site-packages/ambari_server/setupAgent.py
host=hdplztestvm.xxxxxxxxxx.ax.internal.cloudapp.net, exitcode=0
Command end time 2018-05-11 08:32:05
==========================
Running setup agent script...
==========================
Command start time 2018-05-11 08:32:05
('INFO 2018-05-11 08:32:37,604 DataCleaner.py:39 - Data cleanup thread started
INFO 2018-05-11 08:32:37,606 DataCleaner.py:120 - Data cleanup started
INFO 2018-05-11 08:32:37,606 DataCleaner.py:122 - Data cleanup finished
INFO 2018-05-11 08:32:37,606 hostname.py:67 - agent:hostname_script configuration not defined thus read hostname \'HDPLZTESTVM.xxxxxxxxxx.ax.internal.cloudapp.net\' using socket.getfqdn().
INFO 2018-05-11 08:32:37,645 PingPortListener.py:50 - Ping port listener started on port: 8670
INFO 2018-05-11 08:32:37,646 main.py:437 - Connecting to Ambari server at https://hdplztestvm.xxxxxxxxxx.ax.internal.cloudapp.net:8440 (137.xxx.xxx.xx6)
INFO 2018-05-11 08:32:37,646 NetUtil.py:70 - Connecting to https://hdplztestvm.xxxxxxxxxx.ax.internal.cloudapp.net:8440/ca
INFO 2018-05-11 08:32:37,714 main.py:447 - Connected to Ambari server hdplztestvm.xxxxxxxxxx.ax.internal.cloudapp.net
INFO 2018-05-11 08:32:37,715 threadpool.py:58 - Started thread pool with 3 core threads and 20 maximum threads
WARNING 2018-05-11 08:32:37,715 AlertSchedulerHandler.py:280 - [AlertScheduler] /var/lib/ambari-agent/cache/alerts/definitions.json not found or invalid. No alerts will be scheduled until registration occurs.
INFO 2018-05-11 08:32:37,715 AlertSchedulerHandler.py:175 - [AlertScheduler] Starting <ambari_agent.apscheduler.scheduler.Scheduler object at 0x7fde6b3dde50>; currently running: False
INFO 2018-05-11 08:32:37,724 hostname.py:106 - Read public hostname \'hdplztestvm.xxxxxxxxxx.ax.internal.cloudapp.net\' using socket.getfqdn()
INFO 2018-05-11 08:32:37,724 Hardware.py:48 - Initializing host system information.
INFO 2018-05-11 08:32:37,730 Hardware.py:176 - Some mount points were ignored: /run, /dev/shm, /run/lock, /sys/fs/cgroup, /run/user/1000, /run/user/0
INFO 2018-05-11 08:32:37,739 hostname.py:67 - agent:hostname_script configuration not defined thus read hostname \'HDPLZTESTVM.xxxxxxxxxx.ax.internal.cloudapp.net\' using socket.getfqdn().
INFO 2018-05-11 08:32:37,741 Facter.py:202 - Directory: \'/etc/resource_overrides\' does not exist - it won\'t be used for gathering system resources.
INFO 2018-05-11 08:32:37,745 Hardware.py:54 - Host system information: {\'kernel\': \'Linux\', \'domain\': \'xxxxxxxxxx.ax.internal.cloudapp.net\', \'physicalprocessorcount\': 4, \'kernelrelease\': \'4.13.0-1016-azure\', \'uptime_days\': \'0\', \'memorytotal\': 16404012, \'swapfree\': \'0.00 GB\', \'memorysize\': 16404012, \'osfamily\': \'ubuntu\', \'swapsize\': \'0.00 GB\', \'processorcount\': 4, \'netmask\': None, \'timezone\': \'UTC\', \'hardwareisa\': \'x86_64\', \'memoryfree\': 14727340, \'operatingsystem\': \'ubuntu\', \'kernelmajversion\': \'4.13\', \'kernelversion\': \'4.13.0\', \'macaddress\': \'00:0D:3A:29:77:AA\', \'operatingsystemrelease\': \'16.04\', \'ipaddress\': \'137.xxx.xxx.xx6\', \'hostname\': \'hdplztestvm\', \'uptime_hours\': \'1\', \'fqdn\': \'hdplztestvm.xxxxxxxxxx.ax.internal.cloudapp.net\', \'id\': \'root\', \'architecture\': \'x86_64\', \'selinux\': False, \'mounts\': [{\'available\': \'8191228\', \'used\': \'0\', \'percent\': \'0%\', \'device\': \'udev\', \'mountpoint\': \'/dev\', \'type\': \'devtmpfs\', \'size\': \'8191228\'}, {\'available\': \'25306356\', \'used\': \'5105908\', \'percent\': \'17%\', \'device\': \'/dev/sda1\', \'mountpoint\': \'/\', \'type\': \'ext4\', \'size\': \'30428648\'}, {\'available\': \'31152660\', \'used\': \'49084\', \'percent\': \'1%\', \'device\': \'/dev/sdb1\', \'mountpoint\': \'/mnt\', \'type\': \'ext4\', \'size\': \'32895792\'}], \'hardwaremodel\': \'x86_64\', \'uptime_seconds\': \'6084\', \'interfaces\': \'eth0,lo\'}
INFO 2018-05-11 08:32:37,865 Controller.py:170 - Registering with hdplztestvm.xxxxxxxxxx.ax.internal.cloudapp.net (137.xxx.xxx.xx6) (agent=\'{"hardwareProfile": {"kernel": "Linux", "domain": "xxxxxxxxxx.ax.internal.cloudapp.net", "physicalprocessorcount": 4, "kernelrelease": "4.13.0-1016-azure", "uptime_days": "0", "memorytotal": 16404012, "swapfree": "0.00 GB", "memorysize": 16404012, "osfamily": "ubuntu", "swapsize": "0.00 GB", "processorcount": 4, "netmask": null, "timezone": "UTC", "hardwareisa": "x86_64", "memoryfree": 14727340, "operatingsystem": "ubuntu", "kernelmajversion": "4.13", "kernelversion": "4.13.0", "macaddress": "00:0D:3A:29:77:AA", "operatingsystemrelease": "16.04", "ipaddress": "137.xxx.xxx.xx6", "hostname": "hdplztestvm", "uptime_hours": "1", "fqdn": "hdplztestvm.xxxxxxxxxx.ax.internal.cloudapp.net", "id": "root", "architecture": "x86_64", "selinux": false, "mounts": [{"available": "8191228", "used": "0", "percent": "0%", "device": "udev", "mountpoint": "/dev", "type": "devtmpfs", "size": "8191228"}, {"available": "25306356", "used": "5105908", "percent": "17%", "device": "/dev/sda1", "mountpoint": "/", "type": "ext4", "size": "30428648"}, {"available": "31152660", "used": "49084", "percent": "1%", "device": "/dev/sdb1", "mountpoint": "/mnt", "type": "ext4", "size": "32895792"}], "hardwaremodel": "x86_64", "uptime_seconds": "6084", "interfaces": "eth0,lo"}, "currentPingPort": 8670, "prefix": "/var/lib/ambari-agent/data", "agentVersion": "2.5.1.0", "agentEnv": {"transparentHugePage": "always", "hostHealth": {"agentTimeStampAtReporting": 1526027557863, "activeJavaProcs": [], "liveServices": [{"status": "Healthy", "name": "ntp or chrony", "desc": ""}]}, "reverseLookup": true, "alternatives": [], "hasUnlimitedJcePolicy": null, "umask": "18", "firewallName": "ufw", "stackFoldersAndFiles": [], "existingUsers": [], "firewallRunning": false}, "timestamp": 1526027557746, "hostname": "hdplztestvm.xxxxxxxxxx.ax.internal.cloudapp.net", "responseId": -1, "publicHostname": "hdplztestvm.xxxxxxxxxx.ax.internal.cloudapp.net"}\')
INFO 2018-05-11 08:32:37,865 NetUtil.py:70 - Connecting to https://hdplztestvm.xxxxxxxxxx.ax.internal.cloudapp.net:8440/connection_info
INFO 2018-05-11 08:32:37,946 security.py:93 - SSL Connect being called.. connecting to the server
', None)
('INFO 2018-05-11 08:32:37,604 DataCleaner.py:39 - Data cleanup thread started
INFO 2018-05-11 08:32:37,606 DataCleaner.py:120 - Data cleanup started
INFO 2018-05-11 08:32:37,606 DataCleaner.py:122 - Data cleanup finished
INFO 2018-05-11 08:32:37,606 hostname.py:67 - agent:hostname_script configuration not defined thus read hostname \'HDPLZTESTVM.xxxxxxxxxx.ax.internal.cloudapp.net\' using socket.getfqdn().
INFO 2018-05-11 08:32:37,645 PingPortListener.py:50 - Ping port listener started on port: 8670
INFO 2018-05-11 08:32:37,646 main.py:437 - Connecting to Ambari server at https://hdplztestvm.xxxxxxxxxx.ax.internal.cloudapp.net:8440 (137.xxx.xxx.xx6)
INFO 2018-05-11 08:32:37,646 NetUtil.py:70 - Connecting to https://hdplztestvm.xxxxxxxxxx.ax.internal.cloudapp.net:8440/ca
INFO 2018-05-11 08:32:37,714 main.py:447 - Connected to Ambari server hdplztestvm.xxxxxxxxxx.ax.internal.cloudapp.net
INFO 2018-05-11 08:32:37,715 threadpool.py:58 - Started thread pool with 3 core threads and 20 maximum threads
WARNING 2018-05-11 08:32:37,715 AlertSchedulerHandler.py:280 - [AlertScheduler] /var/lib/ambari-agent/cache/alerts/definitions.json not found or invalid. No alerts will be scheduled until registration occurs.
INFO 2018-05-11 08:32:37,715 AlertSchedulerHandler.py:175 - [AlertScheduler] Starting <ambari_agent.apscheduler.scheduler.Scheduler object at 0x7fde6b3dde50>; currently running: False
INFO 2018-05-11 08:32:37,724 hostname.py:106 - Read public hostname \'hdplztestvm.xxxxxxxxxx.ax.internal.cloudapp.net\' using socket.getfqdn()
INFO 2018-05-11 08:32:37,724 Hardware.py:48 - Initializing host system information.
INFO 2018-05-11 08:32:37,730 Hardware.py:176 - Some mount points were ignored: /run, /dev/shm, /run/lock, /sys/fs/cgroup, /run/user/1000, /run/user/0
INFO 2018-05-11 08:32:37,739 hostname.py:67 - agent:hostname_script configuration not defined thus read hostname \'HDPLZTESTVM.xxxxxxxxxx.ax.internal.cloudapp.net\' using socket.getfqdn().
INFO 2018-05-11 08:32:37,741 Facter.py:202 - Directory: \'/etc/resource_overrides\' does not exist - it won\'t be used for gathering system resources.
INFO 2018-05-11 08:32:37,745 Hardware.py:54 - Host system information: {\'kernel\': \'Linux\', \'domain\': \'xxxxxxxxxx.ax.internal.cloudapp.net\', \'physicalprocessorcount\': 4, \'kernelrelease\': \'4.13.0-1016-azure\', \'uptime_days\': \'0\', \'memorytotal\': 16404012, \'swapfree\': \'0.00 GB\', \'memorysize\': 16404012, \'osfamily\': \'ubuntu\', \'swapsize\': \'0.00 GB\', \'processorcount\': 4, \'netmask\': None, \'timezone\': \'UTC\', \'hardwareisa\': \'x86_64\', \'memoryfree\': 14727340, \'operatingsystem\': \'ubuntu\', \'kernelmajversion\': \'4.13\', \'kernelversion\': \'4.13.0\', \'macaddress\': \'00:0D:3A:29:77:AA\', \'operatingsystemrelease\': \'16.04\', \'ipaddress\': \'137.xxx.xxx.xx6\', \'hostname\': \'hdplztestvm\', \'uptime_hours\': \'1\', \'fqdn\': \'hdplztestvm.xxxxxxxxxx.ax.internal.cloudapp.net\', \'id\': \'root\', \'architecture\': \'x86_64\', \'selinux\': False, \'mounts\': [{\'available\': \'8191228\', \'used\': \'0\', \'percent\': \'0%\', \'device\': \'udev\', \'mountpoint\': \'/dev\', \'type\': \'devtmpfs\', \'size\': \'8191228\'}, {\'available\': \'25306356\', \'used\': \'5105908\', \'percent\': \'17%\', \'device\': \'/dev/sda1\', \'mountpoint\': \'/\', \'type\': \'ext4\', \'size\': \'30428648\'}, {\'available\': \'31152660\', \'used\': \'49084\', \'percent\': \'1%\', \'device\': \'/dev/sdb1\', \'mountpoint\': \'/mnt\', \'type\': \'ext4\', \'size\': \'32895792\'}], \'hardwaremodel\': \'x86_64\', \'uptime_seconds\': \'6084\', \'interfaces\': \'eth0,lo\'}
INFO 2018-05-11 08:32:37,865 Controller.py:170 - Registering with hdplztestvm.xxxxxxxxxx.ax.internal.cloudapp.net (137.xxx.xxx.xx6) (agent=\'{"hardwareProfile": {"kernel": "Linux", "domain": "xxxxxxxxxx.ax.internal.cloudapp.net", "physicalprocessorcount": 4, "kernelrelease": "4.13.0-1016-azure", "uptime_days": "0", "memorytotal": 16404012, "swapfree": "0.00 GB", "memorysize": 16404012, "osfamily": "ubuntu", "swapsize": "0.00 GB", "processorcount": 4, "netmask": null, "timezone": "UTC", "hardwareisa": "x86_64", "memoryfree": 14727340, "operatingsystem": "ubuntu", "kernelmajversion": "4.13", "kernelversion": "4.13.0", "macaddress": "00:0D:3A:29:77:AA", "operatingsystemrelease": "16.04", "ipaddress": "137.xxx.xxx.xx6", "hostname": "hdplztestvm", "uptime_hours": "1", "fqdn": "hdplztestvm.xxxxxxxxxx.ax.internal.cloudapp.net", "id": "root", "architecture": "x86_64", "selinux": false, "mounts": [{"available": "8191228", "used": "0", "percent": "0%", "device": "udev", "mountpoint": "/dev", "type": "devtmpfs", "size": "8191228"}, {"available": "25306356", "used": "5105908", "percent": "17%", "device": "/dev/sda1", "mountpoint": "/", "type": "ext4", "size": "30428648"}, {"available": "31152660", "used": "49084", "percent": "1%", "device": "/dev/sdb1", "mountpoint": "/mnt", "type": "ext4", "size": "32895792"}], "hardwaremodel": "x86_64", "uptime_seconds": "6084", "interfaces": "eth0,lo"}, "currentPingPort": 8670, "prefix": "/var/lib/ambari-agent/data", "agentVersion": "2.5.1.0", "agentEnv": {"transparentHugePage": "always", "hostHealth": {"agentTimeStampAtReporting": 1526027557863, "activeJavaProcs": [], "liveServices": [{"status": "Healthy", "name": "ntp or chrony", "desc": ""}]}, "reverseLookup": true, "alternatives": [], "hasUnlimitedJcePolicy": null, "umask": "18", "firewallName": "ufw", "stackFoldersAndFiles": [], "existingUsers": [], "firewallRunning": false}, "timestamp": 1526027557746, "hostname": "hdplztestvm.xxxxxxxxxx.ax.internal.cloudapp.net", "responseId": -1, "publicHostname": "hdplztestvm.xxxxxxxxxx.ax.internal.cloudapp.net"}\')
INFO 2018-05-11 08:32:37,865 NetUtil.py:70 - Connecting to https://hdplztestvm.xxxxxxxxxx.ax.internal.cloudapp.net:8440/connection_info
INFO 2018-05-11 08:32:37,946 security.py:93 - SSL Connect being called.. connecting to the server
', None)
Connection to hdplztestvm.xxxxxxxxxx.ax.internal.cloudapp.net closed.
SSH command execution finished
host=hdplztestvm.xxxxxxxxxx.ax.internal.cloudapp.net, exitcode=0
Command end time 2018-05-11 08:32:40
Registering with the server...
Registration with the server failed.
HDPLZVMSN1 is one of the node where I have installed ambari-agent and below is the registration log for this host: ==========================
Creating target directory...
==========================
Command start time 2018-05-11 08:32:01
Connection to hdplzvmsn1.xxxxxxxxxx.ax.internal.cloudapp.net closed.
SSH command execution finished
host=hdplzvmsn1.xxxxxxxxxx.ax.internal.cloudapp.net, exitcode=0
Command end time 2018-05-11 08:32:01
==========================
Copying ambari sudo script...
==========================
Command start time 2018-05-11 08:32:01
scp /var/lib/ambari-server/ambari-sudo.sh
host=hdplzvmsn1.xxxxxxxxxx.ax.internal.cloudapp.net, exitcode=0
Command end time 2018-05-11 08:32:02
==========================
Copying common functions script...
==========================
Command start time 2018-05-11 08:32:02
scp /usr/lib/python2.6/site-packages/ambari_commons
host=hdplzvmsn1.xxxxxxxxxx.ax.internal.cloudapp.net, exitcode=0
Command end time 2018-05-11 08:32:02
==========================
Copying create-python-wrap script...
==========================
Command start time 2018-05-11 08:32:02
scp /var/lib/ambari-server/create-python-wrap.sh
host=hdplzvmsn1.xxxxxxxxxx.ax.internal.cloudapp.net, exitcode=0
Command end time 2018-05-11 08:32:02
==========================
Copying OS type check script...
==========================
Command start time 2018-05-11 08:32:02
scp /usr/lib/python2.6/site-packages/ambari_server/os_check_type.py
host=hdplzvmsn1.xxxxxxxxxx.ax.internal.cloudapp.net, exitcode=0
Command end time 2018-05-11 08:32:02
==========================
Running create-python-wrap script...
==========================
Command start time 2018-05-11 08:32:02
Connection to hdplzvmsn1.xxxxxxxxxx.ax.internal.cloudapp.net closed.
SSH command execution finished
host=hdplzvmsn1.xxxxxxxxxx.ax.internal.cloudapp.net, exitcode=0
Command end time 2018-05-11 08:32:03
==========================
Running OS type check...
==========================
Command start time 2018-05-11 08:32:03
Cluster primary/cluster OS family is ubuntu16 and local/current OS family is ubuntu16
Connection to hdplzvmsn1.xxxxxxxxxx.ax.internal.cloudapp.net closed.
SSH command execution finished
host=hdplzvmsn1.xxxxxxxxxx.ax.internal.cloudapp.net, exitcode=0
Command end time 2018-05-11 08:32:03
==========================
Checking 'sudo' package on remote host...
==========================
Command start time 2018-05-11 08:32:03
Connection to hdplzvmsn1.xxxxxxxxxx.ax.internal.cloudapp.net closed.
SSH command execution finished
host=hdplzvmsn1.xxxxxxxxxx.ax.internal.cloudapp.net, exitcode=0
Command end time 2018-05-11 08:32:03
==========================
Copying repo file to 'tmp' folder...
==========================
Command start time 2018-05-11 08:32:03
scp /etc/apt/sources.list.d/ambari.list
host=hdplzvmsn1.xxxxxxxxxx.ax.internal.cloudapp.net, exitcode=0
Command end time 2018-05-11 08:32:03
==========================
Moving file to repo dir...
==========================
Command start time 2018-05-11 08:32:03
Connection to hdplzvmsn1.xxxxxxxxxx.ax.internal.cloudapp.net closed.
SSH command execution finished
host=hdplzvmsn1.xxxxxxxxxx.ax.internal.cloudapp.net, exitcode=0
Command end time 2018-05-11 08:32:04
==========================
Changing permissions for ambari.repo...
==========================
Command start time 2018-05-11 08:32:04
Connection to hdplzvmsn1.xxxxxxxxxx.ax.internal.cloudapp.net closed.
SSH command execution finished
host=hdplzvmsn1.xxxxxxxxxx.ax.internal.cloudapp.net, exitcode=0
Command end time 2018-05-11 08:32:04
==========================
Update apt cache of repository...
==========================
Command start time 2018-05-11 08:32:04
0% [Working]
Hit:1 http://public-repo-1.hortonworks.com/ambari/ubuntu16/2.x/updates/2.5.1.0 Ambari InRelease
0% [Working]
0% [1 InRelease gpgv 7,394 B]
20% [Working]
Reading package lists... 0%
Reading package lists... 0%
Reading package lists... 0%
Reading package lists... 0%
Reading package lists... 18%
Reading package lists... Done
W: http://public-repo-1.hortonworks.com/ambari/ubuntu16/2.x/updates/2.5.1.0/dists/Ambari/InRelease: Signature by key DF52ED4F7A3A5882C0994C66B9733A7A07513CAD uses weak digest algorithm (SHA1)
Connection to hdplzvmsn1.xxxxxxxxxx.ax.internal.cloudapp.net closed.
SSH command execution finished
host=hdplzvmsn1.xxxxxxxxxx.ax.internal.cloudapp.net, exitcode=0
Command end time 2018-05-11 08:32:05
==========================
Copying setup script file...
==========================
Command start time 2018-05-11 08:32:05
scp /usr/lib/python2.6/site-packages/ambari_server/setupAgent.py
host=hdplzvmsn1.xxxxxxxxxx.ax.internal.cloudapp.net, exitcode=0
Command end time 2018-05-11 08:32:05
==========================
Running setup agent script...
==========================
Command start time 2018-05-11 08:32:05
('INFO 2018-05-11 08:32:37,836 DataCleaner.py:39 - Data cleanup thread started
INFO 2018-05-11 08:32:37,837 DataCleaner.py:120 - Data cleanup started
INFO 2018-05-11 08:32:37,837 DataCleaner.py:122 - Data cleanup finished
INFO 2018-05-11 08:32:37,838 hostname.py:67 - agent:hostname_script configuration not defined thus read hostname \'HDPLZVMSN1.xxxxxxxxxx.ax.internal.cloudapp.net\' using socket.getfqdn().
INFO 2018-05-11 08:32:37,861 PingPortListener.py:50 - Ping port listener started on port: 8670
INFO 2018-05-11 08:32:37,864 main.py:437 - Connecting to Ambari server at https://hdplztestvm.xxxxxxxxxx.ax.internal.cloudapp.net:8440 (137.xxx.xxx.xx6)
INFO 2018-05-11 08:32:37,864 NetUtil.py:70 - Connecting to https://hdplztestvm.xxxxxxxxxx.ax.internal.cloudapp.net:8440/ca
INFO 2018-05-11 08:32:37,950 main.py:447 - Connected to Ambari server hdplztestvm.xxxxxxxxxx.ax.internal.cloudapp.net
INFO 2018-05-11 08:32:37,951 threadpool.py:58 - Started thread pool with 3 core threads and 20 maximum threads
WARNING 2018-05-11 08:32:37,952 AlertSchedulerHandler.py:280 - [AlertScheduler] /var/lib/ambari-agent/cache/alerts/definitions.json not found or invalid. No alerts will be scheduled until registration occurs.
INFO 2018-05-11 08:32:37,952 AlertSchedulerHandler.py:175 - [AlertScheduler] Starting <ambari_agent.apscheduler.scheduler.Scheduler object at 0x7f056e28bdd0>; currently running: False
INFO 2018-05-11 08:32:37,965 hostname.py:106 - Read public hostname \'hdplzvmsn1.xxxxxxxxxx.ax.internal.cloudapp.net\' using socket.getfqdn()
INFO 2018-05-11 08:32:37,966 Hardware.py:48 - Initializing host system information.
INFO 2018-05-11 08:32:37,973 Hardware.py:176 - Some mount points were ignored: /run, /dev/shm, /run/lock, /sys/fs/cgroup, /run/user/0, /run/user/1000
INFO 2018-05-11 08:32:37,983 hostname.py:67 - agent:hostname_script configuration not defined thus read hostname \'HDPLZVMSN1.xxxxxxxxxx.ax.internal.cloudapp.net\' using socket.getfqdn().
INFO 2018-05-11 08:32:37,985 Facter.py:202 - Directory: \'/etc/resource_overrides\' does not exist - it won\'t be used for gathering system resources.
INFO 2018-05-11 08:32:37,989 Hardware.py:54 - Host system information: {\'kernel\': \'Linux\', \'domain\': \'xxxxxxxxxx.ax.internal.cloudapp.net\', \'physicalprocessorcount\': 4, \'kernelrelease\': \'4.13.0-1014-azure\', \'uptime_days\': \'1\', \'memorytotal\': 16404012, \'swapfree\': \'0.00 GB\', \'memorysize\': 16404012, \'osfamily\': \'ubuntu\', \'swapsize\': \'0.00 GB\', \'processorcount\': 4, \'netmask\': None, \'timezone\': \'UTC\', \'hardwareisa\': \'x86_64\', \'memoryfree\': 14937452, \'operatingsystem\': \'ubuntu\', \'kernelmajversion\': \'4.13\', \'kernelversion\': \'4.13.0\', \'macaddress\': \'00:0D:3A:29:6C:6A\', \'operatingsystemrelease\': \'16.04\', \'ipaddress\': \'137.xxx.xxx.xx0\', \'hostname\': \'hdplzvmsn1\', \'uptime_hours\': \'27\', \'fqdn\': \'hdplzvmsn1.xxxxxxxxxx.ax.internal.cloudapp.net\', \'id\': \'root\', \'architecture\': \'x86_64\', \'selinux\': False, \'mounts\': [{\'available\': \'8191228\', \'used\': \'0\', \'percent\': \'0%\', \'device\': \'udev\', \'mountpoint\': \'/dev\', \'type\': \'devtmpfs\', \'size\': \'8191228\'}, {\'available\': \'28704176\', \'used\': \'1708088\', \'percent\': \'6%\', \'device\': \'/dev/sda1\', \'mountpoint\': \'/\', \'type\': \'ext4\', \'size\': \'30428648\'}, {\'available\': \'31152660\', \'used\': \'49084\', \'percent\': \'1%\', \'device\': \'/dev/sdb1\', \'mountpoint\': \'/mnt\', \'type\': \'ext4\', \'size\': \'32895792\'}], \'hardwaremodel\': \'x86_64\', \'uptime_seconds\': \'99344\', \'interfaces\': \'eth0,lo\'}
INFO 2018-05-11 08:32:38,121 Controller.py:170 - Registering with hdplzvmsn1.xxxxxxxxxx.ax.internal.cloudapp.net (137.xxx.xxx.xx0) (agent=\'{"hardwareProfile": {"kernel": "Linux", "domain": "xxxxxxxxxx.ax.internal.cloudapp.net", "physicalprocessorcount": 4, "kernelrelease": "4.13.0-1014-azure", "uptime_days": "1", "memorytotal": 16404012, "swapfree": "0.00 GB", "memorysize": 16404012, "osfamily": "ubuntu", "swapsize": "0.00 GB", "processorcount": 4, "netmask": null, "timezone": "UTC", "hardwareisa": "x86_64", "memoryfree": 14937452, "operatingsystem": "ubuntu", "kernelmajversion": "4.13", "kernelversion": "4.13.0", "macaddress": "00:0D:3A:29:6C:6A", "operatingsystemrelease": "16.04", "ipaddress": "137.xxx.xxx.xx0", "hostname": "hdplzvmsn1", "uptime_hours": "27", "fqdn": "hdplzvmsn1.xxxxxxxxxx.ax.internal.cloudapp.net", "id": "root", "architecture": "x86_64", "selinux": false, "mounts": [{"available": "8191228", "used": "0", "percent": "0%", "device": "udev", "mountpoint": "/dev", "type": "devtmpfs", "size": "8191228"}, {"available": "28704176", "used": "1708088", "percent": "6%", "device": "/dev/sda1", "mountpoint": "/", "type": "ext4", "size": "30428648"}, {"available": "31152660", "used": "49084", "percent": "1%", "device": "/dev/sdb1", "mountpoint": "/mnt", "type": "ext4", "size": "32895792"}], "hardwaremodel": "x86_64", "uptime_seconds": "99344", "interfaces": "eth0,lo"}, "currentPingPort": 8670, "prefix": "/var/lib/ambari-agent/data", "agentVersion": "2.5.1.0", "agentEnv": {"transparentHugePage": "always", "hostHealth": {"agentTimeStampAtReporting": 1526027558120, "activeJavaProcs": [], "liveServices": [{"status": "Healthy", "name": "ntp or chrony", "desc": ""}]}, "reverseLookup": true, "alternatives": [], "hasUnlimitedJcePolicy": null, "umask": "18", "firewallName": "ufw", "stackFoldersAndFiles": [], "existingUsers": [], "firewallRunning": false}, "timestamp": 1526027557990, "hostname": "hdplzvmsn1.xxxxxxxxxx.ax.internal.cloudapp.net", "responseId": -1, "publicHostname": "hdplzvmsn1.xxxxxxxxxx.ax.internal.cloudapp.net"}\')
INFO 2018-05-11 08:32:38,122 NetUtil.py:70 - Connecting to https://hdplztestvm.xxxxxxxxxx.ax.internal.cloudapp.net:8440/connection_info
INFO 2018-05-11 08:32:38,181 security.py:93 - SSL Connect being called.. connecting to the server
', None)
('INFO 2018-05-11 08:32:37,836 DataCleaner.py:39 - Data cleanup thread started
INFO 2018-05-11 08:32:37,837 DataCleaner.py:120 - Data cleanup started
INFO 2018-05-11 08:32:37,837 DataCleaner.py:122 - Data cleanup finished
INFO 2018-05-11 08:32:37,838 hostname.py:67 - agent:hostname_script configuration not defined thus read hostname \'HDPLZVMSN1.xxxxxxxxxx.ax.internal.cloudapp.net\' using socket.getfqdn().
INFO 2018-05-11 08:32:37,861 PingPortListener.py:50 - Ping port listener started on port: 8670
INFO 2018-05-11 08:32:37,864 main.py:437 - Connecting to Ambari server at https://hdplztestvm.xxxxxxxxxx.ax.internal.cloudapp.net:8440 (137.xxx.xxx.xx6)
INFO 2018-05-11 08:32:37,864 NetUtil.py:70 - Connecting to https://hdplztestvm.xxxxxxxxxx.ax.internal.cloudapp.net:8440/ca
INFO 2018-05-11 08:32:37,950 main.py:447 - Connected to Ambari server hdplztestvm.xxxxxxxxxx.ax.internal.cloudapp.net
INFO 2018-05-11 08:32:37,951 threadpool.py:58 - Started thread pool with 3 core threads and 20 maximum threads
WARNING 2018-05-11 08:32:37,952 AlertSchedulerHandler.py:280 - [AlertScheduler] /var/lib/ambari-agent/cache/alerts/definitions.json not found or invalid. No alerts will be scheduled until registration occurs.
INFO 2018-05-11 08:32:37,952 AlertSchedulerHandler.py:175 - [AlertScheduler] Starting <ambari_agent.apscheduler.scheduler.Scheduler object at 0x7f056e28bdd0>; currently running: False
INFO 2018-05-11 08:32:37,965 hostname.py:106 - Read public hostname \'hdplzvmsn1.xxxxxxxxxx.ax.internal.cloudapp.net\' using socket.getfqdn()
INFO 2018-05-11 08:32:37,966 Hardware.py:48 - Initializing host system information.
INFO 2018-05-11 08:32:37,973 Hardware.py:176 - Some mount points were ignored: /run, /dev/shm, /run/lock, /sys/fs/cgroup, /run/user/0, /run/user/1000
INFO 2018-05-11 08:32:37,983 hostname.py:67 - agent:hostname_script configuration not defined thus read hostname \'HDPLZVMSN1.xxxxxxxxxx.ax.internal.cloudapp.net\' using socket.getfqdn().
INFO 2018-05-11 08:32:37,985 Facter.py:202 - Directory: \'/etc/resource_overrides\' does not exist - it won\'t be used for gathering system resources.
INFO 2018-05-11 08:32:37,989 Hardware.py:54 - Host system information: {\'kernel\': \'Linux\', \'domain\': \'xxxxxxxxxx.ax.internal.cloudapp.net\', \'physicalprocessorcount\': 4, \'kernelrelease\': \'4.13.0-1014-azure\', \'uptime_days\': \'1\', \'memorytotal\': 16404012, \'swapfree\': \'0.00 GB\', \'memorysize\': 16404012, \'osfamily\': \'ubuntu\', \'swapsize\': \'0.00 GB\', \'processorcount\': 4, \'netmask\': None, \'timezone\': \'UTC\', \'hardwareisa\': \'x86_64\', \'memoryfree\': 14937452, \'operatingsystem\': \'ubuntu\', \'kernelmajversion\': \'4.13\', \'kernelversion\': \'4.13.0\', \'macaddress\': \'00:0D:3A:29:6C:6A\', \'operatingsystemrelease\': \'16.04\', \'ipaddress\': \'137.xxx.xxx.xx0\', \'hostname\': \'hdplzvmsn1\', \'uptime_hours\': \'27\', \'fqdn\': \'hdplzvmsn1.xxxxxxxxxx.ax.internal.cloudapp.net\', \'id\': \'root\', \'architecture\': \'x86_64\', \'selinux\': False, \'mounts\': [{\'available\': \'8191228\', \'used\': \'0\', \'percent\': \'0%\', \'device\': \'udev\', \'mountpoint\': \'/dev\', \'type\': \'devtmpfs\', \'size\': \'8191228\'}, {\'available\': \'28704176\', \'used\': \'1708088\', \'percent\': \'6%\', \'device\': \'/dev/sda1\', \'mountpoint\': \'/\', \'type\': \'ext4\', \'size\': \'30428648\'}, {\'available\': \'31152660\', \'used\': \'49084\', \'percent\': \'1%\', \'device\': \'/dev/sdb1\', \'mountpoint\': \'/mnt\', \'type\': \'ext4\', \'size\': \'32895792\'}], \'hardwaremodel\': \'x86_64\', \'uptime_seconds\': \'99344\', \'interfaces\': \'eth0,lo\'}
INFO 2018-05-11 08:32:38,121 Controller.py:170 - Registering with hdplzvmsn1.xxxxxxxxxx.ax.internal.cloudapp.net (137.xxx.xxx.xx0) (agent=\'{"hardwareProfile": {"kernel": "Linux", "domain": "xxxxxxxxxx.ax.internal.cloudapp.net", "physicalprocessorcount": 4, "kernelrelease": "4.13.0-1014-azure", "uptime_days": "1", "memorytotal": 16404012, "swapfree": "0.00 GB", "memorysize": 16404012, "osfamily": "ubuntu", "swapsize": "0.00 GB", "processorcount": 4, "netmask": null, "timezone": "UTC", "hardwareisa": "x86_64", "memoryfree": 14937452, "operatingsystem": "ubuntu", "kernelmajversion": "4.13", "kernelversion": "4.13.0", "macaddress": "00:0D:3A:29:6C:6A", "operatingsystemrelease": "16.04", "ipaddress": "137.xxx.xxx.xx0", "hostname": "hdplzvmsn1", "uptime_hours": "27", "fqdn": "hdplzvmsn1.xxxxxxxxxx.ax.internal.cloudapp.net", "id": "root", "architecture": "x86_64", "selinux": false, "mounts": [{"available": "8191228", "used": "0", "percent": "0%", "device": "udev", "mountpoint": "/dev", "type": "devtmpfs", "size": "8191228"}, {"available": "28704176", "used": "1708088", "percent": "6%", "device": "/dev/sda1", "mountpoint": "/", "type": "ext4", "size": "30428648"}, {"available": "31152660", "used": "49084", "percent": "1%", "device": "/dev/sdb1", "mountpoint": "/mnt", "type": "ext4", "size": "32895792"}], "hardwaremodel": "x86_64", "uptime_seconds": "99344", "interfaces": "eth0,lo"}, "currentPingPort": 8670, "prefix": "/var/lib/ambari-agent/data", "agentVersion": "2.5.1.0", "agentEnv": {"transparentHugePage": "always", "hostHealth": {"agentTimeStampAtReporting": 1526027558120, "activeJavaProcs": [], "liveServices": [{"status": "Healthy", "name": "ntp or chrony", "desc": ""}]}, "reverseLookup": true, "alternatives": [], "hasUnlimitedJcePolicy": null, "umask": "18", "firewallName": "ufw", "stackFoldersAndFiles": [], "existingUsers": [], "firewallRunning": false}, "timestamp": 1526027557990, "hostname": "hdplzvmsn1.xxxxxxxxxx.ax.internal.cloudapp.net", "responseId": -1, "publicHostname": "hdplzvmsn1.xxxxxxxxxx.ax.internal.cloudapp.net"}\')
INFO 2018-05-11 08:32:38,122 NetUtil.py:70 - Connecting to https://hdplztestvm.xxxxxxxxxx.ax.internal.cloudapp.net:8440/connection_info
INFO 2018-05-11 08:32:38,181 security.py:93 - SSL Connect being called.. connecting to the server
', None)
Connection to hdplzvmsn1.xxxxxxxxxx.ax.internal.cloudapp.net closed.
SSH command execution finished
host=hdplzvmsn1.xxxxxxxxxx.ax.internal.cloudapp.net, exitcode=0
Command end time 2018-05-11 08:32:40
Registering with the server...
Registration with the server failed.
I'm unable to get the exact issue from these logs. Please help me to proceed further.
... View more
Labels:
05-11-2018
09:33 AM
@Umair Khan thank you for providing the solution.
... View more
05-08-2018
07:55 AM
Hi All, How to create a multi node HDP cluster in Azure VM without using cloudbreak. And also If I have the azure subscription, do I need to pay to Hortonworks for doing this.
... View more
Labels:
