Thank you Cdraper, Finally I use this combination, RHEL 7.3 (m4.large) , ambari 2.4.10 and HDP 2.5 install 7 nodes on AWS. the only issues I deal with is change the namenode java heap size to 4gb in my case. then everything worked. your info is still very helpful. I am going to setup kafka 3 nodes and mongodb from here. will keep you posted. thanks again. Robin ... View more

thank you for getting back to me. 1 iptables -L dont have any ouput 2. the firewall package is available but not installed in a AWS/EC2 instance. therefore, I did run systemctl stop firewalld and systemctl disable firewalld on REHL 7 and service iptables stop, chkconfig iptables off on REHL6. 3. the point here, if these firewall or iptables package is not installed, I may have to think they are never act, so if not installed, they are not there = disabled. let me know if you think differently. thanks, ... View more

Thank you Cdraper for getting back to me. 1. I dont have HA setup, but I think HDP requires to have secondary name node, I didnt do any configure yet, but snamenode is on the list to start 2. do yo think Hbase space issue could cause this problem. as Hbase stores all metadata for namenode and cluster, if Hbase has problem, then namenode cant start, so does snamenode then HDFS then entire cluster? 3. if I use m4.xlarge instance on AWS, ended up with warnings on hbase_master_heapsize, hbase_collector_heapszie etc, if you change to the recommended values, the new sets of parameters come up. one time I have changed 7 times and never get rid of that warning. if I ignore the warning, my namenode never, never started. 4. if I use m4.large, I dont get the these heapsize warnings, instead I get warning on packages not install, so I tried to manually install them on host, it just dont execute or yum dont have this package. I am kind of give up on HDP 2.4 install on RHEL 6, 7/AWS/EC2, so what do you suggest, shall I try ubuntu or Centos, which version? HDP stable version is 2.4.1.0 and ubuntu 12? After all these experiment, I just feel HDP with AWS is not compatible, I could be wrong. either AWS/EC2 root / size too samll, you can resize in instance creation process, but dont get the size you define. if instance RAM = 8gb or more, then HDP get lost on configure them, that is why all these heap_size size issues.... sorry long story, thank you for your help. Robin ... View more

Clink, I got the same error, and I changed my /etc/hosts file as: 52.60.239.41 ip-172-31-12-19.ca-central-1.compute.internal node1 also, I restart agent on each hosts, 4 of them, I still get the same error. please help.   Robin ... View more

Clint, my hosts file look like this, and seemed it is not working. 52.60.239.41 ip-172-31-12-19.ca-central-1.compute.internal 52.60.157.176 ip-172-31-3-218.ca-central-1.compute.internal 35.182.18.189 ip-172-31-3-179.ca-central-1.compute.internal 35.182.25.52 ip-172-31-2-137.ca-central-1.compute.internal do I have to give a short name at end, where this shortname come from my EC2 instance name, right? ... View more

thank you so much. even I need more time to digest it, I can tell this is what I need to know. thank you for spending time to help. Robin ... View more

both of #openssl s_client -connect ip-172-31-13-143.us-west-2.compute.internal:8440 and 8441 have no output. ... View more

#hostname -f ip-172-31-13-143.us-west-2.compute.internal #telnet ip-172-31-13-143.us-west-2.compute.internal8440 telnet: ip-172-31-13-143.us-west-2.compute.internal8440: Name or service not known ip-172-31-13-143.us-west-2.compute.internal8440: Unknown host # telnet ip-172-31-13-143.us-west-2.compute.internal8441 telnet: ip-172-31-13-143.us-west-2.compute.internal8441: Name or service not known ... View more

Thank you so much Jay. this is the file I modified by replace localhost to $hostname -f on all nodes here. Also, I started agent on all nodes, no problem. so how do I know if agent can communicate with master? ... View more

sorry, I terminated all nodes, however, I know this alert info: connection failed to http://ip-172-31-15-246.us-west-2.compute.internal on every node. some ports just dont open and directories not exist. my passwdless setup correctly for sure. however, connection refused, it is IP problem? if need static IP, limitation is 5. I have more than 10 nodes. Can you help? ... View more

oh, my ambari service is external mode, use http://ec2....amazon.com:8080, if this is so called external mode. after ambari-server reset, I got the same error. the repo is: http://public-repo-1.hortonworks.com/ambari/centos6/2.x/updates/2.4.0.1/ambari.repo rpm -qa|grep ambari-metrics -- show all packages are installed. so the hbase configure info is the problem. it refuse the connection. I dont need to edit hbase configure with ambari 2.4 in the past at all. any changes on HDP? thanks, Robin ... View more

oh, my ambari service is external mode, use http://ec2....amazon.com:8080, if this is so called external mode. after ambari-server reset, I got the same error. the repo is: http://public-repo-1.hortonworks.com/ambari/centos6/2.x/updates/2.4.0.1/ambari.repo rpm -qa|grep ambari-metrics -- show all packages are installed. so the hbase configure info is the problem. it refuse the connection. I dont need to edit hbase configure with ambari 2.4 in the past at all. any changes on HDP? thanks, Robin ... View more

oh, my ambari service is external mode, use http://ec2....amazon.com:8080, if this is so called external mode. after ambari-server reset, I got the same error. the repo is: http://public-repo-1.hortonworks.com/ambari/centos6/2.x/updates/2.4.0.1/ambari.repo rpm -qa|grep ambari-metrics -- show all packages are installed. so the hbase configure info is the problem. it refuse the connection. I dont need to edit hbase configure with ambari 2.4 in the past at all. any changes on HDP? thanks, Robin ... View more

oh, my ambari service is external mode, use http://ec2....amazon.com:8080, if this is so called external mode. after ambari-server reset, I got the same error. the repo is: http://public-repo-1.hortonworks.com/ambari/centos6/2.x/updates/2.4.0.1/ambari.repo rpm -qa|grep ambari-metrics -- show all packages are installed. so the hbase configure info is the problem. it refuse the connection. I dont need to edit hbase configure with ambari 2.4 in the past at all. any changes on HDP? thanks, Robin ... View more

on my ambari server and datanode, both has no ambari-metrics-collector found. [root@ip-172-31-1-92 etc]# ls -lart /etc/ambari-metrics-collector ls: cannot access /etc/ambari-metrics-collector: No such file or directory [root@ip-172-31-12-243 etc]# ls -lart /etc/ambari-metrics-collector ls: cannot access /etc/ambari-metrics-collector: No such file or directory 1. when add volume 100gb or 800gb, I still have the same connection problem 2. this time I didnt add new volume to EBS, only 20gb for root /. 3. I reinstall more than 10 time, always the same issue here. so do you think my security settings is the problem, here is my security group info, all TCP, HTTP all ports, all UDP and all traffice with anywhere access. this is wide open security. Do you think the adding ambari id_rsa.pub to authorized_keys on all servers for passwdless is enough? From ambari server, I can passweless ssh to all other servers including itself, do you think I need to make passwdless between all nodes? thank you for your help. ... View more

605-hdp-error.txt on one of the datanode. no ambari-metrics-collector.log on ambari server. this is metrics log file. I only have 20g for root dir /, no EBS storage assigned here ... View more

you are right, Jay. the problem is port connection refused. however, my /tmp is 777, so what to do from here. [root@ip-172-31-12-243 /]# free -m total used free shared buffers cached Mem: 16077 390 15687 3 81 74 -/+ buffers/cache: 234 15843 Swap: 0 0 0 [root@ip-172-31-12-243 /]# hostname -f ip-172-31-12-243.ca-central-1.compute.internal telnet ip-172-31-1-92.ca-central-1.compute.internal 3000 Trying 52.60.67.105... telnet: connect to address 52.60.67.105: Connection refused [root@ip-172-31-12-243 /]# telnet ip-172-31-1-92.ca-central-1.compute.internal 16000 Trying 52.60.67.105... telnet: connect to address 52.60.67.105: Connection refused thank you so much for helping here. ... View more

I only can upload one of my error file. but most of them are the same, connection failed...and seemed ambari-metrics never can be started. Only ZK is up. so please help. thank you very much. ... View more

I used the following to setup these recommended size I found in this community help ams-env :: collector_heapsize = 2048 ams-hbase-env :: hbase_master_heapsize = 512 ams-hbase-env :: hbase_master_xmn = 102 ams-hbase-env :: hbase_regionserver_heapsize = 4096 ams-hbase-env :: regionserver_xmn_size = 512 604-hdp-errors.png ... View more

Thank you so much, let me try it and get back to you in couple of hours. ... View more

you are right, Since my setting all working before I never use VCP and just thought about this is may be a cause. I will work on it and get back to you. thanks again. ... View more

Thank you so much for this quick respond. why are IP and hostname changed? my network.hostname=localhost, no change on /etc/sysconfig/network file. did I missed anything? is this AWS/EC2 networking DNS problem? I already created a ticket to AWS support. do you think I can fix it on my side. ... View more