Ambari Hosts Tab does not show hosts

[#1]

Hi, we are using ambari 2.2.2.0 and we are unable to see the hosts when i click on hosts tab on the ambari UI. it loads forever or it says no hosts available. I have restarted ambari server and ambari agents capture.pngbut no luck. Please guide me to fix this issue

[#2]

@pv poreddy

You can login to Ambari DB and check whether you have entries in the Database or not.

psql -U ambari
Password: ******

You can get the password from,

# cat /etc/ambari-server/conf/ambari.properties | grep password
server.jdbc.user.passwd=/etc/ambari-server/conf/password.dat
# cat /etc/ambari-server/conf/password.dat

Execute below command to get the hostlist from DB,

ambari=> select * from hosts;

It should give you some list of hosts. Check if you have it or not.

If not, check the previous backups of Ambari-DB in the same tables.

[#3]

@pv poreddy

Check the Id's in hoststate tables as well. If your ambari-agent on the hosts are running it will show (HEALTHY) for that ID. Check whether you it is for correct host. You have to check the ID in hosts table and match up with hoststate. Let me know if you need anything.

[#4]

Hi, i too have same problem, please suggest me from next next.

	ambari=> select *
from hosts;
	 host_id | host_name  | cpu_count | ph_cpu_count | cpu_info |
discovery_status | host_attributes |  ipv4  | ipv6  | public_host_name |
last_registration_time | os_arch | os_info | os_type
	|  rack_info 
| total_mem
	---------+------------+-----------+--------------+----------+------------------+-------------------------------------------------------------------------------------------------------------------------------

	---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

	---------------------------------------------------------------------------------------------------+----------------+----------------+------------------+------------------------+---------+---------+---------

	+---------------+-----------

	  4 | master  | 
4 |  4 |  |  |
{"interfaces":"eno16780,lo,virbr0","os_family":"redhat","kernel":"Linux","timezone":"EST","kernel_release":"3.10.0-862.6.3.el7

	.x86_64","os_release_version":"7.5.1804","physicalprocessors_count":"4","hardware_isa":"x86_64","kernel_majorversion":"3.10","kernel_version":"3.10.0","mac_address":"00:0C:29:47:9D:24","swap_free":"75.10
GB"

	,"swap_size":"90.00
GB","selinux_enabled":"true","hardware_model":"x86_64","processors_count":"4"}
| <IPAddress166> | <IPAddress166> | master  |  1542262557154 | x86_64  | 
| centos7

	| /default-rack
|  32781400

	  3 | hnode3.com |  4 |  4 |  |  |
{"interfaces":"eno16780,lo,virbr0","os_family":"redhat","kernel":"Linux","timezone":"IST","kernel_release":"3.10.0-327.el7.x86

	_64","os_release_version":"7.2.1511","physicalprocessors_count":"4","hardware_isa":"x86_64","kernel_majorversion":"3.10","kernel_version":"3.10.0","mac_address":"00:50:56:B1:4B:92","swap_free":"29.80
GB","sw

	ap_size":"29.80
GB","selinux_enabled":"false","hardware_model":"x86_64","processors_count":"4"}  | <IPAddress60>  | <IPAddress60>  | hnode3.com  |  1542890750770 | x86_64  | 
| centos7

	| /default-rack
|  16268760

	  2 | hnode2.com |  4 |  4 |  |  |
{"interfaces":"eno16780,lo,virbr0","os_family":"redhat","kernel":"Linux","timezone":"IST","kernel_release":"3.10.0-327.el7.x86

	_64","os_release_version":"7.2.1511","physicalprocessors_count":"4","hardware_isa":"x86_64","kernel_majorversion":"3.10","kernel_version":"3.10.0","mac_address":"00:50:56:B1:98:CE","swap_free":"29.80
GB","sw

	ap_size":"29.80
GB","selinux_enabled":"false","hardware_model":"x86_64","processors_count":"4"}  | <IPAddress59>  |<IPAddress59>  | hnode2.com  |  1542890767448 | x86_64  | 
| centos7

	| /default-rack
|  16268760

	  1 | hnode1.com |  4 |  4 |  |  |
{"interfaces":"eno16780,lo,virbr0","os_family":"redhat","kernel":"Linux","timezone":"IST","kernel_release":"3.10.0-327.el7.x86

	_64","os_release_version":"7.2.1511","physicalprocessors_count":"4","hardware_isa":"x86_64","kernel_majorversion":"3.10","kernel_version":"3.10.0","mac_address":"00:50:56:B1:20:32","swap_free":"29.80
GB","sw
	ap_size":"29.80
GB","selinux_enabled":"false","hardware_model":"x86_64","processors_count":"4"}  | <IPAddress81>  | <IPAddress81>  | hnode1.com  |  1542890776524 | x86_64  | 
| centos7	| /default-rack
|  16268760	(4 rows)

[#5]

@pv poreddy

If the "hosts" page is keep loading as seen in the "capture.png" screenshot then it means there is some inconsistency in the ambari DB. You must be getting some indication for the same inside your "/var/log/ambari-server/ambari-server.log". There must be some ERRORs/WARNINGs logged in the ambari server that will help us in finding the inconsistency.

- Can you please share the following file:

/var/log/ambari-server/ambari-server.log

.

Additionally can you please share the output of the following Rest API calls.

http://$AMBARI_HOSTNAME:8080/api/v1/clusters/$CLUSTER_NAME?fields=Clusters/health_report,Clusters/to...


http://$AMBARI_HOSTNAME:8080/api/v1/clusters/$CLUSTER_NAME/hosts?fields=Hosts/host_name,Hosts/mainte...

.

[#6]

@pv poreddy

can you check all the hosts forward and reverse lookup are working fine and resolving to the same hostname yo have defined, make sure iptables is disabled on all the hosts, and you check the hosts existences from DB as given by @nshelke

you can also check for any errors from ambari-server.log

you can also enable ambari-server DEBUG log from /etc/ambari-server/conf/log4j.properties for further debugging

[#7]

Ambari is https so i guess i have to give $Ambari_HOSTANME = hostname:8443. please let me know if i have to give $clustername field value which is equal to dfs.nameservices value in hdfs config so that i can post the rest api response

[#8]

capture.png . These are the errors i get when i look in to the ambari metrics. capture.png The error when i enable the developer tools in chrome. My ambari db is oracle database. I will have a look in to that and i will check hosts table meanwhile

[#9]

@Jay SenSharma @nshelke any inputs to solve would greatly help me. Thanks

[#10]

"Clusters" : {

    "cluster_name" : "prodHadoop01",
    "health_report" : {
      "Host/stale_config" : 1,
      "Host/maintenance_state" : 15,
      "Host/host_state/HEALTHY" : 15,
      "Host/host_state/UNHEALTHY" : 0,
      "Host/host_state/HEARTBEAT_LOST" : 1,
      "Host/host_state/INIT" : 0,
      "Host/host_status/HEALTHY" : 9,
      "Host/host_status/UNHEALTHY" : 0,
      "Host/host_status/UNKNOWN" : 1,
      "Host/host_status/ALERT" : 6
    },
    "total_hosts" : 16,
    "version" : "HDP-2.4"
  }
}

[#11]

I can see from the other Rest API calls one of the decommissioned node is still present and status is UNKNOWN.Does that mean the host table in ambari db is stale ?? If yes what needs to be done

[#12]

@nshelke I ran the select * from hosts. I have found some hostnames which should not be present in the hosts table. How should i proceed in correcting them.

[#13]

@nshelke I ran select * from hosts and i have some unwanted hosts in the table which defnitely show that it is inconsistent. Can you please help me as in what should i do?

[#14]

@nshelke How can we rectify these inconsistencies in ambari db

[#15]

@sparkifyed , see if you can login to decommission node and see if ambari-agent is stopped on that server. If not stop all hadoop services on decommission node and see if you are able to view "hosts" in Ambari.

It may be possible that Amabri-server is trying to connect to decommission server every time and hangs in between. Open a case with Hortonworks , if it is impacting your production system.

Regards,

Fahim

[#16]

Well I fixed this issue eventually by checking the ambari database hosts and one more table relating to hosts and have seen inconsistencies in the table.i have deleted the entries using the curl commands and restarted ambari and hosts tab works now. Before doing this I have checked the browser developer tools and seen the response where it is failing from the up part which also helped me to figure this issue. Hope it helps someone

[#17]

I too had this issue on HDP 2.6

ambari-server.log reported

ERROR: invalid page header in block 13760 of relation base/16995/67484

Just below that error, there was the query which threw that error. Runnig the same on Postgre threw the same error.

In my case ambri.alert_history was corrupt.

Below step fixed the issue

database=# SET zero_damaged_pages = on;
SET
database=# VACUUM FULL damaged_table;
WARNING: invalid page header in block 13748 of relation base/16995/67484; zeroing out page
WARNING: invalid page header in block 13749 of relation base/16995/67484; zeroing out page
WARNING: invalid page header in block 13757 of relation base/16995/67484; zeroing out page
WARNING: invalid page header in block 13758 of relation base/16995/67484; zeroing out page
WARNING: invalid page header in block 13759 of relation base/16995/67484; zeroing out page
WARNING: invalid page header in block 13760 of relation base/16995/67484; zeroing out page
WARNING: invalid page header in block 13762 of relation base/16995/67484; zeroing out page
WARNING: invalid page header in block 13782 of relation base/16995/67484; zeroing out page
(...)
WARNING: index "damaged_table_site_id" contains 14697831 row versions, but table contains 14709258 row versions
HINT: Rebuild the index with REINDEX.
WARNING: invalid page header in block 13762 of relation base/16995/67484; zeroing out page
WARNING: invalid page header in block 13816 of relation base/16995/67484; zeroing out page
WARNING: invalid page header in block 13817 of relation base/16995/67484; zeroing out page
WARNING: invalid page header in block 13818 of relation base/16995/67484; zeroing out page
(...)
WARNING: index "damaged_table_site_id" contains 14697498 row versions, but table contains 14709258 row versions
HINT: Rebuild the index with REINDEX.
VACUUM

database=# REINDEX TABLE damaged_table;