Created 08-08-2017 12:38 AM
Hi, we are using ambari 2.2.2.0 and we are unable to see the hosts when i click on hosts tab on the ambari UI. it loads forever or it says no hosts available. I have restarted ambari server and ambari agents capture.pngbut no luck. Please guide me to fix this issue
Created 08-08-2017 12:50 AM
You can login to Ambari DB and check whether you have entries in the Database or not.
psql -U ambari Password: ******
You can get the password from,
# cat /etc/ambari-server/conf/ambari.properties | grep password server.jdbc.user.passwd=/etc/ambari-server/conf/password.dat # cat /etc/ambari-server/conf/password.dat
Execute below command to get the hostlist from DB,
ambari=> select * from hosts;
It should give you some list of hosts. Check if you have it or not.
If not, check the previous backups of Ambari-DB in the same tables.
Created 08-09-2017 12:41 AM
Check the Id's in hoststate tables as well. If your ambari-agent on the hosts are running it will show (HEALTHY) for that ID. Check whether you it is for correct host. You have to check the ID in hosts table and match up with hoststate. Let me know if you need anything.
Created 11-23-2018 04:31 PM
Hi, i too have same problem, please suggest me from next next.
ambari=> select * from hosts; host_id | host_name | cpu_count | ph_cpu_count | cpu_info | discovery_status | host_attributes | ipv4 | ipv6 | public_host_name | last_registration_time | os_arch | os_info | os_type | rack_info | total_mem ---------+------------+-----------+--------------+----------+------------------+------------------------------------------------------------------------------------------------------------------------------- --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------+----------------+----------------+------------------+------------------------+---------+---------+--------- +---------------+----------- 4 | master | 4 | 4 | | | {"interfaces":"eno16780,lo,virbr0","os_family":"redhat","kernel":"Linux","timezone":"EST","kernel_release":"3.10.0-862.6.3.el7 .x86_64","os_release_version":"7.5.1804","physicalprocessors_count":"4","hardware_isa":"x86_64","kernel_majorversion":"3.10","kernel_version":"3.10.0","mac_address":"00:0C:29:47:9D:24","swap_free":"75.10 GB" ,"swap_size":"90.00 GB","selinux_enabled":"true","hardware_model":"x86_64","processors_count":"4"} | <IPAddress166> | <IPAddress166> | master | 1542262557154 | x86_64 | | centos7 | /default-rack | 32781400 3 | hnode3.com | 4 | 4 | | | {"interfaces":"eno16780,lo,virbr0","os_family":"redhat","kernel":"Linux","timezone":"IST","kernel_release":"3.10.0-327.el7.x86 _64","os_release_version":"7.2.1511","physicalprocessors_count":"4","hardware_isa":"x86_64","kernel_majorversion":"3.10","kernel_version":"3.10.0","mac_address":"00:50:56:B1:4B:92","swap_free":"29.80 GB","sw ap_size":"29.80 GB","selinux_enabled":"false","hardware_model":"x86_64","processors_count":"4"} | <IPAddress60> | <IPAddress60> | hnode3.com | 1542890750770 | x86_64 | | centos7 | /default-rack | 16268760 2 | hnode2.com | 4 | 4 | | | {"interfaces":"eno16780,lo,virbr0","os_family":"redhat","kernel":"Linux","timezone":"IST","kernel_release":"3.10.0-327.el7.x86 _64","os_release_version":"7.2.1511","physicalprocessors_count":"4","hardware_isa":"x86_64","kernel_majorversion":"3.10","kernel_version":"3.10.0","mac_address":"00:50:56:B1:98:CE","swap_free":"29.80 GB","sw ap_size":"29.80 GB","selinux_enabled":"false","hardware_model":"x86_64","processors_count":"4"} | <IPAddress59> |<IPAddress59> | hnode2.com | 1542890767448 | x86_64 | | centos7 | /default-rack | 16268760 1 | hnode1.com | 4 | 4 | | | {"interfaces":"eno16780,lo,virbr0","os_family":"redhat","kernel":"Linux","timezone":"IST","kernel_release":"3.10.0-327.el7.x86 _64","os_release_version":"7.2.1511","physicalprocessors_count":"4","hardware_isa":"x86_64","kernel_majorversion":"3.10","kernel_version":"3.10.0","mac_address":"00:50:56:B1:20:32","swap_free":"29.80 GB","sw ap_size":"29.80 GB","selinux_enabled":"false","hardware_model":"x86_64","processors_count":"4"} | <IPAddress81> | <IPAddress81> | hnode1.com | 1542890776524 | x86_64 | | centos7 | /default-rack | 16268760 (4 rows)
Created 08-08-2017 03:17 AM
If the "hosts" page is keep loading as seen in the "capture.png" screenshot then it means there is some inconsistency in the ambari DB. You must be getting some indication for the same inside your "/var/log/ambari-server/ambari-server.log". There must be some ERRORs/WARNINGs logged in the ambari server that will help us in finding the inconsistency.
- Can you please share the following file:
/var/log/ambari-server/ambari-server.log
.
Additionally can you please share the output of the following Rest API calls.
http://$AMBARI_HOSTNAME:8080/api/v1/clusters/$CLUSTER_NAME?fields=Clusters/health_report,Clusters/to... http://$AMBARI_HOSTNAME:8080/api/v1/clusters/$CLUSTER_NAME/hosts?fields=Hosts/host_name,Hosts/mainte...
.
Created 08-08-2017 12:50 PM
can you check all the hosts forward and reverse lookup are working fine and resolving to the same hostname yo have defined, make sure iptables is disabled on all the hosts, and you check the hosts existences from DB as given by @nshelke
you can also check for any errors from ambari-server.log
you can also enable ambari-server DEBUG log from /etc/ambari-server/conf/log4j.properties for further debugging
Created 08-08-2017 02:05 PM
Ambari is https so i guess i have to give $Ambari_HOSTANME = hostname:8443. please let me know if i have to give $clustername field value which is equal to dfs.nameservices value in hdfs config so that i can post the rest api response
Created 08-08-2017 02:09 PM
capture.png . These are the errors i get when i look in to the ambari metrics. capture.png The error when i enable the developer tools in chrome. My ambari db is oracle database. I will have a look in to that and i will check hosts table meanwhile
Created 08-08-2017 02:10 PM
@Jay SenSharma @nshelke any inputs to solve would greatly help me. Thanks
Created 08-08-2017 02:32 PM
"Clusters" : {
"cluster_name" : "prodHadoop01", "health_report" : { "Host/stale_config" : 1, "Host/maintenance_state" : 15, "Host/host_state/HEALTHY" : 15, "Host/host_state/UNHEALTHY" : 0, "Host/host_state/HEARTBEAT_LOST" : 1, "Host/host_state/INIT" : 0, "Host/host_status/HEALTHY" : 9, "Host/host_status/UNHEALTHY" : 0, "Host/host_status/UNKNOWN" : 1, "Host/host_status/ALERT" : 6 }, "total_hosts" : 16, "version" : "HDP-2.4" } }
Created 08-08-2017 02:39 PM
I can see from the other Rest API calls one of the decommissioned node is still present and status is UNKNOWN.Does that mean the host table in ambari db is stale ?? If yes what needs to be done
Created 08-09-2017 12:14 AM
@nshelke I ran the select * from hosts. I have found some hostnames which should not be present in the hosts table. How should i proceed in correcting them.
Created 08-09-2017 12:16 AM
@nshelke I ran select * from hosts and i have some unwanted hosts in the table which defnitely show that it is inconsistent. Can you please help me as in what should i do?
Created 08-09-2017 12:18 AM
@nshelke How can we rectify these inconsistencies in ambari db
Created 08-11-2017 11:00 AM
@sparkifyed , see if you can login to decommission node and see if ambari-agent is stopped on that server. If not stop all hadoop services on decommission node and see if you are able to view "hosts" in Ambari.
It may be possible that Amabri-server is trying to connect to decommission server every time and hangs in between. Open a case with Hortonworks , if it is impacting your production system.
Regards,
Fahim
Created 08-16-2017 03:47 AM
Well I fixed this issue eventually by checking the ambari database hosts and one more table relating to hosts and have seen inconsistencies in the table.i have deleted the entries using the curl commands and restarted ambari and hosts tab works now. Before doing this I have checked the browser developer tools and seen the response where it is failing from the up part which also helped me to figure this issue. Hope it helps someone
Created 09-27-2017 06:13 PM
I too had this issue on HDP 2.6
ambari-server.log reported
ERROR: invalid page header in block 13760 of relation base/16995/67484
Just below that error, there was the query which threw that error. Runnig the same on Postgre threw the same error.
In my case ambri.alert_history was corrupt.
Below step fixed the issue
database=# SET zero_damaged_pages = on; SET database=# VACUUM FULL damaged_table; WARNING: invalid page header in block 13748 of relation base/16995/67484; zeroing out page WARNING: invalid page header in block 13749 of relation base/16995/67484; zeroing out page WARNING: invalid page header in block 13757 of relation base/16995/67484; zeroing out page WARNING: invalid page header in block 13758 of relation base/16995/67484; zeroing out page WARNING: invalid page header in block 13759 of relation base/16995/67484; zeroing out page WARNING: invalid page header in block 13760 of relation base/16995/67484; zeroing out page WARNING: invalid page header in block 13762 of relation base/16995/67484; zeroing out page WARNING: invalid page header in block 13782 of relation base/16995/67484; zeroing out page (...) WARNING: index "damaged_table_site_id" contains 14697831 row versions, but table contains 14709258 row versions HINT: Rebuild the index with REINDEX. WARNING: invalid page header in block 13762 of relation base/16995/67484; zeroing out page WARNING: invalid page header in block 13816 of relation base/16995/67484; zeroing out page WARNING: invalid page header in block 13817 of relation base/16995/67484; zeroing out page WARNING: invalid page header in block 13818 of relation base/16995/67484; zeroing out page (...) WARNING: index "damaged_table_site_id" contains 14697498 row versions, but table contains 14709258 row versions HINT: Rebuild the index with REINDEX. VACUUM database=# REINDEX TABLE damaged_table;