we want to find the **approach / test /cli / API** that gives us the results about HeartBeat Lost between Ambari agent to Ambari server HeartBeat Lost could be as results of poor connection between Ambari agent to Ambari server or because Ambari server was down for along time , etc   Note - from Ambari GUI the machine with **HeartBeat Lost** state usually colored by yellow state         clarification:     the case as described here appears when `ambari-agent status` is in running state as the following ambari-agent status Found ambari-agent PID: 119315 ambari-agent running. Agent PID at: /run/ambari-agent/ambari-agent.pid Agent out at: /var/log/ambari-agent/ambari-agent.out Agent log at: /var/log/ambari-agent/ambari-agent.log ... View more

we have old hadoop cluster based on HDP from hortonworks version HDP 2.6.4 cluster include 2 namenode services when one is the standby namenode and the second is the active namenode , all machines in the cluster are rhel 7.2 version , and we not see any problem on OS level   also cluster include 12 workers machines ( worker include the datanode and node manager services )   the story begin when we get alerts from the smoke test script that complain about "`Detected pause in JVM or host machine`" on the standby namenode ,   so based on that we decided to increase the namenode heap size from 60G to 100G   and above setting was based on table that show how much memory to set according to number of files in HDFS   and according to the table we decided to set the namenode heapsize to 100G and then we restart the HDFS service   after HDFS is completely restarted , we still see the messages about `Detected pause in JVM or host machine` , and this is really strange because we almost twice the namenode heap size   so we start to perform deeply testing as by `jstat`   for example we get from jsat low very of FGCT that is really good values and not point on namenode heap problem ( 1837 is the HDFS PID number )     # /usr/jdk64/jdk1.8.0_112/bin/jstat -gcutil 1837 10 10 S0 S1 E O M CCS YGC YGCT FGC FGCT GCT 0.00 1.95 32.30 34.74 97.89 - 197 173.922 2 1.798 175.720 0.00 1.95 32.30 34.74 97.89 - 197 173.922 2 1.798 175.720 0.00 1.95 32.30 34.74 97.89 - 197 173.922 2 1.798 175.720 0.00 1.95 32.30 34.74 97.89 - 197 173.922 2 1.798 175.720 0.00 1.95 32.30 34.74 97.89 - 197 173.922 2 1.798 175.720 0.00 1.95 32.30 34.74 97.89 - 197 173.922 2 1.798 175.720 0.00 1.95 32.30 34.74 97.89 - 197 173.922 2 1.798 175.720 0.00 1.95 32.30 34.74 97.89 - 197 173.922 2 1.798 175.720 0.00 1.95 32.30 34.74 97.89 - 197 173.922 2 1.798 175.720 0.00 1.95 32.30 34.74 97.89 - 197 173.922 2 1.798 175.720   and here is the messages from namenode logs     2022-10-27 14:04:49,728 INFO util.JvmPauseMonitor (JvmPauseMonitor.java:run(196)) - Detected pause in JVM or host machine (eg GC): pause of approximately 2044ms 2022-10-27 16:21:33,973 INFO util.JvmPauseMonitor (JvmPauseMonitor.java:run(196)) - Detected pause in JVM or host machine (eg GC): pause of approximately 2524ms 2022-10-27 17:31:35,333 INFO util.JvmPauseMonitor (JvmPauseMonitor.java:run(196)) - Detected pause in JVM or host machine (eg GC): pause of approximately 2444ms 2022-10-27 18:55:55,387 INFO util.JvmPauseMonitor (JvmPauseMonitor.java:run(196)) - Detected pause in JVM or host machine (eg GC): pause of approximately 2134ms 2022-10-27 19:42:00,816 INFO util.JvmPauseMonitor (JvmPauseMonitor.java:run(196)) - Detected pause in JVM or host machine (eg GC): pause of approximately 2153ms 2022-10-27 20:50:23,624 INFO util.JvmPauseMonitor (JvmPauseMonitor.java:run(196)) - Detected pause in JVM or host machine (eg GC): pause of approximately 2050ms 2022-10-27 21:07:01,240 INFO util.JvmPauseMonitor (JvmPauseMonitor.java:run(196)) - Detected pause in JVM or host machine (eg GC): pause of approximately 2343ms 2022-10-27 23:53:00,507 INFO util.JvmPauseMonitor (JvmPauseMonitor.java:run(196)) - Detected pause in JVM or host machine (eg GC): pause of approximately 2120ms 2022-10-28 00:43:30,633 INFO util.JvmPauseMonitor (JvmPauseMonitor.java:run(196)) - Detected pause in JVM or host machine (eg GC): pause of approximately 1811ms 2022-10-28 00:53:35,120 INFO util.JvmPauseMonitor (JvmPauseMonitor.java:run(196)) - Detected pause in JVM or host machine (eg GC): pause of approximately 2192ms 2022-10-28 02:07:39,660 INFO util.JvmPauseMonitor (JvmPauseMonitor.java:run(196)) - Detected pause in JVM or host machine (eg GC): pause of approximately 2353ms 2022-10-28 02:49:25,018 INFO util.JvmPauseMonitor (JvmPauseMonitor.java:run(196)) - Detected pause in JVM or host machine (eg GC): pause of approximately 1698ms 2022-10-28 03:00:20,592 INFO util.JvmPauseMonitor (JvmPauseMonitor.java:run(196)) - Detected pause in JVM or host machine (eg GC): pause of approximately 2432ms 2022-10-28 05:02:15,093 INFO util.JvmPauseMonitor (JvmPauseMonitor.java:run(196)) - Detected pause in JVM or host machine (eg GC): pause of approximately 2016ms 2022-10-28 06:52:46,672 INFO util.JvmPauseMonitor (JvmPauseMonitor.java:run(196)) - Detected pause in JVM or host machine (eg GC): pause of approxim as we can see each 1-2 hours message about pause in `JVM or host machine` is appears   we checked the number of files in HDFS and number of file is 7 million files   so what else we can do , we can increase a little bit the namenode heap size but my feeling is that heap size is really enough   ... View more

Hi all   We have Ambari HDP cluster ( HDP version - 2.6.4 ) , with 420 workers linux machines ( when each worker include data node and node manager service )   Unfortunately Ambari DB is damaged , and we not have Ambari DB dump , so we cant recover Ambari DB  , so actually we not have Ambari and Ambari GUI   But HDFS disks on workers machines include HDFS data , and name node is still working with all data  as ( journal/hdfsha/current/ ) and ( namenode/current ) So HDFS works without Ambari     So regarding what I said until now - it is possible install new Ambari cluster from scratch , and then add existing working HDFS data to the cluster ? Dose hortonworks / cloudera have procedure for this process ? ... View more