Yarn + how to avoid node manager that marked as lost
we have huge production Hadoop cluster, with HDP version 2.6.5 and ambari version 2.6.2.2 , and all machines are with OS RHEL 7.6 version
the cluster size is as the following :
Total workers machines - 425 ( each worker include data node and node manager service )
from time to time we get indication of lost one or two **node-manager** and this identified from Ambari as ( 424/425 when total node-manager are 425 )
in order to fix it we just restart the **node-manager** and this action fix the problem and as results we get 425/425
after some googling , we found the following parameters that maybe should be tune better
yarn.client.nodemanager-connect.max-wait-ms ( its configured to 60000 ms and we think to increase it )
yarn.client.nodemanager-connect.retry-interval-ms ( its configured to 10 sec ms and we think to increase it )
yarn.nm.liveness-monitor.expiry-interval-ms ( this parameter not configured yet and we think to add it with value of 1500000 ms )
based on above details , I will appreciate to get comments or others ideas
background:
NodeManager is LOST means that ResourceManager haven't received heartbeats from it for a duration of nm.liveness-monitor.expiry-interval-ms milliseconds (default is 10 minutes).

