as all know the heartbeat is a signal sent periodically in order to indicate normal operation of the node or synchronize with other parts of the system in our system we have 5 workers machine , while executes run on 3 of them our system include 5 datanodes machines ( workers ) , and 3 master machines , hadoop version is 2.6.4 and thrift server install on the first master1 machine ( and driver is in master1 ) In Spark the heartbeats are the messages sent by executors ( from workers machines ) to the driver.( master1 machine ) the message is represented by case class org.apache.spark.Heartbeat The message is then received by the driver through org.apache.spark.HeartbeatReceiver#receiveAndReply(context: RpcCallContext) method. The driver: the main purpose of heartbeats consists on checking if given node is still alive ( from worker machine to master1 machine ) The driver verifies it at fixed interval (defined in spark.network.timeoutInterval entry) by sending ExpireDeadHosts message to itself. When the message is handled, the driver checks for the executors with no recent heartbeats. until now I explain the concept We notice that the messages sent by the executor can not be delivered to the driver , and from the yarn logs we can see that warning WARN executor.Executor: Issue communicating with driver in heartbeater My question is - what could be the reasons that driver ( master1 machine ) not get the heartbeat from the workers machines
... View more
