Executor is a distributed agent that is responsible for executing tasks. this is very clear but how to know if there are any issues with the executors that runs from datanode machine? I asking this question because when I am looking on the datanode machine I not see any logs that represented the executors , and I not understand how to trace problems about the exectores the second important quastion: heartbeat are sent from the executor to the driver what are the logs that represented this heartbeat ? how to know if there are any issue with heartbeat sending ? ... View more

we have HDP cluster version 2.6.4 , ambari version 2.6.1 with 8 workers machines ( datanode machines ) on each worker machines we have the folder /var/log/spark2 but no any logs under this folder on the master machines - when the spark thrift running we have the /var/log/spark2 and logs are created corectly on this machine but not on the datanode machine spark thrift restasrt twice . but this not help to create the logs on the datanode machine any other ideas what we can do ? ... View more

as all know the heartbeat is a signal sent periodically in order to indicate normal operation of the node or synchronize with other parts of the system in our system we have 5 workers machine , while executes run on 3 of them our system include 5 datanodes machines ( workers ) , and 3 master machines , hadoop version is 2.6.4 and thrift server install on the first master1 machine ( and driver is in master1 ) In Spark the heartbeats are the messages sent by executors ( from workers machines ) to the driver.( master1 machine ) the message is represented by case class org.apache.spark.Heartbeat The message is then received by the driver through org.apache.spark.HeartbeatReceiver#receiveAndReply(context: RpcCallContext) method. The driver: the main purpose of heartbeats consists on checking if given node is still alive ( from worker machine to master1 machine ) The driver verifies it at fixed interval (defined in spark.network.timeoutInterval entry) by sending ExpireDeadHosts message to itself. When the message is handled, the driver checks for the executors with no recent heartbeats. until now I explain the concept We notice that the messages sent by the executor can not be delivered to the driver , and from the yarn logs we can see that warning WARN executor.Executor: Issue communicating with driver in heartbeater My question is - what could be the reasons that driver ( master1 machine ) not get the heartbeat from the workers machines ... View more