Executor is a distributed agent that is responsible for executing tasks. this is very clear but how to know if there are any issues with the executors that runs from datanode machine? I asking this question because when I am looking on the datanode machine I not see any logs that represented the executors , and I not understand how to trace problems about the exectores the second important quastion: heartbeat are sent from the executor to the driver what are the logs that represented this heartbeat ? how to know if there are any issue with heartbeat sending ? ... View more

we have HDP cluster version 2.6.4 , ambari version 2.6.1 with 8 workers machines ( datanode machines ) on each worker machines we have the folder /var/log/spark2 but no any logs under this folder on the master machines - when the spark thrift running we have the /var/log/spark2 and logs are created corectly on this machine but not on the datanode machine spark thrift restasrt twice . but this not help to create the logs on the datanode machine any other ideas what we can do ? ... View more