@Tarun , sorry , yes you rught all cluster are twice then the Vcore value , its my fault ... View more

@Tarun , so the final answer is to set 80% from 32 second , I will give you interesting info some of our ambari clusters the scrollbar is limit as the vcore ( for example if we have 16 vcore then scroll bar is until 16 and some cluster the scrolbar is twice then the real vcore ( for example if we have 16 vcore the scroll bar is 32 ) so how you explain this? ... View more

@Tarun , first thank you , but I will give you interesting example we have ambari cluster with 3 datanode each data node have 32 Vcore with 2 thread ( as we get from lscpu ) so according to your answer we need to set 32 to yarn.nodemanager.resource.cpu-vcores but as you can see from the following picture the scrollbar is until 64Vcore , and seems that during ambari installation , ambari calculate 2 X 32 and that gives us the scroll-bar to 64 so regarding my example , do you still with opinion that we need to set 80% of 32 ? ... View more

@Jay , can you look on my question on thred - https://community.hortonworks.com/questions/221835/how-to-calculate-the-yarnnodemanagerresourcecpu-vc-1.html , I will really apritiate your opinion ... View more

@Jay regarding my question what actuality happens if I set the value - 100 that is more then ambari recomended , dose ambari will take the value 100 anyway ? , or the limit is 96 ? ... View more

so in case we verify the logs of gc by http://gceasy.io/ , and we see that Driver isn't doing full garbage collection , that what are the next steps that we need to do ? ... View more

@Falix , regarding to you answer - "Check if Driver is doing full garbage collection" , please described how to do that? ... View more

@Sandeep , I will explain why we want logs under the datanode machine /var/log/spark2 , maybe you have suggestion We notice that the messages that sent by the executor ( on datanode ) can not be delivered to the driver ( on the master machine ), and from the yarn logs we can see that warning <code>WARN executor.Executor:Issue communicating with driver in heartbeater My question is - what could be the reasons that driver ( master1 machine ) not get the heartbeat from the workers machines second how to debug this problem about - communicating with driver in heartbeater ? ... View more