@Jay regarding my question what actuality happens if I set the value - 100 that is more then ambari recomended , dose ambari will take the value 100 anyway ? , or the limit is 96 ? ... View more

so in case we verify the logs of gc by http://gceasy.io/ , and we see that Driver isn't doing full garbage collection , that what are the next steps that we need to do ? ... View more

@Falix , regarding to you answer - "Check if Driver is doing full garbage collection" , please described how to do that? ... View more

@Sandeep , I will explain why we want logs under the datanode machine /var/log/spark2 , maybe you have suggestion We notice that the messages that sent by the executor ( on datanode ) can not be delivered to the driver ( on the master machine ), and from the yarn logs we can see that warning <code>WARN executor.Executor:Issue communicating with driver in heartbeater My question is - what could be the reasons that driver ( master1 machine ) not get the heartbeat from the workers machines second how to debug this problem about - communicating with driver in heartbeater ? ... View more

we need the debug for Spark Thrift server , we have issue when heartbeat from datanode machine not communicated with the driver , so this is the reason that we need debug mode on for Spark Thrift server ... View more

because the logs are huge , do you want to search specific sting in the logs ? ... View more

about the logs , please remind me what the logs that you want to look for ? ... View more

this is the relevant info from the file ( long file , but I think you want to look on the relevant disks ) <value> /data/sdb/hadoop/hdfs/data,/data/sdc/hadoop/hdfs/data,/data/sdd/hadoop/hdfs/data,/data/sde/hadoop/hdfs/data </value> <source>hdfs-site.xml</source> </property> <value> /data/sdb/hadoop/hdfs/data,/data/sdc/hadoop/hdfs/data,/data/sdd/hadoop/hdfs/data,/data/sde/hadoop/hdfs/data </value> <source>hdfs-site.xml</source> </property> ... View more