@Michael Bronson By default Spark2 has log level as WARN. Set it to INFO to get more context on what is going on in the driver and executor. More over the log will be locally available in Nodemanager when the container is still running. The easiest way is to go to spark UI (yarn application master UI) -> click on executors tab -> Here you should see stderr and stdout corresponding to driver and executors. Regarding the WARN on heartbeat , we'd need to check what driver is doing at that point. I think you already have asked another question with more details on driver and executor. ... View more

so in case we verify the logs of gc by http://gceasy.io/ , and we see that Driver isn't doing full garbage collection , that what are the next steps that we need to do ? ... View more

we need the debug for Spark Thrift server , we have issue when heartbeat from datanode machine not communicated with the driver , so this is the reason that we need debug mode on for Spark Thrift server ... View more

@Michael Bronson The warning message means that the Executor is unable to send the Heartbeat to the driver (might be network issue). This is just a warning message, but each failure increments heartbeat failure count and when we hit the maximum failures the executor will fail and exit with error. There are two configurations that we can tune to avoid this issue. spark.executor.heartbeat.maxFailures (default value: 60) Number of times an executor will try to send heartbeats to the driver before it gives up and exits (with exit code 56). spark.executor.heartbeatInterval ( default value: 10s ) Interval between each executor's heartbeats to the driver. Heartbeats let the driver know that the executor is still alive and update it with metrics for in-progress tasks. spark.executor.heartbeatInterval should be significantly less than spark.network.timeout ... View more

@Jay . please let me know if I understand it as the following let say that one of the replica spark2-hdp-yarn-archive.tar.gz , is corrupted when I run this CLI su - hdfs -c "hdfs fsck /hdp/apps/2.6.4.0-91/spark2/spark2-hdp-yarn-archive.tar.gz" dose its actually means that fsck will replace the bad one with the good one and status finally will be HEALTHY ? ... View more

You can do tail in namenode and datanode log, also you can redirect output to dummy log file during restart. #tailf <namenode log> >/tmp/namenode-`hostname`.log #tailf <datanode log> >/tmp/datanode-`hostname`.log ... View more

@Jonathan Sneep thank you so much ... View more

Is there any way to restart an ABORTED or FAILED request? ... View more

@Jay we run the service check but it fail on python time out , is any other idea to increase the logs ? second from where we get the - yarn-yarn-resourcemanager-.... files ? they are not written in the log4j so I not understand how they create -rw-r--r-- 1 yarn hadoop 1847 Aug 27 12:03 yarn-yarn-resourcemanager-master02.sys76.com.out.1 -rw-r--r-- 1 yarn hadoop 1052 Aug 27 12:05 yarn-yarn-resourcemanager-master02.sys76.com.log.10 -rw-r--r-- 1 yarn hadoop 1180 Aug 27 12:05 yarn-yarn-resourcemanager-master02.sys76.com.log.9 ... View more