@Michael Bronson By default Spark2 has log level as WARN. Set it to INFO to get more context on what is going on in the driver and executor. More over the log will be locally available in Nodemanager when the container is still running. The easiest way is to go to spark UI (yarn application master UI) -> click on executors tab -> Here you should see stderr and stdout corresponding to driver and executors. Regarding the WARN on heartbeat , we'd need to check what driver is doing at that point. I think you already have asked another question with more details on driver and executor. ... View more

@Michael Bronson, You will see Spark thrift server and Spark History Server in /var/log/spark2. The above log4j i proposed is for spark applications (which will not be stored in /var/log/spark2 rather you should use yarn logs command and extract the log). What is it you want to enable DEBUG logging for? spark application , Spark Thrift server or Spark History Server? ... View more

@Michael Bronson Use below : # Set everything to be logged to the console log4j.rootCategory=DBEUG, console log4j.appender.console=org.apache.log4j.ConsoleAppender log4j.appender.console.target=System.err log4j.appender.console.layout=org.apache.log4j.PatternLayout log4j.appender.console.layout.ConversionPattern=%d{yy/MM/dd HH:mm:ss}%p %c{1}:%m%n # Settings to quiet third party logs that are too verbose log4j.logger.org.eclipse.jetty=WARN log4j.logger.org.eclipse.jetty.util.component.AbstractLifeCycle=ERROR log4j.logger.org.apache.spark.repl.SparkIMain$exprTyper=INFO log4j.logger.org.apache.spark.repl.SparkILoop$SparkILoopInterpreter=INFO log4j.logger.org.apache.spark.metrics.MetricsConfig=DEBUG log4j.logger.org.apache.spark.deploy.yarn.Client=DEBUG ... View more

@David Hoyle The code structure has changed since this article was written. 1) checkout trunk 2) brew install protobuf250 (protobuf is needed to build hadoop) 3) Build using : mvn clean package -Phdds -Pdist -Dtar -DskipShade -DskipTests -Dmaven.javadoc.skip=true edit: updated the proto version ... View more

Not Yet... ... View more