Support Questions
Find answers, ask questions, and share your expertise

Monitoring Apache Spark Logs and the Dynamic App/Driver logs


We are running a spark-streaming application on a standalone setup (version 1.6).

The logging in spark seems to be a bit scattered and I am attempting to configure a nagios log file monitor that checks for certain "errors" in log files and sends out alerts.


My current understanding in regards to logs for spark are the following:


  1. Spark-Worker has it's own logs and in my case it's written to a static location /var/log/spark/spark-worker.out
  2. Spark-Master has it's own logs and in my case it's written to a static location /var/log/spark/spark-master.out
  3. I can configure the file under /etc/spark/conf/ to alter format, appenders etc.. for spark-worker and spark-master logs

Now for Driver and Spark/Executor App logs It seems the location for these logs are dynamic and spark will generated new directories under /var/run/spark/work in my case.


My issue:

Monitoring the static location log files is straight forward for spark-worker and spark-master. I am a bit confused as to how the dynamic logs for app and drivers can be monitored.

From what I read in the documentation, it seems upon spark-submit I can pass a -D option with a location to a file.


Can this be configured to stream the logs to a local syslog in a static location and then have nagios monitor that static log?


What have others done in this case?