02-20-2017 04:53 PM
Hi, We need to find a way to maintain and search logs for the Long running Sprk streaming jobs on YARN. We have Log aggregation disabled in our cluster. We are thinking about Solr/Elastic search and may be Flume or Kafka to read the Sprk job logs.
any suggestions on how to implement search the on these logs and easily manage them?
02-21-2017 12:26 AM
02-21-2017 07:26 AM
@mbigelow You are right. We turned it off because of the long runnig jobs.
Do you know any other ways to implement log serach other than Solr/elastic?
02-21-2017 08:13 AM
02-21-2017 08:33 AM
We eant to searh for key phrases and at the same time we want developers to look in to the raw logs too for their troubleshooting and alerts for specific errors.
02-21-2017 11:32 AM
02-21-2017 01:15 PM
The documentation for YARN log aggregation says that logs are aggregated after an application completes.
Streaming jobs run for a much longer duration and potentially don't ever terminate. I want to get the logs into HDFS for my streaming jobs before the application completes or terminates. What are the better ways to do it, since Log aggregation only do it after the jobs are completed.
02-21-2017 01:21 PM
02-21-2017 01:46 PM
So, if I set yarn.log-aggregation.retain-check-interval-seconds to 60 Seconds, It will send the logs to HDFS (every 60 seconds) even when the job was not finished? (Since streaming jobs run forever)