Support Questions

Find answers, ask questions, and share your expertise

Spark for Elasticsearch Alerting

New Contributor


My use case is to frequently (like every 5min using oozie for exemple) query different Elasticsearch hosts then to analyze the query's result, and then notify if there is something wrong.

I should send notifications before the next round of queries.

Is Spark fast enough for alerting (e.g. within a minute it queries ES) ?

Thank you



Spark makes sense here only if you are doing complex calculations on your search results, or your result set is very large (in which case you're more likely to have problems on the Elastic side with search and retrieval speed).

A better way of doing something like this would be with a script direct against elastic, or, much better, using something like NiFi to run your queries and send out alerts. It will also do the scheduling for you, and yes, if plenty fast for this sort of use case with significantly lower resource overhead than a full spark context.

You can also consider doing things like using spark streaming / storm to do your anomaly detection on the way into elastic, as which point you're really just doing stream analytics, and the Elastic side of it is not relevant to the performance problem.

Take a Tour of the Community
Don't have an account?
Your experience may be limited. Sign in to explore more.