Metron ElasticSearch too many indexes

[#1]

Hi,

I have a cluster of Metron running, by default a new index is roll out each hour per topic. After a month worth of indexes, Kibana dashboard is a bit slow when querying ElasticSearch. Is there a way to configure or combine these indexes so that the query is a bit faster?

I believe Metron is designed to have long term data storage, is a month worth of data too long? Have anyone uses Metron to store data longer than that or experience in tuning for the best use case in term of time or the number of Elastic indexes?

Any guidance is greatly appreciated.

[#2 · Accepted solution]

You should be able to change the "es.date.format" in the global config to something less granular than each hour. For example, you could change the default of ""yyyy.MM.dd.HH" to ""yyyy.MM.dd" which would cause the indexes to roll every day instead. See this section in the READMEs for more info: https://github.com/apache/metron/tree/master/metron-platform/metron-common#global-configuration.

[#3]

Thank you so much @rmerriman

shame on me I was not aware of this global.json configuration file at all.

[#4]

BTW, I realized the file global.json gets overwritten by metron/config on Ambari. There's a section called global.json template

Just in case someone might find it useful.