Support Questions
Find answers, ask questions, and share your expertise

Metron ElasticSearch too many indexes

Solved Go to solution
Highlighted

Metron ElasticSearch too many indexes

Contributor

Hi,

I have a cluster of Metron running, by default a new index is roll out each hour per topic. After a month worth of indexes, Kibana dashboard is a bit slow when querying ElasticSearch. Is there a way to configure or combine these indexes so that the query is a bit faster?

I believe Metron is designed to have long term data storage, is a month worth of data too long? Have anyone uses Metron to store data longer than that or experience in tuning for the best use case in term of time or the number of Elastic indexes?

Any guidance is greatly appreciated.

1 ACCEPTED SOLUTION

Accepted Solutions

Re: Metron ElasticSearch too many indexes

Explorer

You should be able to change the "es.date.format" in the global config to something less granular than each hour. For example, you could change the default of ""yyyy.MM.dd.HH" to ""yyyy.MM.dd" which would cause the indexes to roll every day instead. See this section in the READMEs for more info: https://github.com/apache/metron/tree/master/metron-platform/metron-common#global-configuration.

View solution in original post

3 REPLIES 3

Re: Metron ElasticSearch too many indexes

Explorer

You should be able to change the "es.date.format" in the global config to something less granular than each hour. For example, you could change the default of ""yyyy.MM.dd.HH" to ""yyyy.MM.dd" which would cause the indexes to roll every day instead. See this section in the READMEs for more info: https://github.com/apache/metron/tree/master/metron-platform/metron-common#global-configuration.

View solution in original post

Highlighted

Re: Metron ElasticSearch too many indexes

Contributor

Thank you so much @rmerriman

shame on me I was not aware of this global.json configuration file at all.

Highlighted

Re: Metron ElasticSearch too many indexes

Contributor

BTW, I realized the file global.json gets overwritten by metron/config on Ambari. There's a section called global.json template

Just in case someone might find it useful.

Don't have an account?