Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

LogSearch history collection multiple shards and replication factors

LogSearch history collection multiple shards and replication factors

New Contributor

I am experimenting with LogSearch and I have noticed that the history collection in ambari-infra has only one shard and on replication. I tried to change that but everytime I restart logsearch, the history collection is reset to the default. Is there a place I can make my change permanent? like having more shards and replication factors?

5 REPLIES 5

Re: LogSearch history collection multiple shards and replication factors

New Contributor

Hi!

In tech preview, event history is not yet supported, its only contains some log filter data (for 3.0 that will be moved to zookeeper).

History shard and replication number can be set in /etc/ambari-logsearch-portal/conf/logsearch.properties file.

For now it set in a python file (that fills logsearch.properties https://github.com/apache/ambari/blob/branch-2.5/ambari-server/src/main/resources/common-services/LO... which will generate the logsearch.properties file.

If you are using ambari-2.5 you can add custom properties for logsearch-properties config type and override "logsearch.collection.history.replication.factor" and "logsearch.collection.history.numshards" as well.

In case you are using ambari-2.4, it will be a bit harder because that is hard coded in a jinja template file (

https://github.com/apache/ambari/blob/branch-2.4/ambari-server/src/main/resources/common-services/LO..., so you will need to update that file, which is a bit hacky, maybe you can also try out that solution what i suggested for 2.5

Other than that, I think you can update shard replication numbers through solr api as well, after logsearch is started

Re: LogSearch history collection multiple shards and replication factors

New Contributor

Thank you @oszabo for the detailed answer. I am using Ambari 2.4.2 and I did try your suggestions for the jinja template but it does not seem to be working. Would you happen to know the location in the logsearch code where the collections are created? this way I can back track the config to the right config parameters.

Highlighted

Re: LogSearch history collection multiple shards and replication factors

New Contributor

Hi @Theyaa Matti

In 2.4, that will be a bit harder to change, as i see that was hard coded in the java code as well:

https://github.com/apache/ambari/blob/release-2.4.2/ambari-logsearch/ambari-logsearch-portal/src/mai...

Then, I suggest to create the collection manually before starting logsearch application.

Re: LogSearch history collection multiple shards and replication factors

New Contributor

Thank you @oszabo

When are you releasing the next stable version for logsearch and infra?

Re: LogSearch history collection multiple shards and replication factors

New Contributor

hi @Theyaa Matti,

https://ambari.apache.org/whats-new.html

Ambari 2.5.0 is out, Log Search and Infra versions is managed with Ambari (so i recommend to use 2.5.0 for Log Search and Infra, but only if you are using Ambari 2.5.0 as well, because Log Search rest API is used from Ambari as well)

Don't have an account?
Coming from Hortonworks? Activate your account here