03-24-2017 12:03 AM
We are moving the web logs to solr with the flow flume-->kafka-->storm-->solr. We found that storm to solr indexing taking more time. Because of this, delay of moving web logs to solr is approx 1hour. we are using solrj client at the storm to indexing at solr.
On Average of 10minutes 25k logs events are recieved by storm.
For performace improvement,We change the soft autocommit and hard autocommit value from the default of 1sec to 30 sec for soft commit and 1minute to 3 minute for hard commit. but we are not found any performance improvment.
While we reading the articles we found that by increasing below fields at the solrconfig.xml helpful for performance improvement.
1.maxIndexingThread (default 8)
2. rmbuffersizemb (default 128)
Can anyone tell me by changing the above fields will it work in this case?
What are all the general idea for this case to avoid log delay at solr.