@ururu I have a similar use case but the configurations mentioned above are not giving the desired results. Did you add or modify some of the configurations?   E.g., I saw it being mentioned elsewhere that the hadoop.root.logger value needs to be changed as well to include SYSLOG as a value. Did you do this too? Also, in case the external SIEM server expects a particular format, e.g., the RFC 5424 syslog format or a specific SIEM server format like Universal LEEF, what would be the best way to define this property? ... View more

Hello @smdas,   Thanks for the follow up. I did follow the recommended link and implemented most of the suggestions there on my Ambari Infra Solr setup. While that helped with Solr indexing, it did not resolve the issue at hand. On a closer look, I identified that the lagging only affected hadoop-acl enforcer type actions. (see attached picture - which shows no hadoop-acl type actions even though a lot has been performed over the day) I went through the Namenode logs and made some adjustments to the log4j configurations for hdfs audit logging. Unfortunately, this has still not resolved the problem. The actions eventually show but sometimes, after a whole day or even two days. (see attached picture - where the latest hadoop-acl type actions are from last Friday)    It seems as if hadoop-acl type actions are being queued or buffered somehow and only indexed to Solr after a limit has been reached. However, I haven't found any configuration setting which would mitigate this if that is the case.   All ideas are really welcome. Thanks ... View more

Thanks a lot for the pointer @smdas. I have already seen a number of warnings in the Infra Solr logs but I am not sure if they are directly related to the issue at hand since they are mostly start-up warnings  - see below  2021-02-02 17:18:03,938 [main] WARN [ ] org.eclipse.jetty.security.ConstraintSecurityHandler (ConstraintSecurityHandler.java:807) - ServletContext@o.e.j.w.WebAppContext@30b8a058{/solr,file:/usr/lib/ambari-infra-solr/server/solr-webapp/webapp/,STARTING}{/usr/lib/ambari-infra-solr/server/solr-webapp/webapp} has uncovered http methods for path: / 2021-02-02 17:18:04,402 [main] WARN [ ] org.apache.solr.core.CoreContainer (CoreContainer.java:401) - Couldn't add files from /xxxx/xx/ambari_infra_solr/data/lib to classpath: /xxxx/xx/ambari_infra_solr/data/lib 2021-02-02 17:18:06,749 [coreLoadExecutor-6-thread-2-processing-n:XXXXXXXXXX-XXX-XXX-XXX:8886_solr] WARN [c:ranger_audits s:shard1 r:core_node12 x:ranger_audits_shard1_replica1] org.apache.solr.core.Config (Config.java:169) - Beginning with Solr 5.5, <mergeFactor> is deprecated, configure it on the relevant <mergePolicyFactory> instead. 2021-02-02 17:18:06,755 [coreLoadExecutor-6-thread-5-processing-n:XXXXXXXXXX-XXX-XXX-XXX:8886_solr] WARN [c:ranger_audits s:shard4 r:core_node14 x:ranger_audits_shard4_replica1] org.apache.solr.core.Config (Config.java:169) - Beginning with Solr 5.5, <mergeFactor> is deprecated, configure it on the relevant <mergePolicyFactory> instead. 2021-02-02 17:18:06,758 [coreLoadExecutor-6-thread-8-processing-n:XXXXXXXXXX-XXX-XXX-XXX:8886_solr] WARN [c:ranger_audits s:shard8 r:core_node9 x:ranger_audits_shard8_replica2] org.apache.solr.core.Config (Config.java:169) - Beginning with Solr 5.5, <mergeFactor> is deprecated, configure it on the relevant <mergePolicyFactory> instead. 2021-02-02 17:18:06,758 [coreLoadExecutor-6-thread-7-processing-n:XXXXXXXXXX-XXX-XXX-XXX:8886_solr] WARN [c:ranger_audits s:shard7 r:core_node7 x:ranger_audits_shard7_replica1] org.apache.solr.core.Config (Config.java:169) - Beginning with Solr 5.5, <mergeFactor> is deprecated, configure it on the relevant <mergePolicyFactory> instead. 2021-02-02 17:18:06,759 [coreLoadExecutor-6-thread-6-processing-n:XXXXXXXXXX-XXX-XXX-XXX:8886_solr] WARN [c:ranger_audits s:shard5 r:core_node6 x:ranger_audits_shard5_replica2] org.apache.solr.core.Config (Config.java:169) - Beginning with Solr 5.5, <mergeFactor> is deprecated, configure it on the relevant <mergePolicyFactory> instead. 2021-02-02 17:18:06,759 [coreLoadExecutor-6-thread-3-processing-n:XXXXXXXXXX-XXX-XXX-XXX:8886_solr] WARN [c:ranger_audits s:shard10 r:core_node5 x:ranger_audits_shard10_replica1] org.apache.solr.core.Config (Config.java:169) - Beginning with Solr 5.5, <mergeFactor> is deprecated, configure it on the relevant <mergePolicyFactory> instead. 2021-02-02 17:18:06,759 [coreLoadExecutor-6-thread-4-processing-n:XXXXXXXXXX-XXX-XXX-XXX:8886_solr] WARN [c:ranger_audits s:shard2 r:core_node2 x:ranger_audits_shard2_replica2] org.apache.solr.core.Config (Config.java:169) - Beginning with Solr 5.5, <mergeFactor> is deprecated, configure it on the relevant <mergePolicyFactory> instead. 2021-02-02 17:18:07,617 [coreLoadExecutor-6-thread-4-processing-n:XXXXXXXXXX-XXX-XXX-XXX:8886_solr] WARN [c:ranger_audits s:shard2 r:core_node2 x:ranger_audits_shard2_replica2] org.apache.solr.core.SolrResourceLoader (SolrResourceLoader.java:574) - Solr loaded a deprecated plugin/analysis class [solr.admin.AdminHandlers]. Please consult documentation how to replace it accordingly. 2021-02-02 17:18:07,624 [coreLoadExecutor-6-thread-3-processing-n:XXXXXXXXXX-XXX-XXX-XXX:8886_solr] WARN [c:ranger_audits s:shard10 r:core_node5 x:ranger_audits_shard10_replica1] org.apache.solr.core.SolrResourceLoader (SolrResourceLoader.java:574) - Solr loaded a deprecated plugin/analysis class [solr.admin.AdminHandlers]. Please consult documentation how to replace it accordingly. 2021-02-02 17:18:08,112 [coreLoadExecutor-6-thread-6-processing-n:XXXXXXXXXX-XXX-XXX-XXX:8886_solr] WARN [c:ranger_audits s:shard5 r:core_node6 x:ranger_audits_shard5_replica2] org.apache.solr.handler.admin.AdminHandlers (AdminHandlers.java:103) - <requestHandler name="/admin/" class="solr.admin.AdminHandlers" /> is deprecated . It is not required anymore 2021-02-02 17:18:08,113 [coreLoadExecutor-6-thread-5-processing-n:XXXXXXXXXX-XXX-XXX-XXX:8886_solr] WARN [c:ranger_audits s:shard4 r:core_node14 x:ranger_audits_shard4_replica1] org.apache.solr.handler.admin.AdminHandlers (AdminHandlers.java:103) - <requestHandler name="/admin/" class="solr.admin.AdminHandlers" /> is deprecated . It is not required anymore 2021-02-02 17:18:08,113 [coreLoadExecutor-6-thread-4-processing-n:XXXXXXXXXX-XXX-XXX-XXX:8886_solr] WARN [c:ranger_audits s:shard2 r:core_node2 x:ranger_audits_shard2_replica2] org.apache.solr.handler.admin.AdminHandlers (AdminHandlers.java:103) - <requestHandler name="/admin/"   I will, nonetheless, check out the tuning parameters in the link. Thanks again. ... View more

Hello @Madhur Thanks a lot for the reply. I can confirm that the operating system is rhel7. The base url used was a configuration setting passed down but we have used it for other clusters without issues. I will nonetheless check with the client to make sure it is correct.   Concerning the link to the bug report, the upgrade was done for Ambari 2.6.2.2 while the mentioned bug was fixed in version 2.6.0.0. Also, the scenarios presented in the bug are a bit different in our case.   Thanks a lot for the help ... View more

Hello @StevenOD,   Thanks for the reply. We have considered this but at the moment, the client requirements limits our usage to 5.16.x ... View more