Ranger Audit Lagging

[#1]

Hello Everyone,

I am having an issue where some times the ranger audit page in the Ranger UI admin delays in showing audit entries for recent deployments. Sometimes, it does not show for close to 3 hours. Other times, it shows immediately. - see attached picture. 

The recent information also does not show even on refreshing the update time. Is there any configuration setting to regulate how Ranger UI updates the Audit page?

Thanks

[#2]

Hello @CaptainJa 

Thanks for using Cloudera Community. Based on the post, the Ranger UI is showing audits slowly sometime, while being fast during other period. 

The Ranger Audit is powered by Solr (Infra-Solr in HDP) & likely, the delay in Ranger Audit UI Slowness is being caused by the Slowed Indexing of Audit Documents by Solr. Worth reviewing Link [1], especially "Tuning Environment Specific Parameters" Section, which deals with RangerAudit Collection Shard Count based on the Document being indexed per day. Additionally, Review the Infra-Solr Logs for any warning around data ingestion. 

- Smarak

[1] https://docs.cloudera.com/HDPDocuments/Ambari-2.7.5.0/using-ambari-core-services/content/amb_tuning_...

[#3]

Thanks a lot for the pointer @smdas. I have already seen a number of warnings in the Infra Solr logs but I am not sure if they are directly related to the issue at hand since they are mostly start-up warnings  - see below 

2021-02-02 17:18:03,938 [main] WARN  [   ] org.eclipse.jetty.security.ConstraintSecurityHandler (ConstraintSecurityHandler.java:807) - ServletContext@o.e.j.w.WebAppContext@30b8a058{/solr,file:/usr/lib/ambari-infra-solr/server/solr-webapp/webapp/,STARTING}{/usr/lib/ambari-infra-solr/server/solr-webapp/webapp} has uncovered http methods for path: /
2021-02-02 17:18:04,402 [main] WARN  [   ] org.apache.solr.core.CoreContainer (CoreContainer.java:401) - Couldn't add files from /xxxx/xx/ambari_infra_solr/data/lib to classpath: /xxxx/xx/ambari_infra_solr/data/lib
2021-02-02 17:18:06,749 [coreLoadExecutor-6-thread-2-processing-n:XXXXXXXXXX-XXX-XXX-XXX:8886_solr] WARN  [c:ranger_audits s:shard1 r:core_node12 x:ranger_audits_shard1_replica1] org.apache.solr.core.Config (Config.java:169) - Beginning with Solr 5.5, <mergeFactor> is deprecated, configure it on the relevant <mergePolicyFactory> instead.
2021-02-02 17:18:06,755 [coreLoadExecutor-6-thread-5-processing-n:XXXXXXXXXX-XXX-XXX-XXX:8886_solr] WARN  [c:ranger_audits s:shard4 r:core_node14 x:ranger_audits_shard4_replica1] org.apache.solr.core.Config (Config.java:169) - Beginning with Solr 5.5, <mergeFactor> is deprecated, configure it on the relevant <mergePolicyFactory> instead.
2021-02-02 17:18:06,758 [coreLoadExecutor-6-thread-8-processing-n:XXXXXXXXXX-XXX-XXX-XXX:8886_solr] WARN  [c:ranger_audits s:shard8 r:core_node9 x:ranger_audits_shard8_replica2] org.apache.solr.core.Config (Config.java:169) - Beginning with Solr 5.5, <mergeFactor> is deprecated, configure it on the relevant <mergePolicyFactory> instead.
2021-02-02 17:18:06,758 [coreLoadExecutor-6-thread-7-processing-n:XXXXXXXXXX-XXX-XXX-XXX:8886_solr] WARN  [c:ranger_audits s:shard7 r:core_node7 x:ranger_audits_shard7_replica1] org.apache.solr.core.Config (Config.java:169) - Beginning with Solr 5.5, <mergeFactor> is deprecated, configure it on the relevant <mergePolicyFactory> instead.
2021-02-02 17:18:06,759 [coreLoadExecutor-6-thread-6-processing-n:XXXXXXXXXX-XXX-XXX-XXX:8886_solr] WARN  [c:ranger_audits s:shard5 r:core_node6 x:ranger_audits_shard5_replica2] org.apache.solr.core.Config (Config.java:169) - Beginning with Solr 5.5, <mergeFactor> is deprecated, configure it on the relevant <mergePolicyFactory> instead.
2021-02-02 17:18:06,759 [coreLoadExecutor-6-thread-3-processing-n:XXXXXXXXXX-XXX-XXX-XXX:8886_solr] WARN  [c:ranger_audits s:shard10 r:core_node5 x:ranger_audits_shard10_replica1] org.apache.solr.core.Config (Config.java:169) - Beginning with Solr 5.5, <mergeFactor> is deprecated, configure it on the relevant <mergePolicyFactory> instead.
2021-02-02 17:18:06,759 [coreLoadExecutor-6-thread-4-processing-n:XXXXXXXXXX-XXX-XXX-XXX:8886_solr] WARN  [c:ranger_audits s:shard2 r:core_node2 x:ranger_audits_shard2_replica2] org.apache.solr.core.Config (Config.java:169) - Beginning with Solr 5.5, <mergeFactor> is deprecated, configure it on the relevant <mergePolicyFactory> instead.
2021-02-02 17:18:07,617 [coreLoadExecutor-6-thread-4-processing-n:XXXXXXXXXX-XXX-XXX-XXX:8886_solr] WARN  [c:ranger_audits s:shard2 r:core_node2 x:ranger_audits_shard2_replica2] org.apache.solr.core.SolrResourceLoader (SolrResourceLoader.java:574) - Solr loaded a deprecated plugin/analysis class [solr.admin.AdminHandlers]. Please consult documentation how to replace it accordingly.
2021-02-02 17:18:07,624 [coreLoadExecutor-6-thread-3-processing-n:XXXXXXXXXX-XXX-XXX-XXX:8886_solr] WARN  [c:ranger_audits s:shard10 r:core_node5 x:ranger_audits_shard10_replica1] org.apache.solr.core.SolrResourceLoader (SolrResourceLoader.java:574) - Solr loaded a deprecated plugin/analysis class [solr.admin.AdminHandlers]. Please consult documentation how to replace it accordingly.
2021-02-02 17:18:08,112 [coreLoadExecutor-6-thread-6-processing-n:XXXXXXXXXX-XXX-XXX-XXX:8886_solr] WARN  [c:ranger_audits s:shard5 r:core_node6 x:ranger_audits_shard5_replica2] org.apache.solr.handler.admin.AdminHandlers (AdminHandlers.java:103) - <requestHandler name="/admin/"
 class="solr.admin.AdminHandlers" /> is deprecated . It is not required anymore
2021-02-02 17:18:08,113 [coreLoadExecutor-6-thread-5-processing-n:XXXXXXXXXX-XXX-XXX-XXX:8886_solr] WARN  [c:ranger_audits s:shard4 r:core_node14 x:ranger_audits_shard4_replica1] org.apache.solr.handler.admin.AdminHandlers (AdminHandlers.java:103) - <requestHandler name="/admin/"
 class="solr.admin.AdminHandlers" /> is deprecated . It is not required anymore
2021-02-02 17:18:08,113 [coreLoadExecutor-6-thread-4-processing-n:XXXXXXXXXX-XXX-XXX-XXX:8886_solr] WARN  [c:ranger_audits s:shard2 r:core_node2 x:ranger_audits_shard2_replica2] org.apache.solr.handler.admin.AdminHandlers (AdminHandlers.java:103) - <requestHandler name="/admin/"

I will, nonetheless, check out the tuning parameters in the link.

Thanks again.

[#4]

Hello @CaptainJa 

Wish to follow-up with you on the concerned Topic & see how things are for your team. If there are no further issues, Kindly mark the Post as Solved. If there are queries, Please share them.

- Smarak

[#5]

Hello @smdas,

Thanks for the follow up. I did follow the recommended link and implemented most of the suggestions there on my Ambari Infra Solr setup. While that helped with Solr indexing, it did not resolve the issue at hand. On a closer look, I identified that the lagging only affected hadoop-acl enforcer type actions. (see attached picture - which shows no hadoop-acl type actions even though a lot has been performed over the day)

I went through the Namenode logs and made some adjustments to the log4j configurations for hdfs audit logging. Unfortunately, this has still not resolved the problem. The actions eventually show but sometimes, after a whole day or even two days. (see attached picture - where the latest hadoop-acl type actions are from last Friday) 

It seems as if hadoop-acl type actions are being queued or buffered somehow and only indexed to Solr after a limit has been reached. However, I haven't found any configuration setting which would mitigate this if that is the case.

All ideas are really welcome.

Thanks

[#6]

Hello @CaptainJa 

Thanks for your Update. Based on your review, the "hadoop-acl" enforcer is being delayed to be tracked via Ranger Audit UI while other Audits are likely appearing immediately. As far as I know, the Audit Framework from any Service to Solr is same, likely indicating the suspicions raised by you i.e. the "hadoop-acl" events are being buffered prior to being sent to Solr for Indexing.

Currently, I am unfamiliar with any Configuration controlling the same yet wish to confirm if the HDFS Audit Logs or InfraSolr Logs are reporting any issues, which may point to any concerns. I was under the impression that Solr may be the Bottleneck for RangerAudit Lagging yet the synopsis appears to be impacting the "hadoop-acl" alone.

- Smarak