Ranger audit log for Stream Messaging Manager service (SMM)

[#1]

Hello All,

On CDP PB, like we send ranger audit logs of services thats support Ranger plugin to HDFS, can we also send logs of Stream Messaging Manager service to HDFS? 

I tried to look for config to achieve this, however, I could find only 2 configs (listed below) that allow us to configure the local path. However, there was nothing found to send these logs to HDFS.

Ranger Streams Messaging Manager Plugin Audit Hdfs Spool Directory Path

Ranger Streams Messaging Manager Plugin Audit Solr Spool Directory Path

Any help / guidance will be highly appreciated.

Thanks
snm1523

[#2]

Hi @snm1523 those configs are for spool directories, these configs won't help you.

The audit logs are stored in HDFS, It's Solr collections that will store them in HDFS, if your solr is configured to store them in HDFS then by default all the auditing will happen in HDFS. You can refer to the below doc to check where and how solr collections are stored.

https://docs.cloudera.com/cdp-private-cloud-base/7.1.8/security-ranger-auditing/topics/security-rang...

[#3]

Hello @Rajat_710.,

Thank you for the response.

Will definitely go through the article. However, have got a follow up question.

I also noticed on master servers (local location configured using relevant properties), that the audit logs (HDFS and SOLR) are not getting purged automatically once they are sent to HDFS. Instead we have to run a cron job that triggers a shell script which clears the logs the from the location. If the job is stopped even for 15 mins, we will see a pile of logs at that location.

Any suggestions if we have missed anything?

Thanks

snm1523

[#4]

Hi @snm1523 , correct you have to manually delete those files in HDFS, we are aware of it and currently we are working on that issue.

[#5]

Thank you for the confirmation, @Rajat_710.

Is there a Cloudera article / blog post / documentation informing this as a known issue or WIP that I can use as reference in our internal discussions. Will need something to propose and support manual deletion of HDFS and SOLR audit logs from local location.

Thanks
snm1523

[#6 · Accepted solution]

Hi, @snm1523 you can find some info here,

https://community.cloudera.com/t5/Support-Questions/Ranger-audit-log-to-hdfs/m-p/323456/highlight/tr...

[#7]

Thank you for the reference, @Rajat_710 . 

That link is referencing to a community post where someone has discussed about logs in HDFS.

However, here I am more keen towards logs that are stored in local location and not getting deleted even after they are moved to HDFS. Since, you mentioned that this has to be manually deleted and Cloudera is currently working on implementing automatic purge of these logs from local, would it be possible to kindly share a reference article or a Cloudera blog post or a known issue reference where this has been mentioned / recorded.

Thanks
snm1523

[#8]

Hi @snm1523 right now i dont any doc for this.

[#9]

Hello @Rajat_710 ,

This is a last follow up query on this thread.

Any clue how we configure Streams Messaging Manager server to send audit logs to HDFS and / or Solr (just like other services) and then they would get archived to /archive directory from where we will manually delete them?

I am referring to logs that get stored under below location locally (not HDFS)
/var/log/<service name>/audit/<HDFS or Solr>/spool/

Thanks
snm1523

[#10]

@snm1523 apologies for the late response, logs stored under "/var/log/<service name>/audit/<HDFS or Solr>/spool/" location means something is wrong with the plugin service.

By default, audit logs will be stored in HDFS but when the end service is not responding properly then only the spool files will be created under here. It will store on local only when SMM is not responding