How to enable audit logging without Navigator

[#1]

Hello, 

we have 5.11 cluster installed for testing, 2 master nodes, 4 slave nodes, and 1 management node.

now we want to enable the audit logging without using Navigator. 

I have some questions here

1. we have CM installed, can I use log4j.properties to enable the audit logging?

    I read some posts like this:

    https://community.cloudera.com/t5/Cloudera-Manager-Installation/What-is-the-Path-of-hdfs-site-xml-co...  it said that the actual configuration has non-standard location. So my understanding is no matter what I changed on the configruation location(e.g. /etc/hadoop/conf/.....), it won't work. And I should use the snippet to do the configuration.

2. and I read another posts here:

http://community.cloudera.com/t5/Cloudera-Manager-Installation/Audit-trail-for-HDFS-data-use/m-p/504... looks like I can use log4j.properties to do the audit logging.

I am a little bit confused, how can I enable audit logging without Nav?

Thanks in advance!

[#2 · Accepted solution]

Hi There,

Thanks for reaching out on the community. I'm Josh, and I'll help address this for you.

1. log4j.properties:

CM is the central point of configuration for services, so the short answer is that you should adjust log4j settings using safety valves. Below is an engineering blog post with a good description of how CM works.
http://blog.cloudera.com/blog/2013/07/how-does-cloudera-manager-work/

When a CM agent for a host heart beats to Cloudera Manager, Cloudera Manager sends back processes that should be running, and the related config files, one of which is the log4j.properties, for that service and role. From here, the CM agent makes a run time directory for these config files and references those. For instance, the agent will make a directory like the one bellow for a namenode role:

/var/run/cloudera-scm-agent/process/879-hdfs-NAMENODE/

this is why editing config files in on the OS has no effect, and is not recommended. 

1. Enabling audit logging:

To enable audit logging for a service without navigator, you would want to set the appropriate log4j settings in the appropriate safety valve for that service. Let's use HDFS as an example. Cloudera Manager has a configuration property for HDFS labeled "NameNode Logging Advanced Configuration Snippet (Safety Valve)". This is the one you want to put your log4j settings in. Once you've put your settings in, it will insert those into the log4j.properties it sends over to the agent in heartbeats. The specifics for enabling vanilla hadoop HDFS audit logging can be found bellow:

http://apprize.info/security/hadoop/7.html

Considering all of this info, bear in mind that Navigator takes care of all of this for you, as well as adding additional features. For instance, HDFS audit logs can be very bulky and cumbersome by themselves and include many operations that aren't very helpful from an auditing standpoint. Navigator is able to apply event filters to an audit log, store relevant audits, and index them for further searching. Therefore, I highly recommend enabling navigator when doing so becomes feasible.

Please let me know if you have any other questions.

Cheers

[#3]

Thank you so much Josh.It's really helpful!

[#4]

@jmartin1938 

You'e recommended using Navigator as it applies event filters and stores relevant audits.

But, we've noticed it capturing operations which aren't executed by the users but are internally triggered due to some other operation.

For example, a single ListStatus (ls) operation results in two records listStatus and getfileinfo

Also, some of the operations captured are not really useful for auditing purposes. (Example : getEZForPath)

Is there a way to customize the event filters? If yes, could you help us with the relevant documentation?

[#5]

Hi @uv ,

You may want to check this public doc about Navigator Audit Filter:

https://docs.cloudera.com/documentation/enterprise/latest/topics/cn_admcfg_audit_filters.html

Thanks,

Li