Disable log4j logging for HDFS audit log

[#1]

Hi,

How can I disable the logging for the HDFS Audit log? My current config:

hdfs.audit.logger=INFO,console
log4j.logger.org.apache.hadoop.hdfs.server.namenode.FSNamesystem.audit=${hdfs.audit.logger}
log4j.additivity.org.apache.hadoop.hdfs.server.namenode.FSNamesystem.audit=false
log4j.appender.DRFAAUDIT=org.apache.log4j.DailyRollingFileAppender
log4j.appender.DRFAAUDIT.File=${hadoop.log.dir}/hdfs-audit.log
log4j.appender.DRFAAUDIT.layout=org.apache.log4j.PatternLayout
log4j.appender.DRFAAUDIT.layout.ConversionPattern=%d{ISO8601} %p %c{2}: %m%n
log4j.appender.DRFAAUDIT.DatePattern=.yyyy-MM-dd

[#2 · Accepted solution]

@Ward Bekker

In the Ambari UI --> HDFS --> Configs --> Advanced --> "hadoop-env template" you will see the "SHARED_HADOOP_NAMENODE_OPTS" variable that is setting the

-Dhdfs.audit.logger=INFO,DRFAAUDIT

Above actually controls the audit logging. So you might want to delete that "hdfs.audit.logger" system property from the if-else block (same for HADOOP_DATANODE_OPTS option as well)

{% if java_version < 8 %}
SHARED_HADOOP_NAMENODE_OPTS= ........

{% else %}
SHARED_HADOOP_NAMENODE_OPTS=

{% endif %}

.

After that when we restart the HDFS components then in the "ps -ef | grep NameNode" we should not see the "-Dhdfs.audit.logger=INFO,DRFAAUDIT"

.

[#3]

excellent, thx!

[#4]

@Ward Bekker we don't recommend disabling HDFS audit logging. It's hard to debug many HDFS issues without the audit log. Just curious, why would you like to disable it?

[#5]

@Arpit Agarwal good point. The customer uses ranger audit logging. What extra information is in the hdfs audit log, what is not already in the ranger audit logs.

[#6]

Sorry I missed the notification of your reply. That is also a good question. I have not yet come across a customer setup where HDFS audit logging is disabled and Ranger audit logs is on.

I'd recommend tagging someone from Ranger to make sure.

[#7]

In my experience, if you remove the indicated flags, you still get audit logging - but those logs never get purged.

Perhaps it would be better to leave the flags, but to change "INFO" to "OFF", rendering something like:

-Dhdfs.audit.logger=OFF,DRFAAUDIT"

?