question Re: Disable log4j logging for HDFS audit log in Support Questions

Disable log4j logging for HDFS audit log

wbekker — Thu, 04 May 2017 19:45:38 GMT

Hi,

How can I disable the logging for the HDFS Audit log? My current config:

hdfs.audit.logger=INFO,console
log4j.logger.org.apache.hadoop.hdfs.server.namenode.FSNamesystem.audit=${hdfs.audit.logger}
log4j.additivity.org.apache.hadoop.hdfs.server.namenode.FSNamesystem.audit=false
log4j.appender.DRFAAUDIT=org.apache.log4j.DailyRollingFileAppender
log4j.appender.DRFAAUDIT.File=${hadoop.log.dir}/hdfs-audit.log
log4j.appender.DRFAAUDIT.layout=org.apache.log4j.PatternLayout
log4j.appender.DRFAAUDIT.layout.ConversionPattern=%d{ISO8601} %p %c{2}: %m%n
log4j.appender.DRFAAUDIT.DatePattern=.yyyy-MM-dd

Re: Disable log4j logging for HDFS audit log

jsensharma — Thu, 04 May 2017 20:01:04 GMT

@Ward Bekker

In the Ambari UI --> HDFS --> Configs --> Advanced --> "hadoop-env template" you will see the "SHARED_HADOOP_NAMENODE_OPTS" variable that is setting the

-Dhdfs.audit.logger=INFO,DRFAAUDIT

Above actually controls the audit logging. So you might want to delete that "hdfs.audit.logger" system property from the if-else block (same for HADOOP_DATANODE_OPTS option as well)

{% if java_version < 8 %}
SHARED_HADOOP_NAMENODE_OPTS= ........

{% else %}
SHARED_HADOOP_NAMENODE_OPTS=

{% endif %}

After that when we restart the HDFS components then in the "ps -ef | grep NameNode" we should not see the "-Dhdfs.audit.logger=INFO,DRFAAUDIT"

Re: Disable log4j logging for HDFS audit log

wbekker — Thu, 04 May 2017 20:06:44 GMT

excellent, thx!

Re: Disable log4j logging for HDFS audit log

ArpitAgarwal — Mon, 08 May 2017 21:56:44 GMT

@Ward Bekker we don't recommend disabling HDFS audit logging. It's hard to debug many HDFS issues without the audit log. Just curious, why would you like to disable it?

Re: Disable log4j logging for HDFS audit log

wbekker — Mon, 08 May 2017 21:59:45 GMT

@Arpit Agarwal good point. The customer uses ranger audit logging. What extra information is in the hdfs audit log, what is not already in the ranger audit logs.

Re: Disable log4j logging for HDFS audit log

ArpitAgarwal — Tue, 23 May 2017 03:39:50 GMT

Sorry I missed the notification of your reply. That is also a good question. I have not yet come across a customer setup where HDFS audit logging is disabled and Ranger audit logs is on.

I'd recommend tagging someone from Ranger to make sure.

Re: Disable log4j logging for HDFS audit log

jarnold — Fri, 18 May 2018 23:01:39 GMT

In my experience, if you remove the indicated flags, you still get audit logging - but those logs never get purged.

Perhaps it would be better to leave the flags, but to change "INFO" to "OFF", rendering something like:

-Dhdfs.audit.logger=OFF,DRFAAUDIT"