Support Questions

Find answers, ask questions, and share your expertise

Disable log4j logging for HDFS audit log

Hi,

How can I disable the logging for the HDFS Audit log? My current config:

hdfs.audit.logger=INFO,console
log4j.logger.org.apache.hadoop.hdfs.server.namenode.FSNamesystem.audit=${hdfs.audit.logger}
log4j.additivity.org.apache.hadoop.hdfs.server.namenode.FSNamesystem.audit=false
log4j.appender.DRFAAUDIT=org.apache.log4j.DailyRollingFileAppender
log4j.appender.DRFAAUDIT.File=${hadoop.log.dir}/hdfs-audit.log
log4j.appender.DRFAAUDIT.layout=org.apache.log4j.PatternLayout
log4j.appender.DRFAAUDIT.layout.ConversionPattern=%d{ISO8601} %p %c{2}: %m%n
log4j.appender.DRFAAUDIT.DatePattern=.yyyy-MM-dd

1 ACCEPTED SOLUTION

Super Mentor

@Ward Bekker

In the Ambari UI --> HDFS --> Configs --> Advanced --> "hadoop-env template" you will see the "SHARED_HADOOP_NAMENODE_OPTS" variable that is setting the

-Dhdfs.audit.logger=INFO,DRFAAUDIT

Above actually controls the audit logging. So you might want to delete that "hdfs.audit.logger" system property from the if-else block (same for HADOOP_DATANODE_OPTS option as well)

{% if java_version < 8 %}
SHARED_HADOOP_NAMENODE_OPTS= ........

{% else %}
SHARED_HADOOP_NAMENODE_OPTS=

{% endif %}

.

After that when we restart the HDFS components then in the "ps -ef | grep NameNode" we should not see the "-Dhdfs.audit.logger=INFO,DRFAAUDIT"

.

View solution in original post

6 REPLIES 6

Super Mentor

@Ward Bekker

In the Ambari UI --> HDFS --> Configs --> Advanced --> "hadoop-env template" you will see the "SHARED_HADOOP_NAMENODE_OPTS" variable that is setting the

-Dhdfs.audit.logger=INFO,DRFAAUDIT

Above actually controls the audit logging. So you might want to delete that "hdfs.audit.logger" system property from the if-else block (same for HADOOP_DATANODE_OPTS option as well)

{% if java_version < 8 %}
SHARED_HADOOP_NAMENODE_OPTS= ........

{% else %}
SHARED_HADOOP_NAMENODE_OPTS=

{% endif %}

.

After that when we restart the HDFS components then in the "ps -ef | grep NameNode" we should not see the "-Dhdfs.audit.logger=INFO,DRFAAUDIT"

.

excellent, thx!

@Ward Bekker we don't recommend disabling HDFS audit logging. It's hard to debug many HDFS issues without the audit log. Just curious, why would you like to disable it?

@Arpit Agarwal good point. The customer uses ranger audit logging. What extra information is in the hdfs audit log, what is not already in the ranger audit logs.

Sorry I missed the notification of your reply. That is also a good question. I have not yet come across a customer setup where HDFS audit logging is disabled and Ranger audit logs is on.

I'd recommend tagging someone from Ranger to make sure.

Explorer

In my experience, if you remove the indicated flags, you still get audit logging - but those logs never get purged.

Perhaps it would be better to leave the flags, but to change "INFO" to "OFF", rendering something like:

-Dhdfs.audit.logger=OFF,DRFAAUDIT"

?