Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Disable log4j logging for HDFS audit log

avatar

Hi,

How can I disable the logging for the HDFS Audit log? My current config:

hdfs.audit.logger=INFO,console
log4j.logger.org.apache.hadoop.hdfs.server.namenode.FSNamesystem.audit=${hdfs.audit.logger}
log4j.additivity.org.apache.hadoop.hdfs.server.namenode.FSNamesystem.audit=false
log4j.appender.DRFAAUDIT=org.apache.log4j.DailyRollingFileAppender
log4j.appender.DRFAAUDIT.File=${hadoop.log.dir}/hdfs-audit.log
log4j.appender.DRFAAUDIT.layout=org.apache.log4j.PatternLayout
log4j.appender.DRFAAUDIT.layout.ConversionPattern=%d{ISO8601} %p %c{2}: %m%n
log4j.appender.DRFAAUDIT.DatePattern=.yyyy-MM-dd

1 ACCEPTED SOLUTION

avatar
Master Mentor

@Ward Bekker

In the Ambari UI --> HDFS --> Configs --> Advanced --> "hadoop-env template" you will see the "SHARED_HADOOP_NAMENODE_OPTS" variable that is setting the

-Dhdfs.audit.logger=INFO,DRFAAUDIT

Above actually controls the audit logging. So you might want to delete that "hdfs.audit.logger" system property from the if-else block (same for HADOOP_DATANODE_OPTS option as well)

{% if java_version < 8 %}
SHARED_HADOOP_NAMENODE_OPTS= ........

{% else %}
SHARED_HADOOP_NAMENODE_OPTS=

{% endif %}

.

After that when we restart the HDFS components then in the "ps -ef | grep NameNode" we should not see the "-Dhdfs.audit.logger=INFO,DRFAAUDIT"

.

View solution in original post

6 REPLIES 6

avatar
Master Mentor

@Ward Bekker

In the Ambari UI --> HDFS --> Configs --> Advanced --> "hadoop-env template" you will see the "SHARED_HADOOP_NAMENODE_OPTS" variable that is setting the

-Dhdfs.audit.logger=INFO,DRFAAUDIT

Above actually controls the audit logging. So you might want to delete that "hdfs.audit.logger" system property from the if-else block (same for HADOOP_DATANODE_OPTS option as well)

{% if java_version < 8 %}
SHARED_HADOOP_NAMENODE_OPTS= ........

{% else %}
SHARED_HADOOP_NAMENODE_OPTS=

{% endif %}

.

After that when we restart the HDFS components then in the "ps -ef | grep NameNode" we should not see the "-Dhdfs.audit.logger=INFO,DRFAAUDIT"

.

avatar

excellent, thx!

avatar

@Ward Bekker we don't recommend disabling HDFS audit logging. It's hard to debug many HDFS issues without the audit log. Just curious, why would you like to disable it?

avatar

@Arpit Agarwal good point. The customer uses ranger audit logging. What extra information is in the hdfs audit log, what is not already in the ranger audit logs.

avatar

Sorry I missed the notification of your reply. That is also a good question. I have not yet come across a customer setup where HDFS audit logging is disabled and Ranger audit logs is on.

I'd recommend tagging someone from Ranger to make sure.

avatar
Contributor

In my experience, if you remove the indicated flags, you still get audit logging - but those logs never get purged.

Perhaps it would be better to leave the flags, but to change "INFO" to "OFF", rendering something like:

-Dhdfs.audit.logger=OFF,DRFAAUDIT"

?