Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Disable log4j logging for HDFS audit log

Solved Go to solution
Highlighted

Disable log4j logging for HDFS audit log

Hi,

How can I disable the logging for the HDFS Audit log? My current config:

hdfs.audit.logger=INFO,console
log4j.logger.org.apache.hadoop.hdfs.server.namenode.FSNamesystem.audit=${hdfs.audit.logger}
log4j.additivity.org.apache.hadoop.hdfs.server.namenode.FSNamesystem.audit=false
log4j.appender.DRFAAUDIT=org.apache.log4j.DailyRollingFileAppender
log4j.appender.DRFAAUDIT.File=${hadoop.log.dir}/hdfs-audit.log
log4j.appender.DRFAAUDIT.layout=org.apache.log4j.PatternLayout
log4j.appender.DRFAAUDIT.layout.ConversionPattern=%d{ISO8601} %p %c{2}: %m%n
log4j.appender.DRFAAUDIT.DatePattern=.yyyy-MM-dd

1 ACCEPTED SOLUTION

Accepted Solutions
Highlighted

Re: Disable log4j logging for HDFS audit log

Super Mentor

@Ward Bekker

In the Ambari UI --> HDFS --> Configs --> Advanced --> "hadoop-env template" you will see the "SHARED_HADOOP_NAMENODE_OPTS" variable that is setting the

-Dhdfs.audit.logger=INFO,DRFAAUDIT

Above actually controls the audit logging. So you might want to delete that "hdfs.audit.logger" system property from the if-else block (same for HADOOP_DATANODE_OPTS option as well)

{% if java_version < 8 %}
SHARED_HADOOP_NAMENODE_OPTS= ........

{% else %}
SHARED_HADOOP_NAMENODE_OPTS=

{% endif %}

.

After that when we restart the HDFS components then in the "ps -ef | grep NameNode" we should not see the "-Dhdfs.audit.logger=INFO,DRFAAUDIT"

.

View solution in original post

6 REPLIES 6
Highlighted

Re: Disable log4j logging for HDFS audit log

Super Mentor

@Ward Bekker

In the Ambari UI --> HDFS --> Configs --> Advanced --> "hadoop-env template" you will see the "SHARED_HADOOP_NAMENODE_OPTS" variable that is setting the

-Dhdfs.audit.logger=INFO,DRFAAUDIT

Above actually controls the audit logging. So you might want to delete that "hdfs.audit.logger" system property from the if-else block (same for HADOOP_DATANODE_OPTS option as well)

{% if java_version < 8 %}
SHARED_HADOOP_NAMENODE_OPTS= ........

{% else %}
SHARED_HADOOP_NAMENODE_OPTS=

{% endif %}

.

After that when we restart the HDFS components then in the "ps -ef | grep NameNode" we should not see the "-Dhdfs.audit.logger=INFO,DRFAAUDIT"

.

View solution in original post

Highlighted

Re: Disable log4j logging for HDFS audit log

excellent, thx!

Highlighted

Re: Disable log4j logging for HDFS audit log

@Ward Bekker we don't recommend disabling HDFS audit logging. It's hard to debug many HDFS issues without the audit log. Just curious, why would you like to disable it?

Re: Disable log4j logging for HDFS audit log

@Arpit Agarwal good point. The customer uses ranger audit logging. What extra information is in the hdfs audit log, what is not already in the ranger audit logs.

Highlighted

Re: Disable log4j logging for HDFS audit log

Sorry I missed the notification of your reply. That is also a good question. I have not yet come across a customer setup where HDFS audit logging is disabled and Ranger audit logs is on.

I'd recommend tagging someone from Ranger to make sure.

Highlighted

Re: Disable log4j logging for HDFS audit log

Explorer

In my experience, if you remove the indicated flags, you still get audit logging - but those logs never get purged.

Perhaps it would be better to leave the flags, but to change "INFO" to "OFF", rendering something like:

-Dhdfs.audit.logger=OFF,DRFAAUDIT"

?

Don't have an account?
Coming from Hortonworks? Activate your account here