- Subscribe to RSS Feed
- Mark Question as New
- Mark Question as Read
- Float this Question for Current User
- Bookmark
- Subscribe
- Mute
- Printer Friendly Page
Disable log4j logging for HDFS audit log
- Labels:
-
Apache Hadoop
Created ‎05-04-2017 12:45 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi,
How can I disable the logging for the HDFS Audit log? My current config:
hdfs.audit.logger=INFO,console log4j.logger.org.apache.hadoop.hdfs.server.namenode.FSNamesystem.audit=${hdfs.audit.logger} log4j.additivity.org.apache.hadoop.hdfs.server.namenode.FSNamesystem.audit=false log4j.appender.DRFAAUDIT=org.apache.log4j.DailyRollingFileAppender log4j.appender.DRFAAUDIT.File=${hadoop.log.dir}/hdfs-audit.log log4j.appender.DRFAAUDIT.layout=org.apache.log4j.PatternLayout log4j.appender.DRFAAUDIT.layout.ConversionPattern=%d{ISO8601} %p %c{2}: %m%n log4j.appender.DRFAAUDIT.DatePattern=.yyyy-MM-dd
Created ‎05-04-2017 01:01 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
In the Ambari UI --> HDFS --> Configs --> Advanced --> "hadoop-env template" you will see the "SHARED_HADOOP_NAMENODE_OPTS" variable that is setting the
-Dhdfs.audit.logger=INFO,DRFAAUDIT
Above actually controls the audit logging. So you might want to delete that "hdfs.audit.logger" system property from the if-else block (same for HADOOP_DATANODE_OPTS option as well)
{% if java_version < 8 %} SHARED_HADOOP_NAMENODE_OPTS= ........ {% else %} SHARED_HADOOP_NAMENODE_OPTS= {% endif %}
.
After that when we restart the HDFS components then in the "ps -ef | grep NameNode" we should not see the "-Dhdfs.audit.logger=INFO,DRFAAUDIT"
.
Created ‎05-04-2017 01:01 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
In the Ambari UI --> HDFS --> Configs --> Advanced --> "hadoop-env template" you will see the "SHARED_HADOOP_NAMENODE_OPTS" variable that is setting the
-Dhdfs.audit.logger=INFO,DRFAAUDIT
Above actually controls the audit logging. So you might want to delete that "hdfs.audit.logger" system property from the if-else block (same for HADOOP_DATANODE_OPTS option as well)
{% if java_version < 8 %} SHARED_HADOOP_NAMENODE_OPTS= ........ {% else %} SHARED_HADOOP_NAMENODE_OPTS= {% endif %}
.
After that when we restart the HDFS components then in the "ps -ef | grep NameNode" we should not see the "-Dhdfs.audit.logger=INFO,DRFAAUDIT"
.
Created ‎05-04-2017 01:06 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
excellent, thx!
Created ‎05-08-2017 02:56 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
@Ward Bekker we don't recommend disabling HDFS audit logging. It's hard to debug many HDFS issues without the audit log. Just curious, why would you like to disable it?
Created ‎05-08-2017 02:59 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
@Arpit Agarwal good point. The customer uses ranger audit logging. What extra information is in the hdfs audit log, what is not already in the ranger audit logs.
Created ‎05-22-2017 08:39 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Sorry I missed the notification of your reply. That is also a good question. I have not yet come across a customer setup where HDFS audit logging is disabled and Ranger audit logs is on.
I'd recommend tagging someone from Ranger to make sure.
Created ‎05-18-2018 04:01 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
In my experience, if you remove the indicated flags, you still get audit logging - but those logs never get purged.
Perhaps it would be better to leave the flags, but to change "INFO" to "OFF", rendering something like:
-Dhdfs.audit.logger=OFF,DRFAAUDIT"
?
