Support Questions
Find answers, ask questions, and share your expertise
Announcements
Check out our newest addition to the community, the Cloudera Innovation Accelerator group hub.

Can NiFi + Ranger plugin audit log to HDFS?

Explorer

For HDP it doesn't seem too much hassle getting services writing their ranger audit logs to HDFS. However, it is far less clear for NiFi. Out the box the only facility for access logs appears to be SOLR (which is not a viable option for me right now).

Is it possible to get NiFi writing Ranger audit logs to HDFS?

If not is it possible to configure NiFi (through log4j) to SYSLOG the ranger audit logs somewhere?

Thanks

1 REPLY 1

Super Collaborator

Hi @Oliver Fletcher,

let me answer your second question first for using the Log4j to audit syslog,

since the nifi logging done through the logback we need to make changes to the logback configuration (manage to get this work with the below configuration)

In Advanced-nifi-ranger-audit(Ambari-NiFi-Config) section make the flowing parameters values to,

xasecure.audit.destination.log4j=true
xasecure.audit.destination.log4j.logger=ranger.audit 

To capture the logs generated by the logger, configure the logger module(similar to nifi-app module logger).

In Advanced nifi-node-logback-env(Ambari-NiFi-Config) at add the following content logback.xml template

<appender name="RANGER_AUDIT" class="ch.qos.logback.core.rolling.RollingFileAppender">
        <file>${org.apache.nifi.bootstrap.config.log.dir}/ranger_nifi_audit.log</file>
        <rollingPolicy class="ch.qos.logback.core.rolling.SizeAndTimeBasedRollingPolicy">
            <fileNamePattern>${org.apache.nifi.bootstrap.config.log.dir}/ranger_nifi_audit_%d{yyyy-MM-dd_HH}.%i.log</fileNamePattern>
            <maxFileSize>100MB</maxFileSize>
            <maxHistory>30</maxHistory>
        </rollingPolicy>
        <immediateFlush>true</immediateFlush>
        <encoder class="ch.qos.logback.classic.encoder.PatternLayoutEncoder">
            <pattern>%date %level [%thread] %logger{40} %msg%n</pattern>
        </encoder>
</appender>
 
<logger name="ranger.audit" level="INFO" additivity="false">
            <appender-ref ref="RANGER_AUDIT"/>
</logger>

Sample output looks like this:

[centos@xxxxxxx nifi]$ cat ranger_nifi_audit.log
2017-09-08 03:37:47,475 INFO [org.apache.ranger.audit.queue.AuditBatchQueue1] ranger.audit {"repoType":10,"repo":"hdf_clstr_nifi","reqUser":"xxxxx","evtTime":"2017-09-08 03:37:46.699","access":"READ","resource":"/flow","resType":"nifi-resource","action":"READ","result":1,"policy":1,"enforcer":"ranger-acl","cliIP":"xxx.xxx.xxx.xxx","agentHost":"xxxx.xxxx.hortonworks.com","logType":"RangerAudit","id":"0efc4a0d-f634-42c0-9616-5d8298a92892-0","seq_num":1,"event_count":1,"event_dur_ms":0,"tags":[]}
2017-09-08 03:38:41,443 INFO [org.apache.ranger.audit.queue.AuditBatchQueue1] ranger.audit {"repoType":10,"repo":"clstr_nifi","reqUser":"admin","evtTime":"2017-09-08 03:38:39.121","access":"READ","resource":"/flow","resType":"nifi-resource","action":"READ","result":1,"policy":1,"enforcer":"ranger-acl","cliIP":"xxx.xxx.xxx.xxx","agentHost":"xxxx.xxxx.hortonworks.com","logType":"RangerAudit","id":"0efc4a0d-f634-42c0-9616-5d8298a92892-1","seq_num":3,"event_count":1,"event_dur_ms":0,"tags":[]}

on the other note you may write into Flat Files using the below configuration(I have tested for Kafka in HDF cluster not for NiFi, as the same Ranger it should work [advancde-ranger-kafk/nifi-audit Configuration]) :

xasecure.audit.is.enabled=true
xasecure.audit.destination.file=true
xasecure.audit.destination.file.dir=<path to log store logs>
xasecure.audit.destination.file.rolloer.secs=86400

the same can be write to HDFS using the configuration mentioned.