question How can I filter the NiFi log to exclude a certain error message? in Archives of Support Questions (Read Only)

How can I filter the NiFi log to exclude a certain error message?

josh_nicholson — Tue, 21 Apr 2026 12:17:51 GMT

I am constantly seeing this error message in the nifi-app.log file.

2018-04-16 11:00:36,383 ERROR [nifi.async.batch_nifi.async.batch.hdfs_destWriter] o.a.r.audit.provider.BaseAuditHandler Error writing to log file.
java.io.IOException: No FileSystem for scheme: hdfs
        at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2660)
        at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2667)
        at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:94)
        at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2703)
        at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2685)
        at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:373)
        at org.apache.ranger.audit.destination.HDFSAuditDestination.getLogFileStream(HDFSAuditDestination.java:271)
        at org.apache.ranger.audit.destination.HDFSAuditDestination.access$000(HDFSAuditDestination.java:43)
        at org.apache.ranger.audit.destination.HDFSAuditDestination$1.run(HDFSAuditDestination.java:157)
        at org.apache.ranger.audit.destination.HDFSAuditDestination$1.run(HDFSAuditDestination.java:154)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698)
        at org.apache.ranger.audit.provider.MiscUtil.executePrivilegedAction(MiscUtil.java:524)
        at org.apache.ranger.audit.destination.HDFSAuditDestination.logJSON(HDFSAuditDestination.java:154)
        at org.apache.ranger.audit.queue.AuditFileSpool.sendEvent(AuditFileSpool.java:880)
        at org.apache.ranger.audit.queue.AuditFileSpool.runLogAudit(AuditFileSpool.java:828)
        at org.apache.ranger.audit.queue.AuditFileSpool.run(AuditFileSpool.java:758)
        at java.lang.Thread.run(Thread.java:745)

Since this is a bug related to the version of NiFi I am on, I do not need to see it as it happens very frequently and is cluttering the log file. How can I exclude this so it is not logged? Is there some configuration I can do in the logback.xml file?

Re: How can I filter the NiFi log to exclude a certain error message?

MattWho — Tue, 17 Apr 2018 01:59:14 GMT

@Josh Nicholson

NiFi use logback for logging of component classes in NiFi. It is possible to add a logger for that will turn off all logging for specific class but not a specific error.

First you need the complete java class name. Using simply "o.a.r.audit.provider.BaseAuditHandler" will not work. We need to figure out what the "o", "a", and "r" really are.

To get NiFi to print out the entire class you are going to want to edit the following section in the logback.xml:

            <appender name="APP_FILE">
            <file>${org.apache.nifi.bootstrap.config.log.dir}/nifi-app.log</file>
            <rollingPolicy>
            <!--
            For daily rollover, use 'app_%d.log'.
            For hourly rollover, use 'app_%d{yyyy-MM-dd_HH}.log'.
            To GZIP rolled files, replace '.log' with '.log.gz'.
            To ZIP rolled files, replace '.log' with '.log.zip'.
            -->
            <fileNamePattern>${org.apache.nifi.bootstrap.config.log.dir}/nifi-app_%d{yyyy-MM-dd_HH}.%i.log</fileNamePattern>
            <maxFileSize>100MB</maxFileSize>
            <!-- keep 30 log files worth of history -->
            <maxHistory>30</maxHistory>
            <!-- optional setting for keeping 10GB total of log files
            <totalSizeCap>10GB</totalSizeCap>
            -->
            </rollingPolicy>
            <immediateFlush>true</immediateFlush>
            <encoder>
            <pattern>%date %level [%thread] %logger{40} %msg%n</pattern>
            </encoder>
            </appender>

Third line from the bottom is the "pattern" line. Change "{40}" to "{140}".

Save the change.

Tail the nifi-app.log and wait for next occurrence of the above error.

*** The good thing about the logback.xml file is that it can be edited while NiFi changes and those changes will take affect 30 secs to 2 minutes after you save the changes to the file.

Next we will want to edit the logback.xml file again to add a new logger:

look for the following line:

<logger name="org.apache.nifi" level="INFO"/>

Then after this line add a new line similar to above except using the full class name you obtained:

<logger name="org.apache.ranger.audit.provider.BaseAuditHandler" level="OFF"/>

Make sure you set "Level" to "OFF" (all caps).

Save the file and you are done.

Thank you,

Matt

**** If you found this answer addressed your question, please take a moment to login and click "accept"

Re: How can I filter the NiFi log to exclude a certain error message?

josh_nicholson — Tue, 17 Apr 2018 02:32:01 GMT

Hi Matt,

I was able to implement these steps and it did exactly what I needed. Thanks!

Re: How can I filter the NiFi log to exclude a certain error message?

bruno_n_souza — Fri, 16 Nov 2018 22:42:02 GMT

Hi @Josh Nicholson

Can you point what is the bug (URL in jira or any other info) related to the error regarding hdfs scheme? What is your version of NiFi?

Thanks,

Bruno

Hi Josh, can you point what is the bug (URL in jira or any other info) related to the error regarding hdfs scheme? What is your version of NiFi?

Re: How can I filter the NiFi log to exclude a certain error message?

rtheron — Sat, 17 Nov 2018 04:51:51 GMT

Another option could be to use Ambari log search