Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

How can I filter the NiFi log to exclude a certain error message?

avatar
Expert Contributor

I am constantly seeing this error message in the nifi-app.log file.

2018-04-16 11:00:36,383 ERROR [nifi.async.batch_nifi.async.batch.hdfs_destWriter] o.a.r.audit.provider.BaseAuditHandler Error writing to log file.
java.io.IOException: No FileSystem for scheme: hdfs
        at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2660)
        at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2667)
        at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:94)
        at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2703)
        at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2685)
        at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:373)
        at org.apache.ranger.audit.destination.HDFSAuditDestination.getLogFileStream(HDFSAuditDestination.java:271)
        at org.apache.ranger.audit.destination.HDFSAuditDestination.access$000(HDFSAuditDestination.java:43)
        at org.apache.ranger.audit.destination.HDFSAuditDestination$1.run(HDFSAuditDestination.java:157)
        at org.apache.ranger.audit.destination.HDFSAuditDestination$1.run(HDFSAuditDestination.java:154)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698)
        at org.apache.ranger.audit.provider.MiscUtil.executePrivilegedAction(MiscUtil.java:524)
        at org.apache.ranger.audit.destination.HDFSAuditDestination.logJSON(HDFSAuditDestination.java:154)
        at org.apache.ranger.audit.queue.AuditFileSpool.sendEvent(AuditFileSpool.java:880)
        at org.apache.ranger.audit.queue.AuditFileSpool.runLogAudit(AuditFileSpool.java:828)
        at org.apache.ranger.audit.queue.AuditFileSpool.run(AuditFileSpool.java:758)
        at java.lang.Thread.run(Thread.java:745)

Since this is a bug related to the version of NiFi I am on, I do not need to see it as it happens very frequently and is cluttering the log file. How can I exclude this so it is not logged? Is there some configuration I can do in the logback.xml file?

1 ACCEPTED SOLUTION

avatar
Super Mentor
@Josh Nicholson

NiFi use logback for logging of component classes in NiFi. It is possible to add a logger for that will turn off all logging for specific class but not a specific error.

-

First you need the complete java class name. Using simply "o.a.r.audit.provider.BaseAuditHandler" will not work. We need to figure out what the "o", "a", and "r" really are.

To get NiFi to print out the entire class you are going to want to edit the following section in the logback.xml:

            <appender name="APP_FILE">
            <file>${org.apache.nifi.bootstrap.config.log.dir}/nifi-app.log</file>
            <rollingPolicy>
            <!--
            For daily rollover, use 'app_%d.log'.
            For hourly rollover, use 'app_%d{yyyy-MM-dd_HH}.log'.
            To GZIP rolled files, replace '.log' with '.log.gz'.
            To ZIP rolled files, replace '.log' with '.log.zip'.
            -->
            <fileNamePattern>${org.apache.nifi.bootstrap.config.log.dir}/nifi-app_%d{yyyy-MM-dd_HH}.%i.log</fileNamePattern>
            <maxFileSize>100MB</maxFileSize>
            <!-- keep 30 log files worth of history -->
            <maxHistory>30</maxHistory>
            <!-- optional setting for keeping 10GB total of log files
            <totalSizeCap>10GB</totalSizeCap>
            -->
            </rollingPolicy>
            <immediateFlush>true</immediateFlush>
            <encoder>
            <pattern>%date %level [%thread] %logger{40} %msg%n</pattern>
            </encoder>
            </appender>

Third line from the bottom is the "pattern" line. Change "{40}" to "{140}".

Save the change.

-

Tail the nifi-app.log and wait for next occurrence of the above error.

-

*** The good thing about the logback.xml file is that it can be edited while NiFi changes and those changes will take affect 30 secs to 2 minutes after you save the changes to the file.

-

Next we will want to edit the logback.xml file again to add a new logger:

-

look for the following line:

<logger name="org.apache.nifi" level="INFO"/>

-

Then after this line add a new line similar to above except using the full class name you obtained:

<logger name="org.apache.ranger.audit.provider.BaseAuditHandler" level="OFF"/>

-

Make sure you set "Level" to "OFF" (all caps).

Save the file and you are done.

-

Thank you,

Matt

-

**** If you found this answer addressed your question, please take a moment to login and click "accept"

View solution in original post

4 REPLIES 4

avatar
Super Mentor
@Josh Nicholson

NiFi use logback for logging of component classes in NiFi. It is possible to add a logger for that will turn off all logging for specific class but not a specific error.

-

First you need the complete java class name. Using simply "o.a.r.audit.provider.BaseAuditHandler" will not work. We need to figure out what the "o", "a", and "r" really are.

To get NiFi to print out the entire class you are going to want to edit the following section in the logback.xml:

            <appender name="APP_FILE">
            <file>${org.apache.nifi.bootstrap.config.log.dir}/nifi-app.log</file>
            <rollingPolicy>
            <!--
            For daily rollover, use 'app_%d.log'.
            For hourly rollover, use 'app_%d{yyyy-MM-dd_HH}.log'.
            To GZIP rolled files, replace '.log' with '.log.gz'.
            To ZIP rolled files, replace '.log' with '.log.zip'.
            -->
            <fileNamePattern>${org.apache.nifi.bootstrap.config.log.dir}/nifi-app_%d{yyyy-MM-dd_HH}.%i.log</fileNamePattern>
            <maxFileSize>100MB</maxFileSize>
            <!-- keep 30 log files worth of history -->
            <maxHistory>30</maxHistory>
            <!-- optional setting for keeping 10GB total of log files
            <totalSizeCap>10GB</totalSizeCap>
            -->
            </rollingPolicy>
            <immediateFlush>true</immediateFlush>
            <encoder>
            <pattern>%date %level [%thread] %logger{40} %msg%n</pattern>
            </encoder>
            </appender>

Third line from the bottom is the "pattern" line. Change "{40}" to "{140}".

Save the change.

-

Tail the nifi-app.log and wait for next occurrence of the above error.

-

*** The good thing about the logback.xml file is that it can be edited while NiFi changes and those changes will take affect 30 secs to 2 minutes after you save the changes to the file.

-

Next we will want to edit the logback.xml file again to add a new logger:

-

look for the following line:

<logger name="org.apache.nifi" level="INFO"/>

-

Then after this line add a new line similar to above except using the full class name you obtained:

<logger name="org.apache.ranger.audit.provider.BaseAuditHandler" level="OFF"/>

-

Make sure you set "Level" to "OFF" (all caps).

Save the file and you are done.

-

Thank you,

Matt

-

**** If you found this answer addressed your question, please take a moment to login and click "accept"

avatar
Expert Contributor

Hi Matt,

I was able to implement these steps and it did exactly what I needed. Thanks!

avatar
New Contributor

Hi @Josh Nicholson

Can you point what is the bug (URL in jira or any other info) related to the error regarding hdfs scheme? What is your version of NiFi?

Thanks,

Bruno

,

Hi Josh, can you point what is the bug (URL in jira or any other info) related to the error regarding hdfs scheme? What is your version of NiFi?

avatar
Rising Star

Another option could be to use Ambari log search