How to direct Nifi log to console correctly

[#1]

Hi,

In one of my project, I need to redirect Nifi logs to console/terminal.

Here is my logback.xml

    <appender name="CONSOLE" class="ch.qos.logback.core.ConsoleAppender">
        <encoder class="ch.qos.logback.classic.encoder.PatternLayoutEncoder">
            <pattern>%date %level %msg%n</pattern>
        </encoder>
    </appender>

....

    <logger name="org.apache.nifi.bootstrap" level="INFO" additivity="false">
        <appender-ref ref="BOOTSTRAP_FILE" />
    </logger>
    <logger name="org.apache.nifi.bootstrap.Command" level="INFO" additivity="false">
        <appender-ref ref="BOOTSTRAP_FILE" />
    </logger>


    <!-- Everything written to NiFi's Standard Out will be logged with the logger org.apache.nifi.StdOut at INFO level -->
    <logger name="org.apache.nifi.StdOut" level="INFO" additivity="true">
        <appender-ref ref="CONSOLE" />
        <appender-ref ref="BOOTSTRAP_FILE" />
    </logger>
    
    <!-- Everything written to NiFi's Standard Error will be logged with the logger org.apache.nifi.StdErr at ERROR level -->
    <logger name="org.apache.nifi.StdErr" level="ERROR" additivity="false">
        <appender-ref ref="BOOTSTRAP_FILE" />
    </logger>
    

    <root level="INFO">
        <appender-ref ref="CONSOLE" />
        <appender-ref ref="APP_FILE"/>
    </root>

However, messages in console are logged twice. (Timestamp, Level appear twice)

2018-05-04 13:30:32,265 INFO 2018-05-04 13:30:32,265 INFO Initiating checkpoint of FlowFile Repository
2018-05-04 13:30:32,265 INFO 2018-05-04 13:30:32,265 INFO Initiating checkpoint of FlowFile Repository
2018-05-04 13:30:32,403 INFO 2018-05-04 13:30:32,403 INFO org.wali.MinimalLockingWriteAheadLog@22f8be7c checkpointed with 40 Records and 91 Swap Files in 138 milliseconds (Stop-the-world time = 72 milliseconds, Clear Edit Logs time = 61 millis), max Transaction ID 7501026

consoleLogs.png

I try following ways

1. Remove <appender-ref ref="CONSOLE" /> in root logger

2. Remove <appender-ref ref="CONSOLE" /> in stdout logger

3.Use customized stdout appender like this post

But they either make no change or do not show logs on console at all.

Any help would be appreciated.

Thanks!

[#2]

Hi @Alex Ye,

If I understend correctly, your problem is that on the log is showing two lines with the same information. That is it?

About this, let me know how many instances of NiFi do you have on your cluster?

[#3]

Hi @Gilvan,

Thanks for answering. The problem is NOT showing two lines with same information. But two prefix in one log:

2018-05-04 13:30:32,265 INFO 2018-05-04 13:30:32,265 INFO 

Thanks!

Alex

[#4]

Hi @Alex Ye,

I looked on my NiFi logback.xml and my lines is equal to the folow:

<encoder>
    <pattern>%date %level [%thread] %logger{40} %msg%n</pattern>
</encoder>

[#5]

Hi @Alex Ye I've just had same problem. My solution (a workaround would be more correct though) is following:

1. Create as "passthrough" appender:

<appender name="PT">
    <encoder>
        <pattern>%msg%n</pattern>
    </encoder>
</appender>

This appender would simply print any message as is without any timestamps or anything like that.

2. Make org.apache.nifi.StdOut and org.apache.nifi.StdErr use that appender like that:

<logger name="org.apache.nifi.StdOut" level="INFO" additivity="false">
    <appender-ref ref="PT" />
</logger>
<logger name="org.apache.nifi.StdErr" level="ERROR" additivity="false">
    <appender-ref ref="PT" />
</logger>

I think the problem is in RunNiFi class. There is a method that would setup logging handler which would capture anything from System.out and System.err and log them into org.apache.nifi.StdOut/StdErr loggers. Not sure what purpose it has though and I found no way to disable it.

[#6]

I am facing the issue right now. Let me know if you figure it out and i will be sure to do the same 🙂

[#7]

when i tried this pattern

pattern>%date %level [%thread] ~~~ %logger{40} ~~~ %msg%n</pattern>

i end up with the following ... so, the logger entry seems to be the issue here

2018-06-15 03:32:48,475 ERROR [NiFi logging handler] ~~~ org.apache.nifi.StdErr ~~~ 2018-06-15 03:32:48,474 ERROR [Timer-Driven Process Thread-2] ~~~ o.a.nifi.processors.standard.GetHTTP ~~~ GetHTTP[id=0180258e-0164-1000-8663-45f7e737263c] received status code 404:Unknown site! from http://www.zkhzhjgcjhzxgc.com