In Apache Ni-fi, how can I log all the flowFile 'attributes' in a text file

[#1]

Hi, I have a requirement, where I am in need to Log all the flowFlile attributes in a separate text file. What I read is that Apache Ni-Fi by design have Provenance which gives a detailed Log information. But, however, since we are new to Ni-Fi, we are not sure about the access levels that to the Production support teams would have, because of which we would like to Log the Attributes in a Text file for which we can give them all access. I also tried using the existing processor "LogAttribute" where I don't see an option to write out into a physical file location.

[#2 · Accepted solution]

LogAttribute relies on the logback configuration that everything uses in NiFi uses for logging (logback.xml in conf). Right now it goes into nifi-app.log which is the default appender, but you could probably create a new file appender for something like "nifi-logatttribute.log" and then modify this line:

<logger name="org.apache.nifi.processors.standard.LogAttribute" level="INFO"/>

To be something like:

<logger name="org.apache.nifi.processors.standard.LogAttribute" level="INFO">
 <appender-ref ref="LOGATTRIBUTE_FILE" />
</logger>

Assuming you defined a file appender named LOGATTRIBUTE_FILE.

[#3]

Hi @Bryan Bende,

This answer was very helpful ! Thank you !

I still have a question though : suppose I have the following configuration in logback.xml

<logger name="org.apache.nifi.processors" level="INFO"/>

This means that all processors will log to nifi-app.log with INFO Level, right ? In this case, what kind of information will be logged ? What about DEBUG Level ?

Thanks for your help !

[#4]

Correct, setting INFO on a package means all classes in the package will log at the INFO level, unless there is a more specific logger for a given class at a different level. For example:

<logger name="org.apache.nifi.processors" level="INFO"/>
<logger name="org.apache.nifi.processors.MyProcessor" level="DEBUG"/>

MyProcessor would log at DEBUG level and all other processors at INFO level.

The type of information is really dependent on the processor. Generally you should use DEBUG for things you wouldn't expect to be logging all the time, but that might be helpful if someone was trying to troubleshoot something.

[#5]

Ok thanks again !

[#6]

By default the LogAttribute processor logs to the NiFi App log. It does this because processors are configured to log there by the default configuration in the logback.xml. In order to have a new file that just the LogAttribute Processor uses you can add a new Appender and Logger like so:

    <appender name="STATUS_LOG_FILE" class="ch.qos.logback.core.rolling.RollingFileAppender">
    <file>logs/nifi-status.log</file>
    <rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
        <!--
          For daily rollover, use 'user_%d.log'.
          For hourly rollover, use 'user_%d{yyyy-MM-dd_HH}.log'.
          To GZIP rolled files, replace '.log' with '.log.gz'.
          To ZIP rolled files, replace '.log' with '.log.zip'.
        -->
        <fileNamePattern>./logs/nifi-status_%d.log</fileNamePattern>
        <!-- keep 5 log files worth of history -->
        <maxHistory>5</maxHistory>
    </rollingPolicy>
    <encoder class="ch.qos.logback.classic.encoder.PatternLayoutEncoder">
        <pattern>%date %level [%thread] %logger{40} %msg%n</pattern>
    </encoder>
    </appender>


    <logger name="org.apache.nifi.processors.standard.LogAttribute" level="INFO" additivity="false">
        <appender-ref ref="STATUS_LOG_FILE" />
    </logger>

[#7]

Even if we have specific appender for LogAttribute, why does it still logs it to default nifi-app.log file in addition to the new file? How to avoid this duplication of logs

[#8]

@Vin900, as this is an older post, you would have a better chance of receiving a resolution by starting a new thread. This will also be an opportunity to provide details specific to your environment that could aid others in assisting you with a more accurate answer to your question. You can link this thread as a reference in your new post.

[#9]

@jpercivall is it possible to redirect log output for LogAttribute processor only in a specific Process group? I don't want to redirect globally for my nifi installation for anyone using LogAttribute. I only want to set an Appender and a Logger that redirects for LogAttribute in my Process Group.