Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

In Apache Ni-fi, how can I log all the flowFile 'attributes' in a text file

Solved Go to solution
Highlighted

In Apache Ni-fi, how can I log all the flowFile 'attributes' in a text file

New Contributor

Hi, I have a requirement, where I am in need to Log all the flowFlile attributes in a separate text file. What I read is that Apache Ni-Fi by design have Provenance which gives a detailed Log information. But, however, since we are new to Ni-Fi, we are not sure about the access levels that to the Production support teams would have, because of which we would like to Log the Attributes in a Text file for which we can give them all access. I also tried using the existing processor "LogAttribute" where I don't see an option to write out into a physical file location.

1 ACCEPTED SOLUTION

Accepted Solutions

Re: In Apache Ni-fi, how can I log all the flowFile 'attributes' in a text file

LogAttribute relies on the logback configuration that everything uses in NiFi uses for logging (logback.xml in conf). Right now it goes into nifi-app.log which is the default appender, but you could probably create a new file appender for something like "nifi-logatttribute.log" and then modify this line:

<logger name="org.apache.nifi.processors.standard.LogAttribute" level="INFO"/>

To be something like:

<logger name="org.apache.nifi.processors.standard.LogAttribute" level="INFO">
 <appender-ref ref="LOGATTRIBUTE_FILE" />
</logger>

Assuming you defined a file appender named LOGATTRIBUTE_FILE.

6 REPLIES 6

Re: In Apache Ni-fi, how can I log all the flowFile 'attributes' in a text file

LogAttribute relies on the logback configuration that everything uses in NiFi uses for logging (logback.xml in conf). Right now it goes into nifi-app.log which is the default appender, but you could probably create a new file appender for something like "nifi-logatttribute.log" and then modify this line:

<logger name="org.apache.nifi.processors.standard.LogAttribute" level="INFO"/>

To be something like:

<logger name="org.apache.nifi.processors.standard.LogAttribute" level="INFO">
 <appender-ref ref="LOGATTRIBUTE_FILE" />
</logger>

Assuming you defined a file appender named LOGATTRIBUTE_FILE.

Re: In Apache Ni-fi, how can I log all the flowFile 'attributes' in a text file

New Contributor

Hi @Bryan Bende,

This answer was very helpful ! Thank you !

I still have a question though : suppose I have the following configuration in logback.xml

<logger name="org.apache.nifi.processors" level="INFO"/>

This means that all processors will log to nifi-app.log with INFO Level, right ? In this case, what kind of information will be logged ? What about DEBUG Level ?

Thanks for your help !

Re: In Apache Ni-fi, how can I log all the flowFile 'attributes' in a text file

Correct, setting INFO on a package means all classes in the package will log at the INFO level, unless there is a more specific logger for a given class at a different level. For example:

<logger name="org.apache.nifi.processors" level="INFO"/>
<logger name="org.apache.nifi.processors.MyProcessor" level="DEBUG"/>

MyProcessor would log at DEBUG level and all other processors at INFO level.

The type of information is really dependent on the processor. Generally you should use DEBUG for things you wouldn't expect to be logging all the time, but that might be helpful if someone was trying to troubleshoot something.

Re: In Apache Ni-fi, how can I log all the flowFile 'attributes' in a text file

New Contributor

Ok thanks again !

Re: In Apache Ni-fi, how can I log all the flowFile 'attributes' in a text file

By default the LogAttribute processor logs to the NiFi App log. It does this because processors are configured to log there by the default configuration in the logback.xml. In order to have a new file that just the LogAttribute Processor uses you can add a new Appender and Logger like so:

    <appender name="STATUS_LOG_FILE" class="ch.qos.logback.core.rolling.RollingFileAppender">
    <file>logs/nifi-status.log</file>
    <rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
        <!--
          For daily rollover, use 'user_%d.log'.
          For hourly rollover, use 'user_%d{yyyy-MM-dd_HH}.log'.
          To GZIP rolled files, replace '.log' with '.log.gz'.
          To ZIP rolled files, replace '.log' with '.log.zip'.
        -->
        <fileNamePattern>./logs/nifi-status_%d.log</fileNamePattern>
        <!-- keep 5 log files worth of history -->
        <maxHistory>5</maxHistory>
    </rollingPolicy>
    <encoder class="ch.qos.logback.classic.encoder.PatternLayoutEncoder">
        <pattern>%date %level [%thread] %logger{40} %msg%n</pattern>
    </encoder>
    </appender>


    <logger name="org.apache.nifi.processors.standard.LogAttribute" level="INFO" additivity="false">
        <appender-ref ref="STATUS_LOG_FILE" />
    </logger>

Re: In Apache Ni-fi, how can I log all the flowFile 'attributes' in a text file

New Contributor

@jpercivall is it possible to redirect log output for LogAttribute processor only in a specific Process group? I don't want to redirect globally for my nifi installation for anyone using LogAttribute. I only want to set an Appender and a Logger that redirects for LogAttribute in my Process Group.

Don't have an account?
Coming from Hortonworks? Activate your account here