Can we somehow dump the error messgae shown against processor to an email or to a file. In other words, whatever error is being shown in UI of a processor, can we send/dump this in an email

[#1]

While running PutEmail processor in Apache NiFi, if there is an error then we are able to see that error by clicking red notification.

Can we send this error to specified email

[#2]

@Pradhuman Gupta

The WARN and ERROR messages you see when you float your cursor over the red notification icon on a processor are also written to the nifi-app.log. There is no way to capture those bulletins directly from a processor and route them to a putEmail processor.

If there are specific processor types for which you want to monitor for WARN and/or ERROR messages for, you could modify your NiFi's logback.xml file so that logs generated by those processors classes are written to their own output log file. You could then setup a dataflow that tails that new log and sends an email when WARN and/or ERROR log messages are written to it.

Thanks,

Matt

[#3]

@Matt Clarke

You are saying making a processor specific entry into logback.xml. ?

Any example of this?

[#4]

@Pradhuman Gupta

You cannot setup logging for a specific processor. But you can setup a new logger for a specific processor class.

First you would create a new appender in the nifi logback.xml file:

<appender name="PROCESSOR_FILE" class="ch.qos.logback.core.rolling.RollingFileAppender"> <file>${org.apache.nifi.bootstrap.config.log.dir}/nifi-processsor.log</file> <rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy"> <!-- For daily rollover, use 'user_%d.log'. For hourly rollover, use 'user_%d{yyyy-MM-dd_HH}.log'. To GZIP rolled files, replace '.log' with '.log.gz'. To ZIP rolled files, replace '.log' with '.log.zip'. --> <fileNamePattern>${org.apache.nifi.bootstrap.config.log.dir}/nifi-processor_%d.log</fileNamePattern> <!-- keep 5 log files worth of history --> <maxHistory>5</maxHistory> </rollingPolicy> <encoder class="ch.qos.logback.classic.encoder.PatternLayoutEncoder"> <pattern>%date %level [%thread] %logger{120} %msg%n</pattern> <immediateFlush>true</immediateFlush> </encoder> </appender>

Then you create a new logger that will write to that appender log file:

<logger name="org.apache.nifi.processors.attributes.UpdateAttribute" level="WARN" additivity="false">
        <appender-ref ref="PROCESSOR_FILE"/>
</logger>

In the above example i am creating a logger for the UpdateAttribute processor.

Now any WARN or ERROR log messages produced by this specific processor will be written to this new log.

You can expand upon this flow by configuring loggers for each Processor class you want to monitor and send them to the same appender. Then use a SplitText processor to split the content of the FlowFile produced by the TailFile. then use Route On Content processor to route specific log lines produced by each processor class to a different put email or simply create a different message body attribute for each.

Thanks,

Matt

[#5]

It's kind of a weird use, but I was using NiFi to read the NiFi logs and you could parse them and send that to email. Eating your own dog food, though doing this will work. Now if there's an error in reading that log or using the email processor it will generate another error that will go into the log and fail to send that error and there you go.

http://docs.hortonworks.com/HDPDocuments/HDF2/HDF-2.1.1/bk_dataflow-user-guide/content/Summary_Page....

[#6]

@Pradhuman Gupta

Yes, that is what Matt is saying. Here is a sample of writing the output from the UpdateAttribute processor into it's own log file named nifi-upattr.log in the same directory as the nifi-app.log file:

<appender name="UPATT_FILE" class="ch.qos.logback.core.rolling.RollingFileAppender">
        <file>${org.apache.nifi.bootstrap.config.log.dir}/nifi-upattr.log</file>
        <rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
        <fileNamePattern>${org.apache.nifi.bootstrap.config.log.dir}/nifi-upattr_%d.log.gz</fileNamePattern>
            <!-- keep 30 log files worth of history -->
            <maxHistory>30</maxHistory>
        </rollingPolicy>
        <encoder class="ch.qos.logback.classic.encoder.PatternLayoutEncoder">
            <pattern>%date %level [%thread] %logger{40} %msg%n</pattern>
        </encoder>
    </appender>
<logger name="org.apache.nifi.processors.attributes.UpdateAttribute" level="INFO" additivity="false">
        <appender-ref ref="UPATT_FILE"/>
    </logger>

 

And then tail this log file and send an email when WARN and/or ERROR log messages are written to it.

In this example, I just added this to the logback.xml section of Ambari