Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Can we somehow dump the error messgae shown against processor to an email or to a file. In other words, whatever error is being shown in UI of a processor, can we send/dump this in an email

avatar
Rising Star

While running PutEmail processor in Apache NiFi, if there is an error then we are able to see that error by clicking red notification.

Can we send this error to specified email

5 REPLIES 5

avatar
Super Mentor

@Pradhuman Gupta

The WARN and ERROR messages you see when you float your cursor over the red notification icon on a processor are also written to the nifi-app.log. There is no way to capture those bulletins directly from a processor and route them to a putEmail processor.

If there are specific processor types for which you want to monitor for WARN and/or ERROR messages for, you could modify your NiFi's logback.xml file so that logs generated by those processors classes are written to their own output log file. You could then setup a dataflow that tails that new log and sends an email when WARN and/or ERROR log messages are written to it.

Thanks,

Matt

avatar
Rising Star

@Matt Clarke

You are saying making a processor specific entry into logback.xml. ?

Any example of this?

avatar
Super Mentor

@Pradhuman Gupta

You cannot setup logging for a specific processor. But you can setup a new logger for a specific processor class.

First you would create a new appender in the nifi logback.xml file:

<appender name="PROCESSOR_FILE" class="ch.qos.logback.core.rolling.RollingFileAppender"> <file>${org.apache.nifi.bootstrap.config.log.dir}/nifi-processsor.log</file> <rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy"> <!-- For daily rollover, use 'user_%d.log'. For hourly rollover, use 'user_%d{yyyy-MM-dd_HH}.log'. To GZIP rolled files, replace '.log' with '.log.gz'. To ZIP rolled files, replace '.log' with '.log.zip'. --> <fileNamePattern>${org.apache.nifi.bootstrap.config.log.dir}/nifi-processor_%d.log</fileNamePattern> <!-- keep 5 log files worth of history --> <maxHistory>5</maxHistory> </rollingPolicy> <encoder class="ch.qos.logback.classic.encoder.PatternLayoutEncoder"> <pattern>%date %level [%thread] %logger{120} %msg%n</pattern> <immediateFlush>true</immediateFlush> </encoder> </appender>

Then you create a new logger that will write to that appender log file:

<logger name="org.apache.nifi.processors.attributes.UpdateAttribute" level="WARN" additivity="false">
        <appender-ref ref="PROCESSOR_FILE"/>
</logger>

In the above example i am creating a logger for the UpdateAttribute processor.

Now any WARN or ERROR log messages produced by this specific processor will be written to this new log.

15127-screen-shot-2017-05-05-at-55818-pm.png

You can expand upon this flow by configuring loggers for each Processor class you want to monitor and send them to the same appender. Then use a SplitText processor to split the content of the FlowFile produced by the TailFile. then use Route On Content processor to route specific log lines produced by each processor class to a different put email or simply create a different message body attribute for each.

Thanks,

Matt

avatar
Master Guru

It's kind of a weird use, but I was using NiFi to read the NiFi logs and you could parse them and send that to email. Eating your own dog food, though doing this will work. Now if there's an error in reading that log or using the email processor it will generate another error that will go into the log and fail to send that error and there you go.

http://docs.hortonworks.com/HDPDocuments/HDF2/HDF-2.1.1/bk_dataflow-user-guide/content/Summary_Page....

avatar

@Pradhuman Gupta

Yes, that is what Matt is saying. Here is a sample of writing the output from the UpdateAttribute processor into it's own log file named nifi-upattr.log in the same directory as the nifi-app.log file:

<appender name="UPATT_FILE" class="ch.qos.logback.core.rolling.RollingFileAppender">
        <file>${org.apache.nifi.bootstrap.config.log.dir}/nifi-upattr.log</file>
        <rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
        <fileNamePattern>${org.apache.nifi.bootstrap.config.log.dir}/nifi-upattr_%d.log.gz</fileNamePattern>
            <!-- keep 30 log files worth of history -->
            <maxHistory>30</maxHistory>
        </rollingPolicy>
        <encoder class="ch.qos.logback.classic.encoder.PatternLayoutEncoder">
            <pattern>%date %level [%thread] %logger{40} %msg%n</pattern>
        </encoder>
    </appender>
<logger name="org.apache.nifi.processors.attributes.UpdateAttribute" level="INFO" additivity="false">
        <appender-ref ref="UPATT_FILE"/>
    </logger>

 

And then tail this log file and send an email when WARN and/or ERROR log messages are written to it.

In this example, I just added this to the logback.xml section of Ambari

15126-screen-shot-2017-05-05-at-45227-pm.png