@umair ahmed Just spoke with Dave and he cleaned up his template/response to you here: https://community.hortonworks.com/questions/101496/nifi-invokehttp-retries.html#answer-101588 His solution for triggering sleep based on retry count set in my template is perfect for meeting your needs. It also scales very easily by simply adding additional timer rules to the advanced UI of the UpdateAttribute processor. Thanks, Matt ... View more

@umair ahmed The Retry loop template above allows you to configure the number of retry attempts before existing the loop. I am not sure what you mean by "on time that is it retry at certain time". If the intent is to slow how fast the FlowFile is retired, you could add an additional routeOnAttribute processor to the failure loop to to loop until file has aged x amount of time. Thanks, Matt ... View more

@ismail patel Backpressure thresholds are soft limits and some processors do batch processing. The listHDFS processor will produce a list of Files from HDFS and produce a single 0 byte FlowFile for each file in that list. It will then commit all those FlowFiles to the success relationship at once. So if back pressure threshold was set to 5, the ListHDFS processor would still dump all FlowFiles on to it. (even if the listing consisted of 1000s of Files). At that point backpressure would be applied and prevent the listHDFS form running again until the queue dropped back below 5, but this is not the behavior you need here. The RouteOnAttribute processor is one of those processors that works on 1 FlowFile at a time. This allows us to more strictly adhere to the back pressure setting of 5 on its unmatched relationship. The fact that I used a RouteOnAttribute processor is not important, any processor that works on FlowFiles one at a time would work. I picked RouteOnAttribute because it operates off of FlowFile Attributes which live in heap memory which makes processing here very fast. Thanks, Matt ... View more

@Pradhuman Gupta You cannot setup logging for a specific processor. But you can setup a new logger for a specific processor class. First you would create a new appender in the nifi logback.xml file: <appender name="PROCESSOR_FILE" class="ch.qos.logback.core.rolling.RollingFileAppender"> <file>${org.apache.nifi.bootstrap.config.log.dir}/nifi-processsor.log</file> <rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy"> <!-- For daily rollover, use 'user_%d.log'. For hourly rollover, use 'user_%d{yyyy-MM-dd_HH}.log'. To GZIP rolled files, replace '.log' with '.log.gz'. To ZIP rolled files, replace '.log' with '.log.zip'. --> <fileNamePattern>${org.apache.nifi.bootstrap.config.log.dir}/nifi-processor_%d.log</fileNamePattern> <!-- keep 5 log files worth of history --> <maxHistory>5</maxHistory> </rollingPolicy> <encoder class="ch.qos.logback.classic.encoder.PatternLayoutEncoder"> <pattern>%date %level [%thread] %logger{120} %msg%n</pattern> <immediateFlush>true</immediateFlush> </encoder> </appender> Then you create a new logger that will write to that appender log file: <logger name="org.apache.nifi.processors.attributes.UpdateAttribute" level="WARN" additivity="false"> <appender-ref ref="PROCESSOR_FILE"/> </logger> In the above example i am creating a logger for the UpdateAttribute processor. Now any WARN or ERROR log messages produced by this specific processor will be written to this new log. You can expand upon this flow by configuring loggers for each Processor class you want to monitor and send them to the same appender. Then use a SplitText processor to split the content of the FlowFile produced by the TailFile. then use Route On Content processor to route specific log lines produced by each processor class to a different put email or simply create a different message body attribute for each. Thanks, Matt ... View more

@ismail patel Nodes in a nifi cluster really don't know about each other. They find out from zookeeper who the elected cluster coordinator is and start sending health and status messages via heartbeats. Each node in the cluster runs its own copy of the dataflow and works on its own set of FlowFiles. When a Node goes down other nodes do not pickup FlowFiles from that node an work on them. I am not following exactly what you did above. If you disconnect a node from the cluster, that heath and status heartbeats will no longer be coming in. Any queued data on that node should not be reflected in the cluster view until the node is reconnected. If the RPG was not sending FlowFiles to Node 2, there is likely an issue with the connection. Make sure you correctly configured the S2S properties in the nifi.properties file on both node correctly and that there is no firewall blocking the connection. ... View more

@ismail patel The ability to set FlowFile batch size is coming in Apache NiFi 1.2.0 which should be up for vote any day now. https://issues.apache.org/jira/browse/NIFI-1202 Thanks, Matt ... View more