@Gaurav Jain Each node in a cluster is responsible for working on its own FlowFiles. Each node is unaware of what FlowFiles other nodes are working on. If a NiFi processor component is working on a FlowFile at the time the Node goes down, the transformation work will start over once that the node is running again. A node disconnecting will not cause processing of FlowFiles to stop on the disconnected node. Processors that do transformation of FlowFile content will produce a new FlowFile once the transformation is complete. So if failure exists mid processing, the original remains on the incoming queue to the processor and the intermediate work is lost. This is how NiFi ensures no data loss occurs in unexpected failures. That being said Data plane High Availability (HA) is one of NiFi's roadmap items. Thanks, Matt ... View more

@Gaurav Jain Please explain what you mean when you say "complete workflow not working". Screenshots may help if you can provide them. ... View more

@Gaurav Jain When you find an answer that address your question, please accept that answer to benefit other who come to this forum for help. Thank you, Matt ... View more

@Gaurav Jain The flow.xml.gz file contains everything (Processors, connections, controller services, etc. that make up your dataflow(s) on your canvas. If you try to make a change to a dataflow which has a disconnected node, you will get a response from Nifi that says changes are not allowed while a node is disconnected. You can take manual steps to delete the disconnected node from the cluster via the cluster UI. This will return control, but the node you deleted will not be able to rejoin cluster later (because flows will not match) without doing additional manual steps. Matt ... View more

@umair ahmed Just spoke with Dave and he cleaned up his template/response to you here: https://community.hortonworks.com/questions/101496/nifi-invokehttp-retries.html#answer-101588 His solution for triggering sleep based on retry count set in my template is perfect for meeting your needs. It also scales very easily by simply adding additional timer rules to the advanced UI of the UpdateAttribute processor. Thanks, Matt ... View more

@umair ahmed The Retry loop template above allows you to configure the number of retry attempts before existing the loop. I am not sure what you mean by "on time that is it retry at certain time". If the intent is to slow how fast the FlowFile is retired, you could add an additional routeOnAttribute processor to the failure loop to to loop until file has aged x amount of time. Thanks, Matt ... View more

@ismail patel Backpressure thresholds are soft limits and some processors do batch processing. The listHDFS processor will produce a list of Files from HDFS and produce a single 0 byte FlowFile for each file in that list. It will then commit all those FlowFiles to the success relationship at once. So if back pressure threshold was set to 5, the ListHDFS processor would still dump all FlowFiles on to it. (even if the listing consisted of 1000s of Files). At that point backpressure would be applied and prevent the listHDFS form running again until the queue dropped back below 5, but this is not the behavior you need here. The RouteOnAttribute processor is one of those processors that works on 1 FlowFile at a time. This allows us to more strictly adhere to the back pressure setting of 5 on its unmatched relationship. The fact that I used a RouteOnAttribute processor is not important, any processor that works on FlowFiles one at a time would work. I picked RouteOnAttribute because it operates off of FlowFile Attributes which live in heap memory which makes processing here very fast. Thanks, Matt ... View more

@Pradhuman Gupta You cannot setup logging for a specific processor. But you can setup a new logger for a specific processor class. First you would create a new appender in the nifi logback.xml file: <appender name="PROCESSOR_FILE" class="ch.qos.logback.core.rolling.RollingFileAppender"> <file>${org.apache.nifi.bootstrap.config.log.dir}/nifi-processsor.log</file> <rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy"> <!-- For daily rollover, use 'user_%d.log'. For hourly rollover, use 'user_%d{yyyy-MM-dd_HH}.log'. To GZIP rolled files, replace '.log' with '.log.gz'. To ZIP rolled files, replace '.log' with '.log.zip'. --> <fileNamePattern>${org.apache.nifi.bootstrap.config.log.dir}/nifi-processor_%d.log</fileNamePattern> <!-- keep 5 log files worth of history --> <maxHistory>5</maxHistory> </rollingPolicy> <encoder class="ch.qos.logback.classic.encoder.PatternLayoutEncoder"> <pattern>%date %level [%thread] %logger{120} %msg%n</pattern> <immediateFlush>true</immediateFlush> </encoder> </appender> Then you create a new logger that will write to that appender log file: <logger name="org.apache.nifi.processors.attributes.UpdateAttribute" level="WARN" additivity="false"> <appender-ref ref="PROCESSOR_FILE"/> </logger> In the above example i am creating a logger for the UpdateAttribute processor. Now any WARN or ERROR log messages produced by this specific processor will be written to this new log. You can expand upon this flow by configuring loggers for each Processor class you want to monitor and send them to the same appender. Then use a SplitText processor to split the content of the FlowFile produced by the TailFile. then use Route On Content processor to route specific log lines produced by each processor class to a different put email or simply create a different message body attribute for each. Thanks, Matt ... View more