Ok , so can i make a flow like ConsumeKafkaRecord  --> topic A --> PublishKafkaRecord  --> topic B , using both record processor for consuming and publishing data ? Will this be fast ? ... View more

Ok , so can i make a flow like ConsumeKafkaRecord  --> topic A --> PublishKafkaRecord  --> topic B , using both record processor for consuming and publishing data ? Will this be fast ? ... View more

Hi Matt ,  Quartz Cron scheduler method worked , but one issue is coming related to time at which i want data to get consumed , i used 0-30 */6 * * * ? cron query , as per this i was expecting data to get consumed after 6 minutes or around 6 minutes . Data published was 50K , but all data got consumed before 3 minutes around 2 mins 50 sec . How to make this time configure accurately  . Please help on this   ... View more

Hi @MattWho  Thanks for your response . I also don't understand the overhead of ingesting the same messages twice in your NiFi My requirement is to send data to different end point , so that they can perform different operations on the data. Why not have have a single ConsumeKafka ingesting the messages from the topic and then routing the success relationship from the ConsumeKafka twice (once to InvokeHTTP A and once to InvokeHTTP B)?   For me one flow is like one vendor , like this i will be having multiple number of vendors , everyone will have there separate end points. Keeping all in one flow is not possible . So I am creating separate data flow and separate retry logic for them. This above issue is with only 1 vendor , they require same data (consumed from same kafka topic ) to be pushed to 2 separate endpoints . But I am not able to handle the retry logic for them. Why publish failed or retry FlowFile messages to an external topic R just so they can be consumed back in to your NiFi?   yes i want them to be consumed again to nifi . All failed requests iam publishing to retry topic and this is being handled in retry flow . With this iam able to keep my main flow without and failed requests and new requests which does not have any error will get pushed to end point successfully It would be more efficient to just keep them in NiFi and create a retry loop on each InvokeHTTP.  NiFi even offers retry handling directly on the relationships with in the processor configuration if i add a retry loop to invoke http and the endpoint is down for a longer time , too many requests will get queued in nifi . If you must write the message out to just one topic R, you'll need to append something to the message that indicates what InvokeHTTP (A or B) failure or retry resulted in it being written to Topic R.  Then have a single Retry dataflow that consume from Topic R, extracts that A or B identifier from message so that it can be routed to the correct invokeHTTP.  Just seems like a lot of unnecessary overhead. Please help me with the retry logic . Data is going in same retry topic how can i differentiate between the data , whether it failed from data flow 1 or from data flow 2.   ... View more

My  "error" topic  is on different Kafka cluster . So i wanted my data which is failing due to connection error should move to failure and then to error topic . Is there any way to do so ??  ... View more