Support Questions
Find answers, ask questions, and share your expertise
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

How to handler failure state of PublishKafka in NiFi?

How to handler failure state of PublishKafka in NiFi?

New Contributor

Once PublishKafka fails, then processor keeps running. Best way to stop this retrying in PublishKafka?


In case of wrong Kafka Brokers, PublishKafka Processor keeps running. How to handler this running to send the flow to failure Log Attribute.


Re: How to handler failure state of PublishKafka in NiFi?

The PublishKafka doesn't fail on broker, as you should be sending it a list of several brokers.  If your Kafka Broker is only a single node, you are missing the basics on Kafka Design.  By design this "list of brokers" or a Kafka Cluster should be high availability and at least one should always be online.   This behavior allows PublishKafka to retry through the list of brokers.




If you still want tap into the Kafka Broker failure, one idea would be to create a NiFi flow that tails your NiFi Cluster's log files looking for Error Scenario caused when a broker fails.   You could additionally program this flow to execute an API or CLI command to turn the Publish Kafka Processor off to stop it from retrying while the Kafka Cluster issue is resolved.   



If this answer resolves your issue or allows you to move forward, please choose to ACCEPT this solution and close this topic. If you have further dialogue on this topic please comment here or feel free to private message me. If you have new questions related to your Use Case please create separate topic and feel free to tag me in your post.  




Don't have an account?
Coming from Hortonworks? Activate your account here