Nifi and IOT hub connectivity breaks intermittently

[#1]

Hi Team,

I have Nifi connectivity with ioT Hub. In ioT hub i have device under which publisher and subscriber is there. Nifi is configured with ioT hub and its working fine however intermittently the connectivity breaks between ioT and nifi.

Do we have any mechanism or process in nifi to identify when connectivity breaks it retry the connection. If restart the nifi service manually then again connection establishes. I want to automize this process either from ioT hub or from nifi side if connectivity breaks then it should retry connection and restart the nifi service automatically.

@cotopaul @Apache  @ApacheNifi 

[#2]

@AhmedParvez 
How is your NiFi connectivity with you IoT Hub being made?
Is this a connection established through some NiFi processor or controller service component within NiFi?
It is not clear to me why this connectivity issue requires a full NiFi service restart.
How does restarting the service fix the connectivity?

Thank you,
Matt

[#3]

Hi @MattWho, Thanks for the quick response. Nifi is connected through MQTT Processor. When connectivity breaks between ioT Hub and Nifi we don't have solution to again establish the connect so we simply restart the Nifi Service and it work for us. 

I am looking for solution how we can identify the connection between ioT hub and Nifi is active or not is there any Nifi processor which checks the connection health.

If it disconnected then how can we re establish the connection automatically from ioT hub and or from Nifi without restarting the Nifi service. 

[#4]

@AhmedParvez 

Does the ConsumeMQTT or PublishMQTT processors you are using throw an exception or produce a bulletin when the connectivity is not working?

If so you may be able to construct another dataflow on your NiFi canvas that can monitor for that bulletin exception and then trigger a NiFi restart or better yet, just trigger a stop and restart of the processor component which i suspect would also address your issue since all restarting NiFi is doing is restarting all components.

Doing this would require adding a SiteToSiteBulletinReportingTask that would be a remote inout port that starts a new dataflow on your canvas.  You would then need to filter from all bulletins received those related to your MQTT processor. Those filtered out messages could be used to trigger a rest-api request via an InvokeHTTP processor to stop and then another to start the MQTT  processor to re-establish your connection.

If you found any of the suggestions/solutions provided helped you with your issue, please take a moment to login and click "Accept as Solution" on one or more of them that helped.

Thank you,
Matt

[#5]

@AhmedParvez 
Of course digging in to why the connections are failing and addressing that is always going to be the best option rather then building work arounds like this.  MY suggestions should be used a method to sustain operations while working on the correct solution.  Perhaps that solutions means contributing to improvements to these MQTT processors or identifying deficiencies in these Processors and then raising detailed Apache NiFi jira tickets in the community to improve them. 

[#6]

Large-scale implementations of NiFi are rather new, and thus what I'm saying here can be volatile, but nonetheless I am someone who has done it both experimenting personally and as POC research for clients, and am in touch with several engineers at my company who are currently implementing NiFi for a client (I would've been on the project but am currently critical to another client).

[#7]

@ranveera95 It has been +9 years since Apache NiFi was open sourced to the world. I would not say NiFi is rather new to large scale implementations. It has been used with very large scale deployments handling massive amounts of data.  Often built using both Apache MiNiFi agents and large scale Apache NiFi clusters to facilitate the collection of data from small too large scale edged devices to central cluster data management and routing NiFi clusters.