I have a requirement where I need to invoke API at the end of successful data dump to cassandra table. I have used below processors in my nifi flow.
QueryDataBaseTableRecord --> SplitLines --> ExecuteScript --> PutCassandraQL --> InvokeHTTP --> Invoke HTTP --> PutS3
Usecase : Need to dump the data from DB2 to Cassandra, Once after the successful dump it should invoke first API InvokeHTTP(this API has to dump around 1 lakh records to cassandra and send the response message as "Data Inserted" back to nifi) , once we get this response message the second API needs to be invoked InvokeHTTP.
Problem is Nifi automatically routes to failure flow (after ~1 minute or so) as its not waiting for response message from 1st Invoke API, but when we checked at the backend cassandra end data dumping is still in progress.
Could anyone please let me know - How to make invokehttp processor to wait for API response message and route to next step ? Appreciate your suggestion if there is any other effective/alternative way we can try to handle this usecase.
On the FlowFile that is routed to the failure relationship of the first invokeHTTP processor, you will want to look at the FlowFile attributes added to that FlowFile to get a clear understanding of why it was routed to failure.
The invokeHTTTP processor has a "Read Timeout" time property which is used to configure "Max wait time for response from remote service". Make sure this property is set to a value large enough to allow for response to get returned.
If routing to failure is happening sooner then this timeframe, the FlowFile attribute will hopefully shed some more on why the FlowFile was routed to failure.
You can look at the FlowFile attributes by using one of the following methods:
1. Query the FlowFile in NiFi provenance and look at details of event produced by the invokeHTTP processor
2. Allow the FlowFile to queue in the failure relationship connection (stop downstream processor that connection routes to) and then list the queue by right clicking on it and selecting view details icon for FlowFile in the returned list.
3. Route the connection with the failure relationship to a logAttribute processor so that all the FlowFile attributes are logged to the nifi-app.log.
Hope this helps,