Nifi GetHttp question

[#1]

Hey

Anyone knows if it is possible to change URL for GetHTTP processor dynamically--to change the URL string on the fly? It does support expression. But the processor seem to not be able to take any incoming connection.

Thanks!

Shannon

[#2]

Can you provide a little more detail on your use case? Where will the URLs you want to use originate from?

[#3]

@mclark

The URL string includes a number of parameters. One of the parameters needs to be changed. UpdateAttribute is used to change the parameter value, which is a variable in URL string. The problem is that GetHTTP doesn't allow incoming connection from UpdateAttribute. Is there any other way of achieving the same? Thanks, Shannon

[#4 · Accepted solution]

The GetHttp processor does not allow incoming connections. If you would like to dynamically change the URL in response to the attributes of FlowFiles I'd suggest using the InvokeHttp[1] processor instead of GetHttp. [1] https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi.processors.standard.InvokeHTTP/ind...

[#5]

@jpercivall

Hey, got a follow up question-- InvokeHttp Original does not output any content. It should output whatever get from the url, correct? Data provenance content tab shows "Replay: Cannot replay data from Provenance Event because the event does not contain the required Content Claim". Do you know what is the problem for that processor configuration?

Thanks!

Shannon

[#6]

Understanding your flow will help us understand what is going on. 1. Are you creating a zero byte file that you are using as the trigger for your InvokeHTTP processor? 2. How do you have the invokeHTTP processor configured? (Is it set to Put Response Body In Attribute?) If Put Response Body In Attribute is set to an attribute value, the content of the Flowfile on the "original" relationship will still have a zero byte content size. NiFi does not support the replay of flowfiles that are zero bytes in size. (A Jira is being entered for this as i see replay of zero byte file scan have a valid use case at times) If you did not configure "Put Response Body In Attribute" property, a new FlowFile would have been generated where the response becomes the content and the FlowFile is routed to the "response" relationship.

NiFi cannot replay files a creation time in the flow. The way replay works, Flowfiles are reinserted on the connection feeding the processor that produced the event. In cases where the processor producing the event actually created the Flowfile, there is no where to reinsert that claim for replay. You should however be able to replay that file at the next processor that produced an provenance event. If that replay messgae is generated at a later in line processing event, it indicates that the content no longer exist in the content repos archive.  Typically this is because the retention duration configured in the nifi.properties file has been exceeded for this content, but it could also be caused by other factors such as Content repo has exceeded the configured allowable disk utilization threshold percentage (also configured in nifi.properties file) or the content was manually deleted from repo (less likely).  Queued active data in the flow takes precedence over archive data retention, so if you have a lot of queued data in your flow, you may not have an archived data at all because of the max disk utilization percentage configured for your NiFi.

[#7]

@shannon luo

For your first question, the relationships for InvokeHttp are a bit confusing right now but the original flowfile (that came into the processor) will be routed to the "Original" relationship when the status indicates a success. The "Response" relationship, assuming you have the default configuration, will have the server's actual response. This was done so the user has the capability to use both the original flowfile and the server's response later in the flow.

[#8]

@jpercivall and @mclark

Thank you both! I see the Response actually contains the content. Thanks much!

.

[#9]

@shannon luo

The FlowFile routed to 'Response' will have the body of the 'Get' response in the content of the FlowFile. You should be able to download the content or view the bytes using provenance. Depending on what the type is of the content you may also be able to view it properly in the viewer.