using HandleHttpRequests processor in nifi and extract query parameters

[#1]

I need to have a URL that would be a listener to a web hook.

I will register a URL with them and they will be sending GET requests that look like this

Fri Aug 11 17:24:13 IST 2017:Code:405, URL:http://serverURL:port/gushupsmslistener?externalId=3380112592180823371-525731073514300237&deliveredT..., timetaken:577, EntityId:1014907793, CauseId:3380112592180823371, InternalId:525731073514300237, PhoneNo:919971295965, globalErrCode:000

What is the best way in Nifi to get data from a GET request?

Edit: I've set up HandleHttpRequests processor and started it. Gave it an allowed path and the port.

Now, I need to get whatever values are there in the query parameters and store those in HDFS. HOw do I extract and store them?

If I directly point handleHTTPrequest to putHDFS, It creates an empty file.

Besides, what exactly is upstream connection in handlehttpresponse and how do I set it up?

[#2]

The FlowFiles produced by HandleHttpRequest will have the content of the HTTP request, since your case is a GET request with no content, that is why the FlowFile is empty.

The FlowFiles should have an attribute called "http.query.string" which should contain the query params portion of the URL. You should be able to see this by looking in provenance, or by using a LogAttribute processor.

If you want to store the query params in HDFS you need to get them into the content of the flow file. You can do this using a ReplaceText processor, and set the Replacement Value to ${http.query.string}.

HandleHttpRequest is meant to be used with HandleHttpResponse to create a web server. So you would do something like:

HandleHttpRequest -> ReplaceText -> PutHDFS

From PutHDFS you would connect the success relationship to a HandleHttpResponse that sent a 200 response, and connect the failure relationship to a different HandleHttpResponse that sent a 500 response. This way the client knows if the request succeeded or not.

If you don't care about the client knowing about this, you could just have the success relation of HandleHttpRequest go right to a HandleHttpResponse and always return a 200, then also send success to the rest of the flow (ReplaceText, PutHDFS, etc). In this case the client will always see it as successful even if the write to HDFS failed.

[#3]

Bryan,

Is there a way for HandleHttpRequest to receive incoming files and save them locally (where ever that NiFi instance is running for example)?

I want to send images to my NiFi instance through a POST request (using curl), and I want the image to be saved in a certain directory where that NiFi instance is running.

[#4]

Thanks Bryan. So, I get the query parameters in the exam same format now but I would rather have them saved in JSON structure in files. Is there a way to do that as well? @Bryan Bende

[#5]

There isn't an out-of-the-box way, your options would be....

1) Use UpdateAttribute + expression language functions to parse out each parameter from the original query string into its own attribute, then use ReplaceText with a templated JSON where you reference the attribute names and values like:

{ ${attr1} : ${attr1.value} }

Assuming you already create attr1 and attr1.value using UpdateAttribute.

2) Write a script that reads the query params from the flow file, parses them, and writes them back out as json, and then put this script into an ExecuteScript processor.

3) Write a custom Java processor that does the same as #2.

[#6]

@Bryan Bende :

I have a workflow like HandleHttpRequest -> InvokeHttp -> Custom Processor -> HandleHttpResponse.

My requirement is that I want to support multiple templated jsons and URL sent to HandleHttpRequest will have query params who values should replace some values in the templated jsons based on the template I need. These templated jsons are nested and complex in structure. Pleaseee suggest how to do this. Different posts mention about diff processors like EvaluateJsonPath, ReplaceText, UpdateAttribute and so on but which processor should I use to store my templated json and how can I do the values replacement. I am new to NiFi and request your help on this and I am basically from Java background.