Created 12-27-2023 05:17 AM
Hi everyone,
I'm currently encountering an issue related to passing request bodies in my workflow. Here's a breakdown of the situation:
In my workflow, I have a total of 9 processors, all of which are InvokeHTTP Processors utilizing post requests. For each post request, I need to include a request body.
The challenge lies in figuring out how to effectively pass the body using the InvokeHTTP processor. Presently, I am unable to accomplish this by utilizing the InvokeHTTP processor directly. We've attempted using GenerateFlowFile to pass the body, but it cannot be seamlessly integrated into the workflow. Specifically, I am unable to insert GenerateFlowFile between Processor 3 and Processor 4, for example.
One potential solution we've considered is creating individual GenerateFlowFile instances for each processor outside the workflow. However, this approach introduces complications, especially when one processor is dependent on the output of a previous processor. For instance, if Processor 5 relies on the output of Processor 4, I'd need to introduce yet another processor.
I'm reaching out to the community for insights and suggestions on how to efficiently handle passing request bodies in this scenario. Any advice or alternative approaches would be greatly appreciated.
Thank you!
Created on 12-28-2023 12:53 PM - edited 12-28-2023 12:54 PM
Thanks for providing more details. Have you tried what I proposed in my first response? I still think that using ReplaceText would help you create the needed request for each consecutive api call. You can use something like EvaluageJsonPath or ExtractText processors to extract any parameters from the latest response and store as flowfile attributes, which you can use when constructing the next request . For example I can see that step 2 relies on the output of step 1, so after you get the idp token you can use the ExtractText processor to save the token into attribute (by adding dynamic property) as follows:
This will give you the same flowfile (idp token) but it will have new attribute idp_token:
Then you can use ReplaceText to construct the next request body as follows:
The Replacement Value will have the x-www-form-urlencoded parameters:
client_id=CLIENT_ID&grant_type=urn:GRANT_TYPE&company_id=COMPANT_ID&new_token=true&assertion=${idp_token}
Notice how Im using the expression language ${idp_token} to set the assertion parameter. This will generate new flowfile for step 2 invokehttp which will serve as the request body. Keep in mind any request body has to be passed as flowfile to the InvokeHttp processor.
For more information on how to do x-www-form-urlencoded request using invokehttp:
Hopefully you have enough to get going. Let me know if you have any particular questions.
If that helps please accept solution.
Thanks
Created 12-27-2023 06:11 AM
@apache_nifi Welcome to the Cloudera Community!
To help you get the best possible solution, I have tagged our NiFi experts @MattWho @SAMSAL who may be able to assist you further.
Please keep us updated on your post, and we hope you find a satisfactory solution to your query.
Regards,
Diana Torres,Created 12-27-2023 07:36 AM
Hi @apache_nifi ,
Its difficult to give you an exact answer without seeing and understanding the whole scenario. For example how those 9 InvokeHttp processors are related and what kind of request body you need to construct for each POST. If you cant list all the API's and the request body each needs and how are they dependent, maybe you can create an example based on hypothetical situation. However, to give you some direction that might help in your case, there are a lot of processors that you can use in between to help construct the next request body, for example lets say that the first invokeHttp returns a json response and you need some parameters from this response to pass to the next InvokeHttp, then you can use processor like EvaluateJsonPath to get the parameters into flowfile attributes then use ReplaceText processor to replace the json response flowfile content with the new request body and utilize the flowfile attributes you extracted from the json response to populate the needed parameters in the replace value. To demonstrate:
Lets assume I want to use Nifi Api to start a processor and run it once , the first invokehttp will be to get the processor information using the processor Id. The response of this invokation will be something like this:
{
"revision": {
"clientId": "value",
"version": 0,
"lastModifier": "value"
},
"id": "value",
"uri": "value",
"position": {…},
"permissions": {…},
"bulletins": [{…}],
"disconnectedNodeAcknowledged": true,
"component": {…},
"inputRequirement": "value",
"status": {…},
"operatePermissions": {…}
}
To change the run state of processor I need to pass the version number alongside the processor id in the request body. To get the version number (0) from the response above and store it as an attribute I use EvaluateJsonPath like this:
Then I use ReplaceText to replace the response content with the run state api request body :
In the Search Value I'm using the regex "(?s)(^.*$)" that captures everything and the replacement value is the request body for the next invokehttp:
{
"revision": {
"clientId": "${pipeline.start.processor.id}",
"version": ${processorVersion}
},
"state": "RUN_ONCE",
"disconnectedNodeAcknowledged": true
}
Notice how I'm utilizing the flowfile attributes using expression language to populate the required parameters like processor id and processor version which I extracted using the EvaluateJsonPath above. Next will will my InvokeHttp passing the new request body content.
Hope that helps put you on the right direction.
If this helps please accept solution.
Thanks
Created 12-27-2023 09:38 PM
Created on 12-28-2023 12:53 PM - edited 12-28-2023 12:54 PM
Thanks for providing more details. Have you tried what I proposed in my first response? I still think that using ReplaceText would help you create the needed request for each consecutive api call. You can use something like EvaluageJsonPath or ExtractText processors to extract any parameters from the latest response and store as flowfile attributes, which you can use when constructing the next request . For example I can see that step 2 relies on the output of step 1, so after you get the idp token you can use the ExtractText processor to save the token into attribute (by adding dynamic property) as follows:
This will give you the same flowfile (idp token) but it will have new attribute idp_token:
Then you can use ReplaceText to construct the next request body as follows:
The Replacement Value will have the x-www-form-urlencoded parameters:
client_id=CLIENT_ID&grant_type=urn:GRANT_TYPE&company_id=COMPANT_ID&new_token=true&assertion=${idp_token}
Notice how Im using the expression language ${idp_token} to set the assertion parameter. This will generate new flowfile for step 2 invokehttp which will serve as the request body. Keep in mind any request body has to be passed as flowfile to the InvokeHttp processor.
For more information on how to do x-www-form-urlencoded request using invokehttp:
Hopefully you have enough to get going. Let me know if you have any particular questions.
If that helps please accept solution.
Thanks
Created 12-28-2023 01:17 PM
Agree with @SAMSAL's approach and if you can provide a parameter or something in the header or request so your API returns a JSON response each time it'll make things a lot easier for you to parse and build the request for the next step in your flow.