Silver ⭐️

Solved

Request for Support with Passing Request Body in Invoke HTTP Processors

5 replies
9764 views
Translate All

Hi everyone,

I'm currently encountering an issue related to passing request bodies in my workflow. Here's a breakdown of the situation:

In my workflow, I have a total of 9 processors, all of which are InvokeHTTP Processors utilizing post requests. For each post request, I need to include a request body.

The challenge lies in figuring out how to effectively pass the body using the InvokeHTTP processor. Presently, I am unable to accomplish this by utilizing the InvokeHTTP processor directly. We've attempted using GenerateFlowFile to pass the body, but it cannot be seamlessly integrated into the workflow. Specifically, I am unable to insert GenerateFlowFile between Processor 3 and Processor 4, for example.

One potential solution we've considered is creating individual GenerateFlowFile instances for each processor outside the workflow. However, this approach introduces complications, especially when one processor is dependent on the output of a previous processor. For instance, if Processor 5 relies on the output of Processor 4, I'd need to introduce yet another processor.

I'm reaching out to the community for insights and suggestions on how to efficiently handle passing request bodies in this scenario. Any advice or alternative approaches would be greatly appreciated.

Thank you!

Apache NiFi

SAMSALAnswer

Platinum ⭐️⭐️⭐️

@apache_nifi,

Thanks for providing more details. Have you tried what I proposed in my first response? I still think that using ReplaceText would help you create the needed request for each consecutive api call. You can use something like EvaluageJsonPath or ExtractText processors to extract any parameters from the latest response and store as flowfile attributes, which you can use when constructing the next request . For example I can see that step 2 relies on the output of step 1, so after you get the idp token you can use the ExtractText processor to save the token into attribute (by adding dynamic property) as follows:

This will give you the same flowfile (idp token) but it will have new attribute idp_token:

Then you can use ReplaceText to construct the next request body as follows:

The Replacement Value will have the x-www-form-urlencoded parameters:

client_id=CLIENT_ID&grant_type=urn:GRANT_TYPE&company_id=COMPANT_ID&new_token=true&assertion=${idp_token}

Notice how Im using the expression language ${idp_token} to set the assertion parameter. This will generate new flowfile for step 2 invokehttp which will serve as the request body. Keep in mind any request body has to be passed as flowfile to the InvokeHttp processor.

For more information on how to do x-www-form-urlencoded request using invokehttp:

https://community.cloudera.com/t5/Support-Questions/NIFI-Is-it-possible-to-make-a-x-www-form-urlencoded-POST/m-p/339398

Hopefully you have enough to get going. Let me know if you have any particular questions.

If that helps please accept solution.

Thanks

DianaTorres

Community Manager

@apache_nifi Welcome to the Cloudera Community!

To help you get the best possible solution, I have tagged our NiFi experts @MattWho @SAMSAL who may be able to assist you further.

Please keep us updated on your post, and we hope you find a satisfactory solution to your query.

Translate

Regards, Diana Torres, Senior Community Moderator

SAMSAL

Platinum ⭐️⭐️⭐️

Hi @apache_nifi ,

Its difficult to give you an exact answer without seeing and understanding the whole scenario. For example how those 9 InvokeHttp processors are related and what kind of request body you need to construct for each POST. If you cant list all the API's and the request body each needs and how are they dependent, maybe you can create an example based on hypothetical situation. However, to give you some direction that might help in your case, there are a lot of processors that you can use in between to help construct the next request body, for example lets say that the first invokeHttp returns a json response and you need some parameters from this response to pass to the next InvokeHttp, then you can use processor like EvaluateJsonPath to get the parameters into flowfile attributes then use ReplaceText processor to replace the json response flowfile content with the new request body and utilize the flowfile attributes you extracted from the json response to populate the needed parameters in the replace value. To demonstrate:

Lets assume I want to use Nifi Api to start a processor and run it once , the first invokehttp will be to get the processor information using the processor Id. The response of this invokation will be something like this:

{
    "revision": {
	    "clientId": "value",
	    "version": 0,
	    "lastModifier": "value"
	},
    "id": "value",
    "uri": "value",
    "position": {…},
    "permissions": {…},
    "bulletins": [{…}],
    "disconnectedNodeAcknowledged": true,
    "component": {…},
    "inputRequirement": "value",
    "status": {…},
    "operatePermissions": {…}
}

To change the run state of processor I need to pass the version number alongside the processor id in the request body. To get the version number (0) from the response above and store it as an attribute I use EvaluateJsonPath like this:

Then I use ReplaceText to replace the response content with the run state api request body :

In the Search Value I'm using the regex "(?s)(^.*$)" that captures everything and the replacement value is the request body for the next invokehttp:

{
    "revision": {
	    "clientId": "${pipeline.start.processor.id}",
	    "version": ${processorVersion}
	   
	},
    "state": "RUN_ONCE",
    "disconnectedNodeAcknowledged": true
	
}

Notice how I'm utilizing the flowfile attributes using expression language to populate the required parameters like processor id and processor version which I extracted using the EvaluateJsonPath above. Next will will my InvokeHttp passing the new request body content.

Hope that helps put you on the right direction.

If this helps please accept solution.

Thanks

Translate

apache_nifiAuthor

Silver ⭐️

Okay, Let me explain the whole scenario.

The following are the steps to build job descriptions using the Job Profile Builder.

Step 1 => create an idp token and its curl is as below

curl --location '----- URL -----' \

--header 'Content-Type: application/x-www-form-urlencoded' \

--data 'client_id=----CLIENT_ID----&token_url=----TOKEN_URL----&user_id=----USER_ID----&private_key=----PRIVATE_KEY----='

and its output => kmf0P2sW00eJnHS9gQ1bF9vohjG4N4X9QgJ2rbsZ3H4HlvvSiLV8HmbVxoHFYR8CbYHq92cWFRzam4zJkBYI6CkA0cNtmkUzcGza1I07zT2zwcjBnUDl9nZkaQeUHjhwfaXjKaZnzh0zQnZiYmeP6OcWPCmQWjkG7aFwzsC8N=

Step 2 => create an access token

curl --location '----- URL -----' \

--header 'Content-Type: application/x-www-form-urlencoded' \

--data-urlencode 'client_id=----CLIENT_ID----' \

--data-urlencode 'grant_type=urn:----GRANT_TYPE----' \

--data-urlencode 'company_id=----COMPANT_ID----' \

--data-urlencode 'new_token=true' \

--data-urlencode 'assertion=kmf0P2sW00eJnHS9gQ1bF9vohjG4N4X9QgJ2rbsZ3H4HlvvSiLV8HmbVxoHFYR8CbYHq92cWFRzam4zJkBYI6CkA0cNtmkUzcGza1I07zT2zwcjBnUDl9nZkaQeUHjhwfaXjKaZnzh0zQnZiYmeP6OcWPCmQWjkG7aFwzsC8N='

Step 3 => Create a Family and Role (post api)

Step 4 => Assign a Job Code to the Role (post api)

Step 5 => Create skills (post api)

Step 6 => Map Skills to the Role (post api)

Step 7 => Create a Job Profile Template (JobDescTemplate) (post api)

Step 8 => Map the Job Profile Template with Job Families (JDTemplateFamilyMapping) (post api)

Step 9 => Create a Job Code (post api)

Step 10 => Create a Position and assign the Job (post api)

Step 11 => Create a Job Profile (post api)

From step 1 to step 11, those are POST requests and every POST request, I have to request pass body too. So I have to use InvokeHTTP processor..

In simpler terms, sometimes I might use information from the previous step as input for the next step. For example, if I receive data in step 3 that includes a "family_id," I may want to use that specific family_id in step 4. This means that what happens in step 4 depends on what happened in step 3.

Now my problem is that I'm not able to add body inside InvokeHTTP processsor for POST request.

NOTE : my Apache NiFi version is => nifi-2.0.0-M1

Translate

SAMSALAnswer

Platinum ⭐️⭐️⭐️

@apache_nifi,

Thanks for providing more details. Have you tried what I proposed in my first response? I still think that using ReplaceText would help you create the needed request for each consecutive api call. You can use something like EvaluageJsonPath or ExtractText processors to extract any parameters from the latest response and store as flowfile attributes, which you can use when constructing the next request . For example I can see that step 2 relies on the output of step 1, so after you get the idp token you can use the ExtractText processor to save the token into attribute (by adding dynamic property) as follows:

This will give you the same flowfile (idp token) but it will have new attribute idp_token:

Then you can use ReplaceText to construct the next request body as follows:

The Replacement Value will have the x-www-form-urlencoded parameters:

client_id=CLIENT_ID&grant_type=urn:GRANT_TYPE&company_id=COMPANT_ID&new_token=true&assertion=${idp_token}

Notice how Im using the expression language ${idp_token} to set the assertion parameter. This will generate new flowfile for step 2 invokehttp which will serve as the request body. Keep in mind any request body has to be passed as flowfile to the InvokeHttp processor.

For more information on how to do x-www-form-urlencoded request using invokehttp:

https://community.cloudera.com/t5/Support-Questions/NIFI-Is-it-possible-to-make-a-x-www-form-urlencoded-POST/m-p/339398

Hopefully you have enough to get going. Let me know if you have any particular questions.

If that helps please accept solution.

Thanks

Translate

joseomjr

Gold ⭐️⭐️

Agree with @SAMSAL's approach and if you can provide a parameter or something in the header or request so your API returns a JSON response each time it'll make things a lot easier for you to parse and build the request for the next step in your flow.

Translate

Sign up

Please use your Cloudera account.

Login to the community

Please use your Cloudera account.