Gold ⭐️⭐️

Solved

NiFi Execute Stream Command Curl Call

11 replies
18317 views
Translate All

I am trying to use NiFi to make a CURL call to SMM. Because SMM is using kerberos authentication I want to make a POST request to the SMM API using Kerberos. I am now at the last step where I need to sent a POST request to create a Kafka topic.

I am having a problem where when I send the request using the ExecuteStreamCommand I think some formatting happens to the payload before the call gets sent to the SMM API. I tried escaping the quotes in the payload but still no luck.

Here are my configurations for the ExecuteStreamCommand:

Command Arguments Strategy: Command Arguments Property

Command Arguments: -X POST -H referer:${Referer} -H 'Content-Type: application/json' -d '{\"newTopics\": [{\"name\":\"testing123\",\"numPartitions\":3,\"replicationFactor\":3}], \"allTopicNames\":[\"testing123\"]}' --negotiate -u : -b /tmp/cookiejar.txt -c /tmp/cookiejar.txt http://SMM-HOSTNAME:8585/api/v1/admin/topics

Command Path: curl

Ignore STDIN: false

Argument Delimiter: " " (Space)

Right now it's giving me a 415 - Unsupported Media Type Error.

Can someone please help.

MattWhoAnswer

Titanium ⭐️⭐️⭐️

@drewski7
The removal of quotes from the "command arguments" is expected behavior in the ExecuteStreamCommand processor. This processor was introduced to NiFi more than 10 years ago and was originally designed for a more minimal scope of work including the expectation that FlowFile content would be passed to the script/command being executed.

As time passed on the use cases that were trying to be solutioned via the ExecuteStreamCommand expanded; however, handling those use case would potential break users already implemented and working dataflow. So rather then change that default behavior, a new property "Command Arguments Strategy" was added with the original "Command Arguments Property" as the default (legacy method) and a new "Dynamic property arguments" option.

This change is part of this JIra and implemented as of Apache NiFi 1.10:
https://issues.apache.org/jira/browse/NIFI-3221

In your use case, you'll want to switch to using the "Dynamic property arguments". This will then require you to click on the "+" to add a new dynamic property. The property names MUST use this format:

command.argument.<num>

So in your case you might try something like:

command.argument.1 = -X POST -H referer:${Referer} -H 'Content-Type: application/json' -d  '{"newTopics": [{"name":"testing123","numPartitions":3,"replicationFactor":3}], "allTopicNames":["testing123"]}' --negotiate -u : -b /tmp/cookiejar.txt -c /tmp/cookiejar.txt http://SMM-HOSTNAME:8585/api/v1/admin/topics

If you found that the provided solution(s) assisted you with your query, please take a moment to login and click Accept as Solution below each response that helped.

Thank you,

Matt

cotopaul

Platinum ⭐️

Well I am not quite certain what SMM is and what sort of API Calls it accepts, but as far as it goes to the received answer, it could have an endless list of root causes. Error 415 - Unsupported Media type indicates that your server refused the accept that request, due to the payload format. This problem might be due to the content type or the content encoding, as well as the data directly.

You could first of all try from a linux machine and see if that curl command truly works.

Another option would be to use Postman to test the API command and see the results.

Last but not least, try maybe using InvokeHTTP instead of ExecuteStreamCommand, as InvokeHTTP is more tailored for such actions.

InvokeHTTP: https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.20.0/org.apache.nifi.processors.standard.InvokeHTTP/index.html

Translate

drewski7Author

Gold ⭐️⭐️

@cotopaul

I have verified that the same command works from the command line. I think the executeStreamCommand must be doing some sort of formatting prior to sending the request to the Streams Message Manager Server.

A GET request works using the ExecuteStreamCommand because there is no payload.

I can't use the invokeHTTP because it currently doesn't have configuration to authenticate using kerberos which is why I fell back on using ExecuteStreamCommand.

Translate

cotopaul

Platinum ⭐️

Well to be honest it is pretty hard to debug something you have no clue about 😄
From your command I see that you are using the value for "referer" as a variable taken out of an attribute. I assume that the value is correct.

Now, based on my experience with ExecuteStreamCommand, I would give each parameter after a semi colon.... something like: ( it is worth a shot, otherwise you will need to test each parameter until you find the faulty one)
-X;POST;-H;referer:${Referer};-H;'Content-Type: application/json';-d;'{\"newTopics\": [{\"name\":\"testing123\",\"numPartitions\":3,\"replicationFactor\":3}], \"allTopicNames\":[\"testing123\"]}';--negotiate;-u;:;-b;/tmp/cookiejar.txt;-c;/tmp/cookiejar.txt

Translate

drewski7Author

Gold ⭐️⭐️

@cotopaul

The reason why I say that is because if I change the command arguments to this -

-X GET -H referer:${Referer} -H 'Content-Type: application/json' --negotiate -u : -b /tmp/cookiejar.txt -c /tmp/cookiejar.txt http://SMM-HOSTNAME:8585/api/v1/admin/brokers

It returns successfully.

The only that changed is the HTTP Request Method and removing the Payload.

Translate

cotopaul

Platinum ⭐️

Well, besides the fact that you are calling a different API endpoint as you previously did, yea, it could be that the payload is the one messing up your command.

What I would try is to use an UpdateAttribute to generate the curl command as an atrribute and see how it gets constructed. I would then take the command and execute it on my server to see if it works — if it gets constructed correctly. Otherwise, you will see where the error is located and you can try and solve it.

Translate

drewski7Author

Gold ⭐️⭐️

Sorry typo on my part. Even with the */api/v1/admin/topics it works.

I cloned the nifi github repo and added some more log statements to the executeStreamProcessor changed the edited the logback.xml to enable DEBUG for that processor. It looks like it stripping the quotes in payload.

Here is an example request: curl -X POST -H 'Content-Type:application/json' -d '{"name": "Apple AirPods", "data": {"color": "white", "generation": "3rd", "price": 135}}' https://api.restful-api.dev/objects

This works in the command line.

If you configure the executeStreamCommand like this:

Command Arguments Strategy: Command Arguments Property

Command Arguments: -X POST -H 'Content-Type:application/json' -d '{\"name\":\"Apple AirPods\",\"data\":{\"color\":\"white\",\"generation\":\"3rd\",\"price\":135}}' https://api.restful-api.dev/objects

Command Path: curl

Argument Delimiter: " " (Space)

Here's what I see in the logs -

Executing and waiting for command: curl, with the arguments: [-X, POST, -H, 'Content-Type:application/json', -d, '{\name\:\Apple AirPods\,\data\:{\color\:\white\,\generation\:\3rd\,\price\:135}}', https://api.restful-api.dev/objects]

@cotopaul , @ckumar , @MattWho , @steven-matison - Is there a way to get around this?

Translate

Kefkoe

Silver ⭐️

if it works in the command line, you might want to make the script and just pass the variable to it?

curl -X POST -H 'Content-Type:application/json' -d '{"name": "Apple AirPods", "data": {"color": "white", "generation": "3rd", "price": 135}}' https://api.restful-api.dev/objects

If this works. You can make the script like:

curl -X POST -H 'Content-Type:application/json' -d '{"name": "$1", "data": {"color": "$2", "generation": "$3", "price": $4}}' https://api.restful-api.dev/objects

then just pass the variables in the Command Arguments:

Variable1;Variable2;Variable3;Variable4

Command Path: path to the script

Note that for windows it's %1, %, ... 2 and not $1, $2, ... for variables.

Translate

drewski7Author

Gold ⭐️⭐️

@Kefkoe

I really wanted to try and keep it inline and directly in NiFi itself. That way there is no external dependencies to put the script on each node within our NiFi cluster. Good idea as a last resort.

Translate

drewski7Author

Gold ⭐️⭐️

Any updates here?

Translate

MattWhoAnswer

Titanium ⭐️⭐️⭐️

@drewski7
The removal of quotes from the "command arguments" is expected behavior in the ExecuteStreamCommand processor. This processor was introduced to NiFi more than 10 years ago and was originally designed for a more minimal scope of work including the expectation that FlowFile content would be passed to the script/command being executed.

As time passed on the use cases that were trying to be solutioned via the ExecuteStreamCommand expanded; however, handling those use case would potential break users already implemented and working dataflow. So rather then change that default behavior, a new property "Command Arguments Strategy" was added with the original "Command Arguments Property" as the default (legacy method) and a new "Dynamic property arguments" option.

This change is part of this JIra and implemented as of Apache NiFi 1.10:
https://issues.apache.org/jira/browse/NIFI-3221

In your use case, you'll want to switch to using the "Dynamic property arguments". This will then require you to click on the "+" to add a new dynamic property. The property names MUST use this format:

command.argument.<num>

So in your case you might try something like:

command.argument.1 = -X POST -H referer:${Referer} -H 'Content-Type: application/json' -d  '{"newTopics": [{"name":"testing123","numPartitions":3,"replicationFactor":3}], "allTopicNames":["testing123"]}' --negotiate -u : -b /tmp/cookiejar.txt -c /tmp/cookiejar.txt http://SMM-HOSTNAME:8585/api/v1/admin/topics

If you found that the provided solution(s) assisted you with your query, please take a moment to login and click Accept as Solution below each response that helped.

Thank you,

Matt

Translate

Sign up

Please use your Cloudera account.

Login to the community

Please use your Cloudera account.