Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

NiFi Execute Stream Command Curl Call

avatar
Contributor

I am trying to use NiFi to make a CURL call to SMM. Because SMM is using kerberos authentication I want to make a POST request to the SMM API using Kerberos. I am now at the last step where I need to sent a POST request to create a Kafka topic. 

 

I am having a problem where when I send the request using the ExecuteStreamCommand I think some formatting happens to the payload before the call gets sent to the SMM API. I tried escaping the quotes in the payload but still no luck.

 

Here are my configurations for the ExecuteStreamCommand:

Command Arguments Strategy: Command Arguments Property

Command Arguments: -X POST -H referer:${Referer} -H 'Content-Type: application/json' -d  '{\"newTopics\": [{\"name\":\"testing123\",\"numPartitions\":3,\"replicationFactor\":3}], \"allTopicNames\":[\"testing123\"]}' --negotiate -u : -b /tmp/cookiejar.txt -c /tmp/cookiejar.txt http://SMM-HOSTNAME:8585/api/v1/admin/topics

Command Path: curl

Ignore STDIN: false

Argument Delimiter: " " (Space)

 

Right now it's giving me a 415 - Unsupported Media Type Error.

 

Can someone please help.

 

1 ACCEPTED SOLUTION

avatar
Super Mentor

@drewski7 
The removal of quotes from the "command arguments" is expected behavior in the ExecuteStreamCommand processor.  This processor was introduced to NiFi more than 10 years ago and was originally designed for a more minimal scope of work including the expectation that FlowFile content would be passed to the script/command being executed.  

As time passed on the use cases that were trying to be solutioned via the ExecuteStreamCommand expanded; however, handling those use case would potential break users already implemented and working dataflow.  So rather then change that default behavior, a new property "Command Arguments Strategy" was added with the original "Command Arguments Property" as the default (legacy method) and a new "Dynamic property arguments" option.

This change is part of this JIra and implemented as of Apache NiFi 1.10:
https://issues.apache.org/jira/browse/NIFI-3221


In your use case, you'll want to switch to using the "Dynamic property arguments".  This will then require you to click on the "+" to add a new dynamic property.  The property names MUST use this format:

command.argument.<num>

So in your case you might try something like:

command.argument.1 = -X POST -H referer:${Referer} -H 'Content-Type: application/json' -d  '{"newTopics": [{"name":"testing123","numPartitions":3,"replicationFactor":3}], "allTopicNames":["testing123"]}' --negotiate -u : -b /tmp/cookiejar.txt -c /tmp/cookiejar.txt http://SMM-HOSTNAME:8585/api/v1/admin/topics

 

If you found that the provided solution(s) assisted you with your query, please take a moment to login and click Accept as Solution below each response that helped.

Thank you,

Matt

 

 

View solution in original post

11 REPLIES 11

avatar

Well I am not quite certain what SMM is and what sort of API Calls it accepts, but as far as it goes to the received answer, it could have an endless list of root causes. Error 415 - Unsupported Media type indicates that your server refused the accept that request, due to the payload format. This problem might be due to the content type or the content encoding, as well as the data directly.


You could first of all try from a linux machine and see if that curl command truly works.

Another option would be to use Postman to test the API command and see the results.

Last but not least, try maybe using InvokeHTTP instead of ExecuteStreamCommand, as InvokeHTTP is more tailored for such actions.

InvokeHTTP: https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.20.0/org.apach...

avatar
Contributor

@cotopaul 

I have verified that the same command works from the command line. I think the executeStreamCommand must be doing some sort of formatting prior to sending the request to the Streams Message Manager Server. 

 

A GET request works using the ExecuteStreamCommand because there is no payload. 


I can't use the invokeHTTP because it currently doesn't have configuration to authenticate using kerberos which is why I fell back on using ExecuteStreamCommand. 

 

avatar

Well to be honest it is pretty hard to debug something you have no clue about 😄
From your command I see that you are using the value for "referer" as a variable taken out of an attribute. I assume that the value is correct.

Now, based on my experience with ExecuteStreamCommand, I would give each parameter after a semi colon.... something like: ( it is worth a shot, otherwise you will need to test each parameter until you find the faulty one)
-X;POST;-H;referer:${Referer};-H;'Content-Type: application/json';-d;'{\"newTopics\": [{\"name\":\"testing123\",\"numPartitions\":3,\"replicationFactor\":3}], \"allTopicNames\":[\"testing123\"]}';--negotiate;-u;:;-b;/tmp/cookiejar.txt;-c;/tmp/cookiejar.txt

avatar
Contributor

@cotopaul 

The reason why I say that is because if I change the command arguments to this - 

-X GET -H referer:${Referer} -H 'Content-Type: application/json' --negotiate -u : -b /tmp/cookiejar.txt -c /tmp/cookiejar.txt http://SMM-HOSTNAME:8585/api/v1/admin/brokers

 

It returns successfully.


The only that changed is the HTTP Request Method and removing the Payload. 

avatar

Well, besides the fact that you are calling a different API endpoint as you previously did, yea, it could be that the payload is the one messing up your command.

 

What I would try is to use an UpdateAttribute to generate the curl command as an atrribute and see how it gets constructed. I would then take the command and execute it on my server to see if it works — if it gets constructed correctly. Otherwise, you will see where the error is located and you can try and solve it.

avatar
Contributor

Sorry typo on my part. Even with the */api/v1/admin/topics it works.

 

I cloned the nifi github repo and added some more log statements to the executeStreamProcessor changed the edited the logback.xml to enable DEBUG for that processor. It looks like it stripping the quotes in payload.

 

Here is an example request: curl -X POST -H 'Content-Type:application/json' -d '{"name": "Apple AirPods", "data": {"color": "white", "generation": "3rd", "price": 135}}' https://api.restful-api.dev/objects

 

This works in the command line.

 

If you configure the executeStreamCommand like this:

Command Arguments Strategy: Command Arguments Property

Command Arguments: -X POST -H 'Content-Type:application/json' -d '{\"name\":\"Apple AirPods\",\"data\":{\"color\":\"white\",\"generation\":\"3rd\",\"price\":135}}' https://api.restful-api.dev/objects

Command Path: curl

Argument Delimiter: " " (Space)

 

Here's what I see in the logs - 

Executing and waiting for command: curl, with the arguments: [-X, POST, -H, 'Content-Type:application/json', -d, '{\name\:\Apple AirPods\,\data\:{\color\:\white\,\generation\:\3rd\,\price\:135}}', https://api.restful-api.dev/objects]

 

@cotopaul , @ckumar , @MattWho , @steven-matison - Is there a way to get around this?

avatar

To be really honest, I do not know how you could not remove those double quotes. Maybe you can try escaping the quotes with a backslash and also wrap the entire command in single quotes.

Something like: (I might have added a few extra single quotes by mistake but you get the point)

'-X POST -H '\''Content-Type:application/json'\'' -d '\''{"name": "Apple AirPods", "data": {"color": "white", "generation": "3rd", "price": 135}}'\'' https://api.restful-api.dev/objects'

In theory, the double quotes around the JSON string are escaped with a backslash and the entire command is wrapped in single quotes. The single quotes allow the command to be passed to the ExecuteStreamCommand processor as a single argument, which ensures that the quotes are preserved.

PS: I did not test this and I am not 100% it will work, but at least it is worth a shot until somebody with more experience can guide your further.

avatar
New Contributor

if it works in the command line, you might want to make the script and just pass the variable to it?

curl -X POST -H 'Content-Type:application/json' -d '{"name": "Apple AirPods", "data": {"color": "white", "generation": "3rd", "price": 135}}' https://api.restful-api.dev/objects

If this works. You can make the script like:

curl -X POST -H 'Content-Type:application/json' -d '{"name": "$1", "data": {"color": "$2", "generation": "$3", "price": $4}}' https://api.restful-api.dev/objects

then just pass the variables in the Command Arguments:

Variable1;Variable2;Variable3;Variable4

Command Path: path to the script 

Note that for windows it's %1, %, ... 2 and not $1, $2, ... for variables.
 

avatar
Contributor

@Kefkoe 

I really wanted to try and keep it inline and directly in NiFi itself. That way there is no external dependencies to put the script on each node within our NiFi cluster. Good idea as a last resort.