NiFi Execute Stream Command Curl Call

[#1]

I am trying to use NiFi to make a CURL call to SMM. Because SMM is using kerberos authentication I want to make a POST request to the SMM API using Kerberos. I am now at the last step where I need to sent a POST request to create a Kafka topic. 

I am having a problem where when I send the request using the ExecuteStreamCommand I think some formatting happens to the payload before the call gets sent to the SMM API. I tried escaping the quotes in the payload but still no luck.

Here are my configurations for the ExecuteStreamCommand:

Command Arguments Strategy: Command Arguments Property

Command Arguments: -X POST -H referer:${Referer} -H 'Content-Type: application/json' -d  '{\"newTopics\": [{\"name\":\"testing123\",\"numPartitions\":3,\"replicationFactor\":3}], \"allTopicNames\":[\"testing123\"]}' --negotiate -u : -b /tmp/cookiejar.txt -c /tmp/cookiejar.txt http://SMM-HOSTNAME:8585/api/v1/admin/topics

Command Path: curl

Ignore STDIN: false

Argument Delimiter: " " (Space)

Right now it's giving me a 415 - Unsupported Media Type Error.

Can someone please help.

[#2]

Well I am not quite certain what SMM is and what sort of API Calls it accepts, but as far as it goes to the received answer, it could have an endless list of root causes. Error 415 - Unsupported Media type indicates that your server refused the accept that request, due to the payload format. This problem might be due to the content type or the content encoding, as well as the data directly.

You could first of all try from a linux machine and see if that curl command truly works.

Another option would be to use Postman to test the API command and see the results.

Last but not least, try maybe using InvokeHTTP instead of ExecuteStreamCommand, as InvokeHTTP is more tailored for such actions.

InvokeHTTP: https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.20.0/org.apach...

[#3]

@cotopaul 

I have verified that the same command works from the command line. I think the executeStreamCommand must be doing some sort of formatting prior to sending the request to the Streams Message Manager Server. 

A GET request works using the ExecuteStreamCommand because there is no payload. 

I can't use the invokeHTTP because it currently doesn't have configuration to authenticate using kerberos which is why I fell back on using ExecuteStreamCommand. 

[#4]

Well to be honest it is pretty hard to debug something you have no clue about 😄
From your command I see that you are using the value for "referer" as a variable taken out of an attribute. I assume that the value is correct.

Now, based on my experience with ExecuteStreamCommand, I would give each parameter after a semi colon.... something like: ( it is worth a shot, otherwise you will need to test each parameter until you find the faulty one)
-X;POST;-H;referer:${Referer};-H;'Content-Type: application/json';-d;'{\"newTopics\": [{\"name\":\"testing123\",\"numPartitions\":3,\"replicationFactor\":3}], \"allTopicNames\":[\"testing123\"]}';--negotiate;-u;:;-b;/tmp/cookiejar.txt;-c;/tmp/cookiejar.txt

[#5]

@cotopaul 

The reason why I say that is because if I change the command arguments to this - 

-X GET -H referer:${Referer} -H 'Content-Type: application/json' --negotiate -u : -b /tmp/cookiejar.txt -c /tmp/cookiejar.txt http://SMM-HOSTNAME:8585/api/v1/admin/brokers

It returns successfully.

The only that changed is the HTTP Request Method and removing the Payload. 

[#6]

Well, besides the fact that you are calling a different API endpoint as you previously did, yea, it could be that the payload is the one messing up your command.

What I would try is to use an UpdateAttribute to generate the curl command as an atrribute and see how it gets constructed. I would then take the command and execute it on my server to see if it works — if it gets constructed correctly. Otherwise, you will see where the error is located and you can try and solve it.

[#7]

Sorry typo on my part. Even with the */api/v1/admin/topics it works.

I cloned the nifi github repo and added some more log statements to the executeStreamProcessor changed the edited the logback.xml to enable DEBUG for that processor. It looks like it stripping the quotes in payload.

Here is an example request: curl -X POST -H 'Content-Type:application/json' -d '{"name": "Apple AirPods", "data": {"color": "white", "generation": "3rd", "price": 135}}' https://api.restful-api.dev/objects

This works in the command line.

If you configure the executeStreamCommand like this:

Command Arguments Strategy: Command Arguments Property

Command Arguments: -X POST -H 'Content-Type:application/json' -d '{\"name\":\"Apple AirPods\",\"data\":{\"color\":\"white\",\"generation\":\"3rd\",\"price\":135}}' https://api.restful-api.dev/objects

Command Path: curl

Argument Delimiter: " " (Space)

Here's what I see in the logs - 

Executing and waiting for command: curl, with the arguments: [-X, POST, -H, 'Content-Type:application/json', -d, '{\name\:\Apple AirPods\,\data\:{\color\:\white\,\generation\:\3rd\,\price\:135}}', https://api.restful-api.dev/objects]

@cotopaul , @ckumar , @MattWho , @steven-matison - Is there a way to get around this?

[#8]

To be really honest, I do not know how you could not remove those double quotes. Maybe you can try escaping the quotes with a backslash and also wrap the entire command in single quotes.

Something like: (I might have added a few extra single quotes by mistake but you get the point)

'-X POST -H '\''Content-Type:application/json'\'' -d '\''{"name": "Apple AirPods", "data": {"color": "white", "generation": "3rd", "price": 135}}'\'' https://api.restful-api.dev/objects'

In theory, the double quotes around the JSON string are escaped with a backslash and the entire command is wrapped in single quotes. The single quotes allow the command to be passed to the ExecuteStreamCommand processor as a single argument, which ensures that the quotes are preserved.

PS: I did not test this and I am not 100% it will work, but at least it is worth a shot until somebody with more experience can guide your further.

[#9]

if it works in the command line, you might want to make the script and just pass the variable to it?

curl -X POST -H 'Content-Type:application/json' -d '{"name": "Apple AirPods", "data": {"color": "white", "generation": "3rd", "price": 135}}' https://api.restful-api.dev/objects

If this works. You can make the script like:

curl -X POST -H 'Content-Type:application/json' -d '{"name": "$1", "data": {"color": "$2", "generation": "$3", "price": $4}}' https://api.restful-api.dev/objects

then just pass the variables in the Command Arguments:

Variable1;Variable2;Variable3;Variable4

Command Path: path to the script 

Note that for windows it's %1, %, ... 2 and not $1, $2, ... for variables.
 

[#10]

@Kefkoe 

I really wanted to try and keep it inline and directly in NiFi itself. That way there is no external dependencies to put the script on each node within our NiFi cluster. Good idea as a last resort.