nifi toolkit merge-param-context help

[#1]

Hello, 

I'm investigating Nifi toolkit (1.16 if it matters)
to add key/value pairs to a param-context programmatically
( I can't use existing PC providers )

Nifi toolkit documents the following action:

#> nifi merge-param-context -help

Adds any parameters that exist in the exported context that don't exist in the
existing context. Overwrites any existing inherited parameter contexts with the
provided list.
...
-i,--input <arg> A local file to read as input contents, or a
public URL to fetch

I would like to know if the key/value pairs to add to the PC ( -pcid argument )
can come from the contents of the -i file

If yes, what is expected format of this file please ?
( random testing and error messages didn't help much)


Regards

[#2 · Accepted solution]

@Alf015 

The merge-param-context command will merge an exported parameter context's parameters into another existing parameter context.

As simple exported parameter context with three parameters defined in it looks like this:

{
  "name" : "test-param",
  "description" : "",
  "parameters" : [ {
    "parameter" : {
      "name" : "test1",
      "description" : "",
      "sensitive" : false,
      "value" : "test1",
      "provided" : false,
      "inherited" : false
    }
  }, {
    "parameter" : {
      "name" : "test2",
      "description" : "",
      "sensitive" : false,
      "value" : "test2",
      "provided" : false,
      "inherited" : false
    }
  }, {
    "parameter" : {
      "name" : "test3",
      "description" : "",
      "sensitive" : false,
      "value" : "test3",
      "provided" : false,
      "inherited" : false
    }
  } ],
  "inheritedParameterContexts" : [ ]
}

You can then use the merge-param-context command to merge the exported parameter context with another parameter context that already exists in the target NiFi:

./cli.sh nifi merge-param-context -i /tmp/test-param.json -p /opt/nifi-toolkit/conf/mytarget.properties -pcid b47a13ab-0195-1000-cc06-dc7779729310 --verbose

• "test-param.json" contains above shared json
• "mytarget.properties" contains the connection info for target NiFi.
• "b47a13ab-0195-1000-cc06-dc7779729310" is the id for the destination parameter context in which I am merging the input parameters

There is a sample .properties file (../conf/cli.properties.example) in the toolkit conf directory.

Contents of mine looks like this:

baseUrl=https://nifihostname:8443
keystore=/opt/nifi/conf/keystore.p12
keystoreType=PKCS12
keystorePasswd=<password>
keyPasswd=<password>
truststore=/opt/nifi/conf/truststore.p12
truststoreType=PKCS12
truststorePasswd=<password>
proxiedEntity=<nifiadmin>

You can get the values for above from your nifi.properties file of the target/destination NiFi.  The proxiedEntity is my NiFi user identity that has permissions to edit the destination parameter context I am importing the new parameters into.  NOTE:  Your NiFi node must be authorized to /proxy user requests with Access Policies in destination NiFi.  If you choose not to use a proxiedEntity, your NiFi node will need to be directly authorized to edit the target parameter context.

Please help our community grow. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped.

Thank you,
Matt

[#3 · Accepted solution]

@Alf015 

The merge-param-context command will merge an exported parameter context's parameters into another existing parameter context.

As simple exported parameter context with three parameters defined in it looks like this:

{
  "name" : "test-param",
  "description" : "",
  "parameters" : [ {
    "parameter" : {
      "name" : "test1",
      "description" : "",
      "sensitive" : false,
      "value" : "test1",
      "provided" : false,
      "inherited" : false
    }
  }, {
    "parameter" : {
      "name" : "test2",
      "description" : "",
      "sensitive" : false,
      "value" : "test2",
      "provided" : false,
      "inherited" : false
    }
  }, {
    "parameter" : {
      "name" : "test3",
      "description" : "",
      "sensitive" : false,
      "value" : "test3",
      "provided" : false,
      "inherited" : false
    }
  } ],
  "inheritedParameterContexts" : [ ]
}

You can then use the merge-param-context command to merge the exported parameter context with another parameter context that already exists in the target NiFi:

./cli.sh nifi merge-param-context -i /tmp/test-param.json -p /opt/nifi-toolkit/conf/mytarget.properties -pcid b47a13ab-0195-1000-cc06-dc7779729310 --verbose

• "test-param.json" contains above shared json
• "mytarget.properties" contains the connection info for target NiFi.
• "b47a13ab-0195-1000-cc06-dc7779729310" is the id for the destination parameter context in which I am merging the input parameters

There is a sample .properties file (../conf/cli.properties.example) in the toolkit conf directory.

Contents of mine looks like this:

baseUrl=https://nifihostname:8443
keystore=/opt/nifi/conf/keystore.p12
keystoreType=PKCS12
keystorePasswd=<password>
keyPasswd=<password>
truststore=/opt/nifi/conf/truststore.p12
truststoreType=PKCS12
truststorePasswd=<password>
proxiedEntity=<nifiadmin>

You can get the values for above from your nifi.properties file of the target/destination NiFi.  The proxiedEntity is my NiFi user identity that has permissions to edit the destination parameter context I am importing the new parameters into.  NOTE:  Your NiFi node must be authorized to /proxy user requests with Access Policies in destination NiFi.  If you choose not to use a proxiedEntity, your NiFi node will need to be directly authorized to edit the target parameter context.

Please help our community grow. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped.

Thank you,
Matt

[#4]

Hello,

Thanks for the quick and detailed answer.

I've tested the solution and managed to use it
to insert new key / value pairs int he PC if the key is not already present

But I could not update the value of a key already existing in the PC.

Is this expected behaviour ?
Or is there a way to just update the value of an existing key that I would have missed ?

Regards

[#5]

A quick series of command I used to test the above scenario
outlined in my reply just above
where an existing key did not get its corresponding value updated

#> nifi get-param-context -pcid 5d3d3ac8-0196-1000-ffff-ffffaf0565d4

# Name Value Sensitive Description
- ---------- ------------ --------- -----------
1 test-key-1 test-value-1 false
2 test-key-2 test-value-2 false

#> nifi merge-param-context -pcid 5d3d3ac8-0196-1000-ffff-ffffaf0565d4 -i sample-pc.json
Found parameter to add - 'test-key-3'
Waiting for update request to complete...
Waiting for update request to complete...
Waiting for update request to complete...
Waiting for update request to complete...

#> nifi get-param-context -pcid 5d3d3ac8-0196-1000-ffff-ffffaf0565d4

# Name Value Sensitive Description
- ---------- ------------------ --------- -----------
1 test-key-1 test-value-1 false
2 test-key-2 test-value-2 false
3 test-key-3 test-added-value-3 false

The new key test-key-3 is added with its value to the PC.
But the value corresponding to test-key-1 is left unchanged


--- The sample json contents used below

{
"name" : "test-param-ctx",
"description" : "",
"parameters" : [ {
"parameter" : {
"name" : "test-key-1",
"description" : "",
"sensitive" : false,
"value" : "test-modified-value-1",
"provided" : true,
"inherited" : false
}
}, {
"parameter" : {
"name" : "test-key-3",
"description" : "",
"sensitive" : false,
"value" : "test-added-value-3",
"provided" : false,
"inherited" : false
}
} ],
"inheritedParameterContexts" : [ ]
}

[#6 · Accepted solution]

@Alf015 

The  "merge-param-context" cli.sh NiFi command does the following when merging one parameter context into another:

Adds any parameters that exist in the exported context that don't exist in the
existing context.

 So your observations align with the documented behavior of this command.  It will not modify value of any already existing parameter in the destination parameter context.

It sounds like what you want to be using in the "set-param" command instead:

Creates or updates a parameter in the given parameter context.

Above will allow you to modify an existing parameter within the target parameter context.

Please help our community grow. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped.

Thank you,
Matt

[#7]

Thanks for the reminder regarding the semantic of merge-param-context.
I overlooked the contents of the help and my initial posting.

Regards