Nifi Dynamic Parameter Context / ListAzureBlobStorage SAS Token

Deejay — Mon, 10 Jun 2024 21:07:23 GMT

Hello,
I'm quite new to Nifi. I'm using v. 1.24.0

I'm working on a project which requires fetching blobs from Azure Blob Storage and putting them into SFTP.

To achieve the desired result, there are two major steps :

Get Token Bearer from Authentification Server
Calling a webservice to get SAS key with Token Bearer gotten from previous call.

There are two limitations :

The SAS Key expires every 1440 minutes.
The ListAzureBlobStorage processor doesn't accept any incoming flowfiles.

Therefore, I thought about dynamically updating Parameter Context and using the parameter as property value in the ListAzureBlobStorage.

Following is the high-level approach of my nifi job :

GenerateFlowFile : scheduled to trigger at every 1430 minutes + contains the body of the InvokeHTTP request
InvokeHTTP (Get Bearer Token) : processor to request a bearer token from the authentication server
EvaluateJsonPath (Extract Bearer Token): Extract the bearer token from the response
InvokeHTTP (Get SAS Token): processor to obtain the SAS token
EvaluateJsonPath (Extract SAS Token): Extract the SAS token from the response
ReplaceText (Prepare for Parameter Context Update): Prepare a JSON payload with the SAS token and update the flow file content
PutDistributedMapCache (Store SAS Token): Store the SAS token in a distributed cache for later retrieval.
FetchDistributedMapCache (Retrieve SAS Token): Retrieve the SAS token from the distributed cache.
ReplaceText (Update Parameter Context): Prepare a JSON payload to update the parameter context with the new SAS token.
InvokeHTTP (Update Parameter Context)
EvaluateJsonPath (Check Update Status): Extract the completion status from the response to check if the update is complete.
RouteOnAttribute (Check Update Completion): Route the flow based on whether the update is complete or not
InvokeHTTP (GET Method to Check Progress): If the update is not complete, send a GET request to check the progress.
InvokeHTTP (DELETE Method to Remove Completed Request): If the update is complete, send a DELETE request to remove the completed request from memory.

I took inspirations from the following post : https://community.cloudera.com/t5/Support-Questions/Update-parameter-contexts-dynamically-using-REST-API/m-p/347697

The SAS Token is a sensitive field in ListAzureBlobStorage and FetchStorageBlobStorage. I'm getting the following error : "SAS Token is invalid because the property SAS Token cannot reference parameter 'sasToken' because the Sensitivity of the parameter does not match the Sensitivity of the property".

I tried the following :

1. Attempting to update the sensitivity of the parameter 'sasToken' from not sensitive to sensitive by adding a field sensitivity = True in the ReplaceText processor in step 9.

{
"revision": {
"clientId": "bea2399d-2e37-1fff-1f82-17e8cfbc9514",
"version": 3
},
"component": {
"id": "bea239b6-2e37-1fff-7fe8-a9a01a8d79e7",
"parameters": [
{
"parameter": {
"name": "sasToken",
"sensitive": true,
"value": "${sasToken}"
}
}
]
}
}

2. Pre-set Parameter Sensitivity i.e. ensuring that the 'sasToken' parameter is initially created with the correct sensitivity (i.e., sensitive) in my NiFi parameter context

In both the cases, I get the following error :

invokehttp.request.url
http://localhost:8080/nifi-api/parameter-contexts/bea239b6-2e37-1fff-7fe8-a9a01a8d79e7/update-requests

invokehttp.response.body
Could not update inherited Parameter Contexts for Parameter Context [ABCD-abcd] because: Cannot update Parameters because doing so would change Parameter 'sasToken' from sensitive to not sensitive

Please let me know what do to ?

Best regards,

Deejay

PS : I'm happy to provide more details if needed.

Re: Nifi Dynamic Parameter Context / ListAzureBlobStorage SAS Token

DianaTorres — Tue, 11 Jun 2024 01:03:21 GMT

@Deejay Welcome to the Cloudera Community!

To help you get the best possible solution, I have tagged our NiFi expert @steven-matison who may be able to assist you further.

Please keep us updated on your post, and we hope you find a satisfactory solution to your query.

question Re: Nifi Dynamic Parameter Context / ListAzureBlobStorage SAS Token in Support Questions

Nifi Dynamic Parameter Context / ListAzureBlobStorage SAS Token

Re: Nifi Dynamic Parameter Context / ListAzureBlobStorage SAS Token