Member since
06-10-2024
1
Post
0
Kudos Received
0
Solutions
06-10-2024
02:07 PM
Hello, I'm quite new to Nifi. I'm using v. 1.24.0 I'm working on a project which requires fetching blobs from Azure Blob Storage and putting them into SFTP. To achieve the desired result, there are two major steps : Get Token Bearer from Authentification Server Calling a webservice to get SAS key with Token Bearer gotten from previous call. There are two limitations : The SAS Key expires every 1440 minutes. The ListAzureBlobStorage processor doesn't accept any incoming flowfiles. Therefore, I thought about dynamically updating Parameter Context and using the parameter as property value in the ListAzureBlobStorage. Following is the high-level approach of my nifi job : GenerateFlowFile : scheduled to trigger at every 1430 minutes + contains the body of the InvokeHTTP request InvokeHTTP (Get Bearer Token) : processor to request a bearer token from the authentication server EvaluateJsonPath (Extract Bearer Token): Extract the bearer token from the response InvokeHTTP (Get SAS Token): processor to obtain the SAS token EvaluateJsonPath (Extract SAS Token): Extract the SAS token from the response ReplaceText (Prepare for Parameter Context Update): Prepare a JSON payload with the SAS token and update the flow file content PutDistributedMapCache (Store SAS Token): Store the SAS token in a distributed cache for later retrieval. FetchDistributedMapCache (Retrieve SAS Token): Retrieve the SAS token from the distributed cache. ReplaceText (Update Parameter Context): Prepare a JSON payload to update the parameter context with the new SAS token. InvokeHTTP (Update Parameter Context) EvaluateJsonPath (Check Update Status): Extract the completion status from the response to check if the update is complete. RouteOnAttribute (Check Update Completion): Route the flow based on whether the update is complete or not InvokeHTTP (GET Method to Check Progress): If the update is not complete, send a GET request to check the progress. InvokeHTTP (DELETE Method to Remove Completed Request): If the update is complete, send a DELETE request to remove the completed request from memory. I took inspirations from the following post : https://community.cloudera.com/t5/Support-Questions/Update-parameter-contexts-dynamically-using-REST-API/m-p/347697 The SAS Token is a sensitive field in ListAzureBlobStorage and FetchStorageBlobStorage. I'm getting the following error : "SAS Token is invalid because the property SAS Token cannot reference parameter 'sasToken' because the Sensitivity of the parameter does not match the Sensitivity of the property". I tried the following : 1. Attempting to update the sensitivity of the parameter 'sasToken' from not sensitive to sensitive by adding a field sensitivity = True in the ReplaceText processor in step 9. { "revision": { "clientId": "bea2399d-2e37-1fff-1f82-17e8cfbc9514", "version": 3 }, "component": { "id": "bea239b6-2e37-1fff-7fe8-a9a01a8d79e7", "parameters": [ { "parameter": { "name": "sasToken", "sensitive": true, "value": "${sasToken}" } } ] } } 2. Pre-set Parameter Sensitivity i.e. ensuring that the 'sasToken' parameter is initially created with the correct sensitivity (i.e., sensitive) in my NiFi parameter context In both the cases, I get the following error : invokehttp.request.url http://localhost:8080/nifi-api/parameter-contexts/bea239b6-2e37-1fff-7fe8-a9a01a8d79e7/update-requests invokehttp.response.body Could not update inherited Parameter Contexts for Parameter Context [ABCD-abcd] because: Cannot update Parameters because doing so would change Parameter 'sasToken' from sensitive to not sensitive Please let me know what do to ? Best regards, Deejay PS : I'm happy to provide more details if needed.
... View more
Labels:
- Labels:
-
Apache NiFi