Support Questions

Find answers, ask questions, and share your expertise
Announcements
Share your experience with Cloudera on G2 and get a $25 Amazon Gift card.

Nifi PutIceberg Processor failed to acquire SAS token on Azure

avatar
Contributor

I'm running several ETL flows in CDF/Nifi. The flows all involve writing some data to Iceberg tables. These flows are mostly running on a schedule that ranges from hourly to weekly (i.e., infrequent writes). Each data flow deployment has a few independent DAGs that all end with the PutIceberg processor, such that there are multiple separate ETL processes running at different intervals all in the same Nifi deployment.

The problem I see occurs pretty rarely (~once a month) and only for some deployments. Occasionally, the PutIceberg processor will error with 'Failed to acquire a SAS token' (see full error log below).

This will continue happening every time the processor runs unless I restart the flow and then it will run fine again. It seems to happen more often when the processor runs at a daily interval. My flows are using Nifi runtime 1.27.0.2.3.14.0-14 and I'm on CDP Public Cloud on Azure.

```

org.apache.hadoop.fs.azurebfs.contracts.exceptions.SASTokenProviderException: Failed to acquire a SAS token for create-file on [my-data-warehouse-bucket-and-table]/metadata/bb545710-14ea-4b07-b0f5-668978be4e8d-m1.avro due to org.apache.hadoop.security.AccessControlException: org.apache.ranger.raz.intg.RangerRazException: <!doctype html><html lang="en"><head><title>HTTP Status 401 ??? Unauthorized</title><style type="text/css">body {font-family:Tahoma,Arial,sans-serif;} h1, h2, h3, b {color:white;background-color:#525D76;} h1 {font-size:22px;} h2 {font-size:16px;} h3 {font-size:14px;} p {font-size:12px;} a {color:black;} .line {height:1px;background-color:#525D76;border:none;}</style></head><body><h1>HTTP Status 401 ??? Unauthorized</h1><hr class="line" /><p><b>Type</b> Status Report</p><p><b>Message</b> org.apache.hadoop.security.authentication.util.SignerException: Invalid signature</p><p><b>Description</b> The request has not been applied to the target resource because it lacks valid authentication credentials for that resource.</p><hr class="line" /><h3>Apache Tomcat/8.5.96</h3></body></html>; HttpStatus: 401
at org.apache.hadoop.fs.azurebfs.services.AbfsClient.appendSASTokenToQuery(AbfsClient.java:1233)
at org.apache.hadoop.fs.azurebfs.services.AbfsClient.appendSASTokenToQuery(AbfsClient.java:1199)
at org.apache.hadoop.fs.azurebfs.services.AbfsClient.createPath(AbfsClient.java:396)
at org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore.conditionalCreateOverwriteFile(AzureBlobFileSystemStore.java:625)
at org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore.createFile(AzureBlobFileSystemStore.java:568)
at org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem.create(AzureBlobFileSystem.java:335)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1177)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1157)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1046)
at org.apache.iceberg.hadoop.HadoopOutputFile.createOrOverwrite(HadoopOutputFile.java:85)
... 19 common frames omitted

```

7 REPLIES 7

avatar
Community Manager

@MattWho @SAMSAL @venkatsambath Hi! Do you have any insights here? Thanks!


Regards,

Diana Torres,
Senior Community Moderator


Was your question answered? Make sure to mark the answer as the accepted solution.
If you find a reply useful, say thanks by clicking on the thumbs up button.
Learn more about the Cloudera Community:

avatar
New Contributor

Hi,

we experience very the same issue but on AWS. This is Kerberos authN related.
The Kerberos token expires leading to issues when an AWS STS or Azure SAS token is about to be acquired. 

avatar
Contributor

Good to know it's not an Azure specific issue then. Thanks @DanielR 

avatar
Contributor

Do you know how to keep it from expiring or renew the token from within Nifi?

avatar
Community Manager

@ipson Has the reply helped resolve your issue? If so, please mark the appropriate reply as the solution, as it will make it easier for others to find the answer in the future.  Thanks.


Regards,

Diana Torres,
Senior Community Moderator


Was your question answered? Make sure to mark the answer as the accepted solution.
If you find a reply useful, say thanks by clicking on the thumbs up button.
Learn more about the Cloudera Community:

avatar
New Member
Thanks for sharing this helpful troubleshooting information. The explanation of SAS token authentication and renewal issues in NiFi is very useful for anyone working with Azure and Iceberg integrations. Hopefully a permanent solution for automatic token refresh will be available in future updates.

avatar
Master Collaborator

Hello @Softwarebyt

Based on the error posted for @ipson, looks like this matches two NiFi Jiras: 
https://issues.apache.org/jira/browse/NIFI-13720
https://issues.apache.org/jira/browse/NIFI-13722 

Those fixes get included on CFM 2.1.7.1000: https://docs.cloudera.com/cfm/2.1.7/release-notes/topics/cfm-fixed-issues.html 
And CDF for DataHub 7.3.1.0: https://docs.cloudera.com/cdf-datahub/7.3.1/release-notes/topics/cdf-datahub-fixed-issues-flow-manag... 


Regards,
Andrés Fallas
--
Was your question answered? Please take some time to click on "Accept as Solution" below this post.
If you find a reply useful, say thanks by clicking on the thumbs-up button.