AzureLogAnalyticsReportingTask Failed to Publish metrics to Azure Log Analytics ... 403 Forbidden

[#1]

Hi,

Im having strange situation and I would really appreciate your help. Basically I was able to setup AzureLogAnalyticsReportingTask (Nifi 1.14) on a cluster and it was working after providing the Workspace ID and Key so I was able to see the log under Azure CustomLog and create my queries and dashboard, however after couple of days it  started generating the following error:

AzureLogAnalyticsReportingTask [id=] Failed to publish metrics to Azure Log Analytics: java.lang.RuntimeException: HTTP/1.1 403 Forbidden

Not sure what happened. I re enter the workspace ID and Key but that did not help.

Cam you please help.

Thanks

[#2]

Can some one help verify this please. Im thinking that there is bug in the 1.14 version around this, because I have tried the same ReportingTask on version 1.11.4 against the same Azure Log Analytics workspace ID and Key and its working, so that tells me its not something with Azure. Both versions have the same settings when it comes to nifi.web.https and nifi.remote.input properties except for nifi.remote.input.socket.port where its set for different ports.  Thank you.

[#3]

@SAMSAL 

If it works initially and then begins to fail, I would start by taking a look at what is being reported on the azure side at the time of each exception.   Is there more details in the nifi-app.log that goes along with the exception you shared?

Cloudera does not have any official support releases yet built off of the Apache NiFi 1.14 release.
I only see one change reported in Apache NiFi Jira related to this reporting task that happened in a release later then 1.11.x you reported no issues with:
https://issues.apache.org/jira/browse/NIFI-6977

Nothing in that jira that went in to Apache NiFi 1.12 leads me to think it would result in the issue you are seeing.
I also see you mentioned a couple properties from the nifi.properties file:

• nifi.web.https.(host, port, port.forwarding, ciphersuites.include, cyphersuites.exclude, network.interface*)?
  • These properties are specific to NiFi's web interface and would not affect the functionality of this reporting task.
• nifi.remote.input.(host, secure,socket.port, http.enabled, http.transaction.ttl)?
  
  • These properties are used for NiFi's Site-To-Site capability.  They also would not be utilized by this particular reporting task.

Do you see the same issues using Apache NiFi 1.13.2?

Thank you,

Matt

[#4]

Matt,

Thanks for your reply. Basically I have two local nifi's: one is that 1.11 and another is 1.14 both of them are setup almost identical in the nifi.properties, however 1.11 is working when passing the azure workspace key and Id and the other 1.14 throwing the error. the log file doesnt have much info besides the message I posted above but here is the stack trace from the log file if that helps in anyway:

java.lang.RuntimeException: HTTP/1.1 403 Forbidden
at org.apache.nifi.reporting.azure.loganalytics.AbstractAzureLogAnalyticsReportingTask.postRequest(AbstractAzureLogAnalyticsReportingTask.java:164)
at org.apache.nifi.reporting.azure.loganalytics.AbstractAzureLogAnalyticsReportingTask.sendToLogAnalytics(AbstractAzureLogAnalyticsReportingTask.java:147)
at org.apache.nifi.reporting.azure.loganalytics.AzureLogAnalyticsReportingTask.sendMetrics(AzureLogAnalyticsReportingTask.java:137)
at org.apache.nifi.reporting.azure.loganalytics.AzureLogAnalyticsReportingTask.onTrigger(AzureLogAnalyticsReportingTask.java:107)
at org.apache.nifi.controller.tasks.ReportingTaskWrapper.run(ReportingTaskWrapper.java:44)
at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)

Im hoping someone can try the same on their end to verify if they are getting the same results