Nifi application error on AKS

[#1]

We have recently migrated NIFI to AKS and it was working perfectly. Recently there was AKS version upgrade to 1.27.7 and since then UAT cluster has issues. The error while accessing Nifi UI.

javax.net.ssl.SSLException: Connection reset

When I get the logs of 3 pods, I am getting error

ERROR [Timer-Driven Process Thread-3] o.a.n.c.s.StandardControllerServiceNode Failed to invoke @OnEnabled method of DBCPConnectionPool[id=12d922eb-87dc-3b72-9550-5e49f6849617] due to org.apache.nifi.reporting.InitializationException: Can't load Database Driver

Need suggestions to fix this error

[#2]

@C1082 Welcome to the Cloudera Community!

To help you get the best possible solution, I have tagged our NiFi experts @mburgess @bbahamondes  who may be able to assist you further.

Please keep us updated on your post, and we hope you find a satisfactory solution to your query.

[#3]

Thank you Diana 

[#4]

@mburgess @bbahamondes, I have checked Nifi properties file to check the location of the driver file but was not able to locate it. I was wondering if the Jar file can be located

[#5]

@C1082 

The  DBCPConnectionPool is a controller service that an end user would have added via the NiFi UI.  The configuration of this controller service is done by the user and one of the properties specifies the user defined location of the Database Driver which the user must provide and is not included with NiFi.

The Dataflow components added to the NiFi canvas have not relationship to UI access issues.  The "javax.net.ssl.SSLException: Connection reset" exception when trying to access the UI is an issue with the TLS exchange between your client (browser) and NIFi.   You'll need to look closer at the nifi-app.log and nifi-user.log for this exception and review the entire stack trace that goes with it.  Without knowing the specific of your NiFi setup, I can't say whether your NiFi is enforcing a Mutual TLS exchange or only a one-way TLS exchange.  A securely configured NiFi depending on configuration will either "REQUIRE" the client to provide a trusted clientAuth certificate in the TSL response or "WANT" a trusted clientAuth certificate in the response.  A Connection Reset may happen if the TLS exchange was not successful which could be a trusted chain issue, network issue, or missing clientAuth certificate when NiFi configuration required it in the TLS response.

If you found any of the suggestions/solutions provided helped you with your issue, please take a moment to login and click "Accept as Solution" on one or more of them that helped.

Thank you,
Matt

[#6]

Hi Matt, 

I checked the nifi-app.log and it is giving this error 

2024-03-28 12:52:55,028 ERROR [Timer-Driven Process Thread-3] o.a.n.c.s.StandardControllerServiceNode StandardControllerServiceNode[service=DBCPCo
nnectionPool[id=8c23244e-6b42-38c5-aaf2-effc40ab1d4b], versionedComponentId=null, processGroup=StandardProcessGroup[identifier=43891bd4-018b-1000-
0000-00006f0c9f3e], active=true] Failed to invoke @OnEnabled method due to org.apache.nifi.reporting.InitializationException: Can't load Database
Driver: {}
org.apache.nifi.reporting.InitializationException: Can't load Database Driver
at org.apache.nifi.dbcp.DBCPConnectionPool.getDriverClassLoader(DBCPConnectionPool.java:397)
at org.apache.nifi.dbcp.DBCPConnectionPool.onConfigured(DBCPConnectionPool.java:336)
at sun.reflect.GeneratedMethodAccessor113.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:142)
at org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:130)
at org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:75)
at org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotation(ReflectionUtils.java:52)
at org.apache.nifi.controller.service.StandardControllerServiceNode$2.run(StandardControllerServiceNode.java:432)
at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.ClassNotFoundException: com.microsoft.sqlserver.jdbc.SQLServerDriver
at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
at java.lang.ClassLoader.loadClass(ClassLoader.java:419)
at java.lang.ClassLoader.loadClass(ClassLoader.java:352)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:348)

[#7]

@C1082 

That ERROR has nothing to do with the community question you asked about:

The error while accessing Nifi UI.

javax.net.ssl.SSLException: Connection reset

Fixing the ERROR logged by the controller service DBCPConnectionPool shared in yoru last post will not resolve your UI access issue.

Are you still having issue accessing the NiFi UI?

If not, try searching for DBCPConnectionPool that is throwing this exception and verify its configuration and driver the user has configured it to use.  You can find this specific NiFi controller Service by searching on it unique assigned ID: "8c23244e-6b42-38c5-aaf2-effc40ab1d4b".  You'll want to make sure the driver still exists and the configured location and is owned and accessible by the NiFi service user.   Sharing the exact SQL DB version and currently used database driver being used would also help here. 

Was this Controller Service working before the AKS version upgrade?  

Please help our community continue to thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped.

Thank you,
Matt

[#8]

Hi Matt,

I scaled down the stateful set from 3 to 1 and with primary node, the error disappears about the DBCBpool connection error. I am able to login using UI. But when I scale it up to 2 or 3. The error returns.  It seems there is some replication and communication issue with primary node 0 with node 1 and node 2.

[#9]

Yes the controller service was working before aks version upgrade