Nifi Restarts continuously after the upgrade of AKS Cluster to 1.29.4

[#1]

Hi Team

We recently upgraded our AKS cluster from 1.28.5 to 1.29.4 . The apache nifi - version 1.16.3 was already installed in it and running successfully.  After the upgrade to 1.29.4 , one of the Pod - Nifi was continuously restarting and goes to Crashloopbackoff. When we investigate, the server container keeps restarting and there were no logs showing any error as it is changing status to Running and then to Not ready  very quickly and pod reports Crashloopbackoff. We are also unable to login inside the pod due to this. At times when the status is running , it is allowing to connect for few seconds and we noticed that the KeyStore.jks and truststore,jks from certmanager is not present within tls folder. 

We are not sure what is the cause of the continuous restart of the server container after upgrade of aks cluster to 1.29.4

when we compared the logs of the nifi-0 and nifi-1 pod, the nifi -0 pod showing below log and running successfully. Whereas nifi-1 and nifi-2 do not have these logs.

/opt/nifi/nifi-current/tls/truststore.jks is not readable! Waiting for cert-manager sidecar to populate it.

Could you please help share some inputs to resolve this issue ?

Best Regards

Dhinesh Kumar Ganeshan

[#2]

@GDK Welcome to the Cloudera Community!

To help you get the best possible solution, I have tagged our NiFi expert @steven-matison  who may be able to assist you further.

Please keep us updated on your post, and we hope you find a satisfactory solution to your query.

[#3]

 @steven-matison ,  Requesting you to let us know if you have any suggestion on this issue.

@DianaTorres 

[#4]

@MattWho @SAMSAL Hi! Do you have any insights here? Thanks!

[#5]

Hi Team

We would like to share some additional findings while observing the node server logs, we noticed that the order in which the nifi pod execution seem to have changed.

An healthy nifi pod execution has logs in following order

1. PROP REPLACE IS FIRST STEP EXECUTED BY SERVER CONTAINER. FOLLOWED BY APP LOG, BOOTSRAP AND USER LOG.
2. TLS CERT MANAGER DIRECTORY CREATION
3. COPY AND CLASS PATH EXECUTION

After upgrading to AKS 1.29 and restart of pods display a different order of execution

2. TLS CERT MANAGER DIRECTORY CREATION

1.PROP REPLACE IS FIRST STEP EXECUTED BY SERVER CONTAINER. FOLLOWED BY APP LOG, BOOTSRAP AND USER LOG.

3.COPY AND CLASS PATH EXECUTION

we are not sure why such behavior in Nifi. We have nifi version 1.16.3 and java - 8.

Any inputs would certainly help.