question Re: Nifi Restarts continuously after the upgrade of AKS Cluster to 1.29.4 in Support Questions

Nifi Restarts continuously after the upgrade of AKS Cluster to 1.29.4

GDK — Thu, 06 Jun 2024 14:11:34 GMT

Hi Team

We recently upgraded our AKS cluster from 1.28.5 to 1.29.4 . The apache nifi - version 1.16.3 was already installed in it and running successfully. After the upgrade to 1.29.4 , one of the Pod - Nifi was continuously restarting and goes to Crashloopbackoff. When we investigate, the server container keeps restarting and there were no logs showing any error as it is changing status to Running and then to Not ready very quickly and pod reports Crashloopbackoff. We are also unable to login inside the pod due to this. At times when the status is running , it is allowing to connect for few seconds and we noticed that the KeyStore.jks and truststore,jks from certmanager is not present within tls folder.

We are not sure what is the cause of the continuous restart of the server container after upgrade of aks cluster to 1.29.4

when we compared the logs of the nifi-0 and nifi-1 pod, the nifi -0 pod showing below log and running successfully. Whereas nifi-1 and nifi-2 do not have these logs.

/opt/nifi/nifi-current/tls/truststore.jks is not readable! Waiting for cert-manager sidecar to populate it.

Could you please help share some inputs to resolve this issue ?

Best Regards

Dhinesh Kumar Ganeshan

Re: Nifi Restarts continuously after the upgrade of AKS Cluster to 1.29.4

DianaTorres — Thu, 06 Jun 2024 17:39:40 GMT

@GDK Welcome to the Cloudera Community!

To help you get the best possible solution, I have tagged our NiFi expert @steven-matison who may be able to assist you further.

Please keep us updated on your post, and we hope you find a satisfactory solution to your query.

Re: Nifi Restarts continuously after the upgrade of AKS Cluster to 1.29.4

GDK — Tue, 18 Jun 2024 15:25:01 GMT

@steven-matison , Requesting you to let us know if you have any suggestion on this issue.

@DianaTorres

Re: Nifi Restarts continuously after the upgrade of AKS Cluster to 1.29.4

DianaTorres — Tue, 25 Jun 2024 19:41:12 GMT

@MattWho @SAMSAL Hi! Do you have any insights here? Thanks!

Re: Nifi Restarts continuously after the upgrade of AKS Cluster to 1.29.4

GDK — Wed, 10 Jul 2024 16:34:35 GMT

Hi Team

We would like to share some additional findings while observing the node server logs, we noticed that the order in which the nifi pod execution seem to have changed.

An healthy nifi pod execution has logs in following order

PROP REPLACE IS FIRST STEP EXECUTED BY SERVER CONTAINER. FOLLOWED BY APP LOG, BOOTSRAP AND USER LOG.
TLS CERT MANAGER DIRECTORY CREATION
COPY AND CLASS PATH EXECUTION

After upgrading to AKS 1.29 and restart of pods display a different order of execution

2. TLS CERT MANAGER DIRECTORY CREATION

1.PROP REPLACE IS FIRST STEP EXECUTED BY SERVER CONTAINER. FOLLOWED BY APP LOG, BOOTSRAP AND USER LOG.

3.COPY AND CLASS PATH EXECUTION

we are not sure why such behavior in Nifi. We have nifi version 1.16.3 and java - 8.

Any inputs would certainly help.

Re: Nifi Restarts continuously after the upgrade of AKS Cluster to 1.29.4

VVPeter — Wed, 29 Jan 2025 13:57:41 GMT

Are there any updates on this? I'm experiencing the same issue