Intermittent 502 Errors Nifi behind ALB in AWS

[#1]

I'm running Nifi in ECS with an ALB. Everything is up and seems normal except there are intermittent 502 errors that pop up in the Nifi ui that look like this

 

It seems that the source of the 502 is the load balancer but since it's intermittent it seems like an interaction between the load balancer and Nifi that is causing this behavior.

I've tried messing with the following settings so that the idle timeout (60 seconds) on the ALB is lower

nifi.web.request.timeout

nifi.remote.input.http.transaction.ttl

I've also made sure the nifi.web.proxy.host configuration is set the hostname that I'm using to hit the loadbalancer.

Has anyone else experienced something similar getting a load balancer in front of Nifi?

[#2]

I picked up Nifi in December and had the same issue with 502s. I couldn't find anything other than make sure your Nifi idle timeout is greater than your load balancer. Which I did. First by 1 second. Then by 10 seconds. They by doubling it. Still the same issues.

I just dealt with it until today it did it to me 3 times on a complex UpdateRecord Expression Language property. AAAAhhhggg!!!

During the subsequent frustratingly redundant Google result, I came across this bug for Spring Boot: https://github.com/spring-projects/spring-boot/issues/38960

I'm not Java programmer but I've heard of Spring and found that Nifi uses Spring Framework. But I didn't see Spring Boot mentioned for Nifi. So out of spite, desperation and hoping it was a similar bug, I changed my load balancer's timeout to 25 seconds. I haven't had a 502 in over 4 hours now!

I was preparing to upgrade to Nifi 2.1.0 as a last ditch effort. And now I see the release notes mention updating Spring Framework. Hopefully that includes the same bug fix from January 2024 for Spring Boot. But at least now I can finish building out what I need to and worry about the upgrade when I'm done. 

Hopefully this helps you (if you still need it) and it's not just a fluke or the planetary alignment.

[#3]

We have the same issue. Are currently running with NiFi 2.3 and are still having this issue.

It seems to be something between NiFi and the load balancer indeed. But we haven't found out the cause yet.

Interested in updates from others facing this issue.

[#4]

I'm having this same issue on the 2.3 also. AWS support says:

"target instances are likely to close TCP connections prematurely (with TCP RST or FIN) while the ALB has outstanding requests, causing 502 errors. This could be due to mismatched keep-alive timeout settings between the ALB and targets."

The ALB is set to a 1m timeout so it could be NiFi closing it early. Looking at that next. 

[#5]

Possibly related to https://issues.apache.org/jira/browse/NIFI-14462

I suggest reviewing discussion in this Apache Jira and reviewing your "nifi.web.request.timeout" setting in the nifi.properties file.
Making adjustments to this setting may help here.

Please help our community grow. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped.

Thank you,
Matt