NiFi web UI timeouts

[#1]

Running HDF 3.1.0.

I'm quite often getting this error in the NiFi web UI:

"An unexpected error has occured:

javax.ws.rs.ProcessingException: java.net.SocketTimeoutException: Read timed out"

Would appreciate a tip on how to avoid this. Thanks.

[#2 · Accepted solution]

@Garyy 

So the first place you may want to start is opening developer tools in your browser and then trying to connect your NiFi UI and take note of what calls are taking the longest to return and which it eventually times out on.

You may also want to enable garbage collection logging within your NiFi JVM (do this by adding GC logging java args in the NiFI bootstrap.conf file).   If your JVM is encountering long and/or frequent GC (all GC events are stop-the-world events), this can result in timeouts made to the UI.

It is also not entirely clear to me what you mean by your UI access get terminated.  NiFi does not use sessions. What are you observing? Can you provide more detail and screenshots?  

Matt

[#3 · Accepted solution]

@MattWho, By saying my UI session getting terminated unexpected, I meant I was forced out of my nifi session a while after I logged in and worked on it, so I had to log in again. I observed more and recorded my issue. I always got kicked out exactly one hour after my last login. So I searched the configuration folder and found the following in login-identity-providers.xml:

<property name="Authentication Expiration">1 hour</property>

So I believe this is root cause. After updating this value, I should be fine now.

Thank you a lot.

[#4]

@Henrik Olsen

-

NiFi's various timeout settings are very aggressive. They are more ideal for a standalone NiFi instance running a fairly simple dataflow. In a NiFi cluster the following timeouts should be increased:

nifi.cluster.node.connection.timeout=5 secs  (increase to 30 secs)

nifi.cluster.node.read.timeout=5 secs (Increase to 30 secs)

nifi.zookeeper.connect.timeout=3 secs  (increase to 60 secs)

nifi.zookeeper.session.timeout=3 secs  (Increase to 60 secs)

-

A restart of NiFi will be needed after making these changes.

Another thing you could do when this condition is present is to use the browser developer tools to try to catch what action is timing out.

Are you seeing a lot of full garbage collection occurring (if these stop-the-world events are long enough, it can also cause this).

-

Thank you,

Matt

[#5]

@MattWho , Do these parameters only apply to the cases of cluster? I am using a standalone nifi server, but my UI sessions frequently get terminated w/o any messages during development. Could something else, such as security settings, also have such effect? Could you please shed more light? Thanks.

[#6]

@Garyy 

That depends on what your definition is of "Standalone" NiFi server.

A one node NiFi cluster and a standalone NiFi are two different things and I have seen many users with a one Node NiFi cluster refer to that as standalone.  A true standalone NIFi does not use Zookeeper and does not need to replicate requests to other nodes in the NiFi cluster, so none of the "cluster" configuration properties are used.  Additionally, a standalone NiFi has no dependency on zookeeper at the core level.  NiFi clusters use ZK for Cluster Coordinator and Primary node election (these roles only exist in a cluster setup) and Cluster wide state storage (Standalone has no need to share state with other nodes, so all state is simply stored locally).

You can tell if your NiFi is a true standalone by checking the following property in the nifi.properties file:

nifi.cluster.is.node

If it is set to true, then this NiFi is configured to operate as a cluster even if their is only one node
If it is set to false, then node is truly standalone.

Hope this helps,

Matt

[#7]

Thanks a lot @MattWho ! Your explanation is very educational.

I checked my nifi setting. It's a real standalone server.

nifi.cluster.is.node=false

So, do you know what else could terminate UI sessions?

[#8 · Accepted solution]

@Garyy 

So the first place you may want to start is opening developer tools in your browser and then trying to connect your NiFi UI and take note of what calls are taking the longest to return and which it eventually times out on.

You may also want to enable garbage collection logging within your NiFi JVM (do this by adding GC logging java args in the NiFI bootstrap.conf file).   If your JVM is encountering long and/or frequent GC (all GC events are stop-the-world events), this can result in timeouts made to the UI.

It is also not entirely clear to me what you mean by your UI access get terminated.  NiFi does not use sessions. What are you observing? Can you provide more detail and screenshots?  

Matt

[#9 · Accepted solution]

@MattWho, By saying my UI session getting terminated unexpected, I meant I was forced out of my nifi session a while after I logged in and worked on it, so I had to log in again. I observed more and recorded my issue. I always got kicked out exactly one hour after my last login. So I searched the configuration folder and found the following in login-identity-providers.xml:

<property name="Authentication Expiration">1 hour</property>

So I believe this is root cause. After updating this value, I should be fine now.

Thank you a lot.

[#10 · Accepted solution]

@Garyy 

You are correct.  Since NiFi does not use sessions as mentioned in my last response, the client must authenticate every action performed.  When you "login" to NiFi, the result is a bearer token being issued to the user which your browser stores and reuses in all subsequent request to the NiFi endpoints.  At the same time a server side token for your user is also stored on the specific NIFi node you logged in to.  The configuration in your NiFi login provider dictates how long those bearer tokens are good for.  With your setting of 1 hour, you would be forced to re-login again every hour.

Thanks,

Matt