@venu413  If you open the NiFi summary UI (NiFi UI --> Global menu --> Summary), select the connections tab, locate this connection with the 54 queued flowfiles, and then click the cluster connection summary icon ( )to far right, Are all 54 queued FlowFiles on same node? Is anything being logged in the nifi-app.log on that node were these FlowFiles are queued? Any observed errors in nifi-app.log during startup if you restart this node?   In your nifi.properties file, what values are configured for these properties: nifi.cluster.load.balance.comms.timeout= nifi.cluster.load.balance.connections.per.node= nifi.cluster.load.balance.host= nifi.cluster.load.balance.max.thread.count= nifi.cluster.load.balance.port= nifi.cluster.node.address= If recommend that both the "nifi.cluster.node.address=" and "nifi.cluster.load.balance.host=" have been configured uniquely per node in your cluster to the resolvable hostname for the given node.  So if you node has a hostname of node1.mycompany.com, then this hostname should be used in both these properties in the NiFi running on that host.  Restart is needed anytime you edit the nifi.properties file. ... View more

@saivenkatg55    A very common reason for UI slowness is JVM Garbage Collection (GC).  All GC events are stop-the-world events whether it is a partial or full GC event.  Partial/young GC is normal and healthy, but if it is being triggered back to back non stop or is running against a very large configured JVM heap it can take time to complete. You can enable some GC logging in your NiFi bootstrap.conf file so you can see how often GC is running to attempt to free space in your NiFi JVM.  To do this you need to add some additional java,arg.<unique num>= entries in your NiFi bootstrap.conf as follows: java.arg.20=-XX:+PrintGCDetails java.arg.21=-XX:+PrintGCTimeStamps java.arg.22=-XX:+PrintGCDateStamps java.arg.23=-Xloggc:<file> The last entry allows you to specific a separate log file for this output to be written in to rather than stdout. NiFi does store information about component status in heap memory.  This is the info you can see on any component (processor, connection, process group, etc.) when you right click on it and select "view status history" from the displayed context menu.  You'll notice that these component report status for a number of data points.  When your restart your NiFi, everything in the JVM heap memory is gone.  So over the next 24 hours (default data point retention) the JVM heap will be holding a full set of status points again.   You can adjust the component status history buffer size and datapoint frequency to reduce heap usage here if this status history is not that important to you via the following properties in the nifi.properties file: nifi.components.status.repository.buffer.size=1440 nifi.components.status.snapshot.frequency=1 min above represents defaults.  For every status history point for every single component, NiFi will retain 1440 status points (recording 1 point every 1 min).   This totals 24 hours worth of status history for every status point.  Changing the buffer to 288 and frequency to 5 minutes will reduce number of points retained by 80% while still giving your 24 hours worth of points. The dataflows you build may result in high heap usage triggering a lot of heap pressure.  Those NiFi components that can result in high heap usage are documented.  From the NiFi Global menu in the upper right corner of the NIFi UI, select "Help".  You will see the complete list of components on the left had side.  When you select a component, details about that component will be displayed.  One of the details is "System Resource Considerations". For example, here is the system resource considerations for the MergeContent processor: You may need to make adjustments to your dataflow designs to reduce heap usage.   NiFi also holds FlowFile metadata for queued FlowFiles in heap memory.  NiFi Does have a configurable swap threshold (which is applied per connection) to help with heap usage here.  When a queue grows too large, FlowFile metatdata in excess of the configured swap threshold will be written to disk.  The swapping in and swapping out of FlowFiles from disk can affect dataflow performance.  NiFi's default backpressure object thresholds settings for connections is set low enough that swapping would typically never occur.  However, if you have lots and lots of connections with queued FlowFiles, that heap usage can add up.  This article I wrote may help you here: https://community.cloudera.com/t5/Community-Articles/Dissecting-the-NiFi-quot-connection-quot-Heap-usage-and/ta-p/248166 ----- Other than heap usage, component validation can affect NiFi UI responsiveness.  Here is an article i wrote about that: https://community.cloudera.com/t5/Community-Articles/HDF-NiFi-Improving-the-performance-of-your-UI/ta-p/248211   Here is another useful article you may want to read: https://community.cloudera.com/t5/Community-Articles/HDF-NIFI-Best-practices-for-setting-up-a-high-performance/ta-p/244999   Hope this helps you with some direction to help improve your NiFi UI responsiveness/performance, Matt     ... View more

@Former Member  Simply configuring the ldap-provider in the identity-providers.xml file will not result in NiFi-Registry using it. Make sure you have set the following property in the nifi-registry.properties file: nifi.registry.security.identity.provider=ldap-provider This tells NiFi to use the "ldap-provider" configured in that file. Also make sure the file is named "identity-providers.xml" and not "login-identity-providers.xml".   NiFi-Registry uses the former while NiFi uses the latter identity providers filename. One other things to consider... If NiFi-Registry is configured to support Spnego: nifi.registry.kerberos.spnego.authentication.expiration=12 hours nifi.registry.kerberos.spnego.keytab.location= nifi.registry.kerberos.spnego.principal= Spnego auth will be attempted before any configured identity provider.  So all it takes is to have Spnego enabled in your browser and NiFi-Registry to be setup to support Spnego auth and you will not see login page as well.  If you do not have Spnego enabled in your browser, then this is not your issue because even if configured if browser does not return Spengo creds, NiFi-Registry will move on to next configured authentication provider.   Hope this helps, Matt ... View more

@Siraj  Does your ConsumeMQTT processor produce all output FlowFIles with the same Filename? Your MergeContent processor will merge if both configured min settings are satisfied at the end of execution.  If you have your MergeContent processor configured to run as fast as possible (run schedule set to 0 sec default). It may upon execution only see one FlowFile on incoming connection at that moment in time and put only one FlowFile in a bin and merge just that one FlowFile since you set your min num entries to "1".   I suggest you edit your MergeContent as follows: 1. Configure "Correlation Attribute Name" to "filename".  2. Perhaps increase your min setting from 1 to some higher value. 3. Always set "Max Bin Age".  (This is your forced merge property, it will force a bin to merge even if it has not reach both min values within this configured amount of time.) 4. Make sure you have enough bins to accommodate the expected number of unique filenames plus 1 extra.  (If all bins have allocated FlowFiles to them and the next FlowFile cannot be added to an existing bin, the oldest bin will be forced to merge to free a bin).   The NiFi putFile processor does not support append.  Append actions can cause all kinds of issue especially with a NiFi cluster where the target directory of the putFile is mounted to all the NiFi nodes. You can't have multiple nodes trying to write/append to the same file at the same time. My suggestion would be to use the UpdateAttribute processor before your putFile to modify the filename attribute.   Perhaps prepend a uuid to the filename to ensure uniqueness across multiple files or NiFi nodes (if clustered). ${UUID()}-${filename}   Hope this helps you, Matt ... View more

@Siraj    The NiFi putFile processor does not support append.  Append actions can cause all kinds of issue especially with a NiFi cluster where the target directory of the putFile is mounted to all the NiFi nodes. You can't have multiple nodes trying to write/append to the same file at the same time. My suggestion would be to use the UpdateAttribute processor before your putFile to modify the filename attribute.   Perhaps prepend a uuid to the filename to ensure uniqueness across multiple files or NiFi nodes (if clustered). ${UUID()}-${filename}   Hope this helps you, Matt ... View more