Thanks Matt ... View more

@saivenkatg55    Your Out Of Memory (OOM) issue is occurring in the NiFi toolkit script and not in NiFi itself.  Ambari will execute the NiFi toolkit before the NiFi service itself is ever started.  If the NiFi toolkit fails, Ambari does not proceed any further. I can see you are running HDF 3.2.0 which exposes the heap memory settings under the NiFi configs in Ambari: Locate the above line and increase your heap settings here.  Once the toolkit script finished running which must occur before NiFi is even started, that JVM is gone.  So the memory allocation set here are short lived and will not impact available memory for your NiFi. Hope this helps, Matt   ... View more

@venkii    You need to generate a certificate for each of your NiFi nodes in your NiFi cluster. While you can create just one certificate (single DN) with multiple Subject Alternative Names (SAN) entries for each host, this is not security best practices. There is no point in setting multiple identities for the same DN string. For each "Initial User Identity <num>", the configured string is added as a user in the users.xml file for each "Node Identity <num>", the configured string has authorizations needed for NiFi nodes associated to that user's uuid in the authorizations.xml file. In order to access the NiFi UI after securing NiFi, your user/client will need to authenticate with your NiFi.  By default the only configured user authentication method setup is via TLS certificates which requires you to to create a certificate for each user accessing NiFi.  This certificate would be loaded in to your browser and not NiFi itself (the public certificate for your user's private certificate would need to be added to the NiFi truststore in order for NiFi to trust the cert).   NiFi can also be configured to support kerberos, ldap, openId connect, etc alternative methods for user authentication. https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#user_authentication Once a user is successfully authenticated, that user must be authorized before they can even see the NiFi UI. https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#multi-tenant-authorization I understand you have already started down this path, but I recommend you carefully read above document links for more details.         ... View more

Agree with @MattWho . You can use the nifi expression language to choose the certain files from your source. Based on the date or file name you can filter out the files using FetchSFTP processor. i can see in the latest version of nifi expression language is false in ListSFTP processor. to handle your case you have to do it in two steps ListSFTP ==> FetchSFTP (If you want to put dynamic date filters in your source directories).   e.g. you can mention the below in Remote file property of FetchSFTP 0000000001_${now():toNumber():minus(86400000):format('yyyyMMdd')}235959_filename this will gives us output as 0000000001_20200112235959_filename.   Regards Nitin     ... View more

Thanks Steven, I removed the hostname and it started working after I set hostname box empty. ... View more

@saivenkatg55    A very common reason for UI slowness is JVM Garbage Collection (GC).  All GC events are stop-the-world events whether it is a partial or full GC event.  Partial/young GC is normal and healthy, but if it is being triggered back to back non stop or is running against a very large configured JVM heap it can take time to complete. You can enable some GC logging in your NiFi bootstrap.conf file so you can see how often GC is running to attempt to free space in your NiFi JVM.  To do this you need to add some additional java,arg.<unique num>= entries in your NiFi bootstrap.conf as follows: java.arg.20=-XX:+PrintGCDetails java.arg.21=-XX:+PrintGCTimeStamps java.arg.22=-XX:+PrintGCDateStamps java.arg.23=-Xloggc:<file> The last entry allows you to specific a separate log file for this output to be written in to rather than stdout. NiFi does store information about component status in heap memory.  This is the info you can see on any component (processor, connection, process group, etc.) when you right click on it and select "view status history" from the displayed context menu.  You'll notice that these component report status for a number of data points.  When your restart your NiFi, everything in the JVM heap memory is gone.  So over the next 24 hours (default data point retention) the JVM heap will be holding a full set of status points again.   You can adjust the component status history buffer size and datapoint frequency to reduce heap usage here if this status history is not that important to you via the following properties in the nifi.properties file: nifi.components.status.repository.buffer.size=1440 nifi.components.status.snapshot.frequency=1 min above represents defaults.  For every status history point for every single component, NiFi will retain 1440 status points (recording 1 point every 1 min).   This totals 24 hours worth of status history for every status point.  Changing the buffer to 288 and frequency to 5 minutes will reduce number of points retained by 80% while still giving your 24 hours worth of points. The dataflows you build may result in high heap usage triggering a lot of heap pressure.  Those NiFi components that can result in high heap usage are documented.  From the NiFi Global menu in the upper right corner of the NIFi UI, select "Help".  You will see the complete list of components on the left had side.  When you select a component, details about that component will be displayed.  One of the details is "System Resource Considerations". For example, here is the system resource considerations for the MergeContent processor: You may need to make adjustments to your dataflow designs to reduce heap usage.   NiFi also holds FlowFile metadata for queued FlowFiles in heap memory.  NiFi Does have a configurable swap threshold (which is applied per connection) to help with heap usage here.  When a queue grows too large, FlowFile metatdata in excess of the configured swap threshold will be written to disk.  The swapping in and swapping out of FlowFiles from disk can affect dataflow performance.  NiFi's default backpressure object thresholds settings for connections is set low enough that swapping would typically never occur.  However, if you have lots and lots of connections with queued FlowFiles, that heap usage can add up.  This article I wrote may help you here: https://community.cloudera.com/t5/Community-Articles/Dissecting-the-NiFi-quot-connection-quot-Heap-usage-and/ta-p/248166 ----- Other than heap usage, component validation can affect NiFi UI responsiveness.  Here is an article i wrote about that: https://community.cloudera.com/t5/Community-Articles/HDF-NiFi-Improving-the-performance-of-your-UI/ta-p/248211   Here is another useful article you may want to read: https://community.cloudera.com/t5/Community-Articles/HDF-NIFI-Best-practices-for-setting-up-a-high-performance/ta-p/244999   Hope this helps you with some direction to help improve your NiFi UI responsiveness/performance, Matt     ... View more