Hi,   I've tried to set a global variable for "Minimum Number of Entries" but it doesn't work. How did you manage to do this? I have n number of flowfiles and I need to wait for all of them to arrive at "MergeContent" processor so they can be merged. I only know the number of flowfiles at runtime so I keep updating a  counter set as a global variable, but then I tried to set the "Minimum Number of Entries" of "MergeContent" processor to that variable but it doesn't work. I don't know how long I need wait to get all the flowfiles so I can't use that either. I don't know what to do, can someone help? ... View more

Thanks Matt ... View more

@saivenkatg55    Your Out Of Memory (OOM) issue is occurring in the NiFi toolkit script and not in NiFi itself.  Ambari will execute the NiFi toolkit before the NiFi service itself is ever started.  If the NiFi toolkit fails, Ambari does not proceed any further. I can see you are running HDF 3.2.0 which exposes the heap memory settings under the NiFi configs in Ambari: Locate the above line and increase your heap settings here.  Once the toolkit script finished running which must occur before NiFi is even started, that JVM is gone.  So the memory allocation set here are short lived and will not impact available memory for your NiFi. Hope this helps, Matt   ... View more

@venkii    You need to generate a certificate for each of your NiFi nodes in your NiFi cluster. While you can create just one certificate (single DN) with multiple Subject Alternative Names (SAN) entries for each host, this is not security best practices. There is no point in setting multiple identities for the same DN string. For each "Initial User Identity <num>", the configured string is added as a user in the users.xml file for each "Node Identity <num>", the configured string has authorizations needed for NiFi nodes associated to that user's uuid in the authorizations.xml file. In order to access the NiFi UI after securing NiFi, your user/client will need to authenticate with your NiFi.  By default the only configured user authentication method setup is via TLS certificates which requires you to to create a certificate for each user accessing NiFi.  This certificate would be loaded in to your browser and not NiFi itself (the public certificate for your user's private certificate would need to be added to the NiFi truststore in order for NiFi to trust the cert).   NiFi can also be configured to support kerberos, ldap, openId connect, etc alternative methods for user authentication. https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#user_authentication Once a user is successfully authenticated, that user must be authorized before they can even see the NiFi UI. https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#multi-tenant-authorization I understand you have already started down this path, but I recommend you carefully read above document links for more details.         ... View more

Agree with @MattWho . You can use the nifi expression language to choose the certain files from your source. Based on the date or file name you can filter out the files using FetchSFTP processor. i can see in the latest version of nifi expression language is false in ListSFTP processor. to handle your case you have to do it in two steps ListSFTP ==> FetchSFTP (If you want to put dynamic date filters in your source directories).   e.g. you can mention the below in Remote file property of FetchSFTP 0000000001_${now():toNumber():minus(86400000):format('yyyyMMdd')}235959_filename this will gives us output as 0000000001_20200112235959_filename.   Regards Nitin     ... View more

Thanks Steven, I removed the hostname and it started working after I set hostname box empty. ... View more