@MattWho ,   Thank you for the details. I can understand your point that same file cannot be accessed across cluster nodes. But, I am using Nifi is single node (without cluster) and was thinking that this should work.   Yes, i do use "Update Attribute" with the below file name conventions. This generates separate flow file for every message. I am trying to have this as one single file per node.   ${filename:replace(${filename},"FileName_"):append(${now():format("yyyy-MM-dd-HH-mm-ss")})}.Json   Thank you ... View more

@Siraj  Does your ConsumeMQTT processor produce all output FlowFIles with the same Filename? Your MergeContent processor will merge if both configured min settings are satisfied at the end of execution.  If you have your MergeContent processor configured to run as fast as possible (run schedule set to 0 sec default). It may upon execution only see one FlowFile on incoming connection at that moment in time and put only one FlowFile in a bin and merge just that one FlowFile since you set your min num entries to "1".   I suggest you edit your MergeContent as follows: 1. Configure "Correlation Attribute Name" to "filename".  2. Perhaps increase your min setting from 1 to some higher value. 3. Always set "Max Bin Age".  (This is your forced merge property, it will force a bin to merge even if it has not reach both min values within this configured amount of time.) 4. Make sure you have enough bins to accommodate the expected number of unique filenames plus 1 extra.  (If all bins have allocated FlowFiles to them and the next FlowFile cannot be added to an existing bin, the oldest bin will be forced to merge to free a bin).   The NiFi putFile processor does not support append.  Append actions can cause all kinds of issue especially with a NiFi cluster where the target directory of the putFile is mounted to all the NiFi nodes. You can't have multiple nodes trying to write/append to the same file at the same time. My suggestion would be to use the UpdateAttribute processor before your putFile to modify the filename attribute.   Perhaps prepend a uuid to the filename to ensure uniqueness across multiple files or NiFi nodes (if clustered). ${UUID()}-${filename}   Hope this helps you, Matt ... View more

@Former Member    This should not be happening unless you are clearing NiFi state between restarts or NiFi is having a state issue.  When the listSFTP processor runs, it will record state so it does not list the same files more than once. If the Files being written to the SFTP server from which you are listing are not using dot rename as they are being written, it is possible your ListSFTP processor is listing them while they are still being written to.  Since state is based off timestamps, a file still being written to will have its timestamp updated between executions of the listSFTP processor.  This would result in the file being listed again. If the standard ssh dot rename method is not being used, you may need to "minimum File Age" property in listSFTP to make sure only files whose timestamp has not updated for X amount of time are listed.  This of course depends on your version of NiFi being used since this property was introduced as part of https://issues.apache.org/jira/browse/NIFI-5977 There is also the possibility if you are using NiFi 1.8 you are hitting this known bug: https://issues.apache.org/jira/browse/NIFI-5849   If you found a solution that worked, please tae a moment to accept the solution to resolve this community question.   Thank you, Matt ... View more

@saivenkatg55    NiFi musty be secured before you can use any form of authentication and authorization.   Along with securing NiFi you can configure NiFi's login-identity-providers.xml to use the "ldap-provider" for user authentication.   Here are some useful links: Securing NiFi: https://docs.cloudera.com/HDPDocuments/HDF3/HDF-3.4.1.1/nifi-authentication/content/nifi_authentication.html Setting up ldap-provider: https://docs.cloudera.com/HDPDocuments/HDF3/HDF-3.4.1.1/nifi-security/content/ldap_login_identity_provider.html Setting up authorization: https://docs.cloudera.com/HDPDocuments/HDF3/HDF-3.4.1.1/nifi-authorization/content/nifi_authorization.html Hope this helps, Matt ... View more

HI!! the problem was in CSVReader the decision was to change Schema Access Strategy after this manipulation everything began to work.   ... View more

Thanks Matt for the quick response. ... View more

Sure thanks. @MattWho. it works! ... View more