@Jagapriyan  The exception is telling you that the following properties were not configured in the bootstrap.conf file for your MiNiFi: nifi.minifi.sensitive.props.key= nifi.minifi.sensitive.props.algorithm=   Please help our community thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt ... View more

@alan18080  The Postgres DB only holds metadata and does not contain the actual datflow in NiFi-registry.  The version controlled dataflows are stored in the configured FlowPersistenceProvider.   So if you are not preserving the actual flow contents, then the metadata loaded from your postgresSQL will not find it after redeployment. I would also recommend upgrading your NiFi-Registry and NiFi versions. NiFi-Registry is going on 4 years old. Please help our community thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt ... View more

@Racketmojster  So you do want to ingest all the files within a specific parent directory path.  Use case details are important here.  I was not sure if perhaps your script fetched the files when passed a folder date string. There are certainly challenges in with using MergeContent processor which may require modifications to your script to make it work. MergeContent provides numerous Merge Formats for how you want this processor to merge the content from multiple source FlowFiles: The default is binary concatenation which just appends the bytes from one FlowFile to the end of the previous FlowFiles content.  You could specify a delimiter to keep track of filename of each section of bytes and to mark where one file's  bytes start and end.  This would require your script to parse this binary concatenated content and split the files by delimiter and obtain the individual content's filenames from the delimiter.  This is rather messy. You could use ZIP or TAR formats to merge the files in to a TAR or ZIP files that retains the directory structure and filenames.  This format would require that your script can then untar or unzip the content in order to process the individual files within the bundle.  This is a less messy option. Once you have decided on the merge format, you need to configure the MergeContent to make sure all the files from within a unique parent directory (unique date directory) are merged with one another.  This is where the "Correlation Attribute Name" setting is used.  All source FlowFiles where the value from the configured FlowFile attribute is identical will be placed in same MergeContent bin. For example, you might use "absolute.path" in this property, but you'll need to make sure this will work in your use case since i don't know what your complete source structure tree looks like.  You could also use an UpdateAttribute processor before MergeContent to create a new FlowFIle attribute with just the extracted date string from absolute.path and then use that new FlowFile attribute as your correlation attribute setting. Now you need to make sure MergeContent does not merge a bin before all expected FlowFiles have been added to bin.  The way MergeContent processor works is that when executed it reads FlowFile(s) from inbound connection and allocates it/them to a bin.  After allocating that FlowFile to a bin, it checks to see if that bin is eligible to be merged.  Since NiFi executes processor by default as fast as possible (milliseconds count here), it is possible at the very moment it looks at the inbound connection not all FlowFiles from a specific directory may be present yet.  Determining if a bin should be merged is controlled by the  "Minimum Number of Entries", "Minimum Group Size", and "Max Bin Age" configuration properties.  If both min settings are satisfied a bin will be merged.  If both mins have not been met, but the max bin age (age of bin since first FlowFile was allocated to bin) has been reached or exceeded, the bin will be merged.  With the defaults you can see how it is very easy that a bin may get merged before all necessary FlowFiles have been added to it.    The "Maximum number of Bins" is also very important here.  If you have 5 bins (default) and you have 20 source dated directories.  it becomes possible that FlowFiles get allocated by that correlation attribute to 5 different bins, and then another FlowFile is processed that belong to none of those 5 bins (6th dated directory).  What happens in this scenario is that MergeContent forces the merge of the oldest bin regardless or mins or max bin age in order to free a bin to allocated the next FlowFile to.   If all your source unique dated directories contain same number of files, setting the min number entries is easy, but that is probably not the case.  So what a user would typically do is set mins num entries to a value larger then any source directory would ever have to prevent bin from merging until the max bin age is reached.  This introduces the latency in your dataflow by that configured max bin age. Hope this helps you understand the MergeContent processor configuration options better within the context of your specific dataflow needs. The above does not to me seem like an efficient dataflow.   ListFile is listing files from the local filesystem, the fetchFile fetches the content for those listed files and adds it to the FlowFile.  Then if you use the MergeContent to create a tar or zip,  your script is going to need to untar or unzip that bundle somewhere to process the files. So you are effectively reading content rom a directory, which means writing content to NiFi's content repository, then tar/zipping which is another write to NiFi's content repository, then your script is untar/unzip that files (another write to somewhere local) in order to process the files. That is why in my original response i suggested avoiding the FetchFile and only use the ListFile to get the unique date source directory and pass that absolute path to your script that processes the files directly our of the source path.  Reduces latency and a lot of disk IO.  Please help our community thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt ... View more