@Racketmojster  So you do want to ingest all the files within a specific parent directory path.  Use case details are important here.  I was not sure if perhaps your script fetched the files when passed a folder date string. There are certainly challenges in with using MergeContent processor which may require modifications to your script to make it work. MergeContent provides numerous Merge Formats for how you want this processor to merge the content from multiple source FlowFiles: The default is binary concatenation which just appends the bytes from one FlowFile to the end of the previous FlowFiles content.  You could specify a delimiter to keep track of filename of each section of bytes and to mark where one file's  bytes start and end.  This would require your script to parse this binary concatenated content and split the files by delimiter and obtain the individual content's filenames from the delimiter.  This is rather messy. You could use ZIP or TAR formats to merge the files in to a TAR or ZIP files that retains the directory structure and filenames.  This format would require that your script can then untar or unzip the content in order to process the individual files within the bundle.  This is a less messy option. Once you have decided on the merge format, you need to configure the MergeContent to make sure all the files from within a unique parent directory (unique date directory) are merged with one another.  This is where the "Correlation Attribute Name" setting is used.  All source FlowFiles where the value from the configured FlowFile attribute is identical will be placed in same MergeContent bin. For example, you might use "absolute.path" in this property, but you'll need to make sure this will work in your use case since i don't know what your complete source structure tree looks like.  You could also use an UpdateAttribute processor before MergeContent to create a new FlowFIle attribute with just the extracted date string from absolute.path and then use that new FlowFile attribute as your correlation attribute setting. Now you need to make sure MergeContent does not merge a bin before all expected FlowFiles have been added to bin.  The way MergeContent processor works is that when executed it reads FlowFile(s) from inbound connection and allocates it/them to a bin.  After allocating that FlowFile to a bin, it checks to see if that bin is eligible to be merged.  Since NiFi executes processor by default as fast as possible (milliseconds count here), it is possible at the very moment it looks at the inbound connection not all FlowFiles from a specific directory may be present yet.  Determining if a bin should be merged is controlled by the  "Minimum Number of Entries", "Minimum Group Size", and "Max Bin Age" configuration properties.  If both min settings are satisfied a bin will be merged.  If both mins have not been met, but the max bin age (age of bin since first FlowFile was allocated to bin) has been reached or exceeded, the bin will be merged.  With the defaults you can see how it is very easy that a bin may get merged before all necessary FlowFiles have been added to it.    The "Maximum number of Bins" is also very important here.  If you have 5 bins (default) and you have 20 source dated directories.  it becomes possible that FlowFiles get allocated by that correlation attribute to 5 different bins, and then another FlowFile is processed that belong to none of those 5 bins (6th dated directory).  What happens in this scenario is that MergeContent forces the merge of the oldest bin regardless or mins or max bin age in order to free a bin to allocated the next FlowFile to.   If all your source unique dated directories contain same number of files, setting the min number entries is easy, but that is probably not the case.  So what a user would typically do is set mins num entries to a value larger then any source directory would ever have to prevent bin from merging until the max bin age is reached.  This introduces the latency in your dataflow by that configured max bin age. Hope this helps you understand the MergeContent processor configuration options better within the context of your specific dataflow needs. The above does not to me seem like an efficient dataflow.   ListFile is listing files from the local filesystem, the fetchFile fetches the content for those listed files and adds it to the FlowFile.  Then if you use the MergeContent to create a tar or zip,  your script is going to need to untar or unzip that bundle somewhere to process the files. So you are effectively reading content rom a directory, which means writing content to NiFi's content repository, then tar/zipping which is another write to NiFi's content repository, then your script is untar/unzip that files (another write to somewhere local) in order to process the files. That is why in my original response i suggested avoiding the FetchFile and only use the ListFile to get the unique date source directory and pass that absolute path to your script that processes the files directly our of the source path.  Reduces latency and a lot of disk IO.  Please help our community thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt ... View more

@jpconver2  @RAGHUY  All 1.x versions of NiFi do not support rolling upgrades.   With the major release of the NiFi 2.x versions, NiFi added rolling upgrade support as part of NIFI-12016 - Improve leniency for bundle compatibility to allow for rolling upgrades. The above Apache NiFi jira does a great job of explaining why historically in the NiFi 1.x branch this was not implemented.  Above new feature improvement is included in NiFi 2.0.0-M1 (NiFi 2.0.0 milestone 1) or newer. Please help our community grow. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt ... View more

@SAMSAL  This is not a new problem, but rather something that has existed with NiFi on Windows fro a very long time.  You'll need to avoid using space in directory names or warp that directory name in quotes to avoid the issue. NIFI-200 - Bootstrap loader doesn't handle directories with spaces in it on Windows   Please help our community thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt ... View more

@Lorenzo  Based on log output shared the Spnego based authentication was successful and you have an authorization problem for your Spnego authenticated user. NiFi Authorization is case sensitive, so the user identity returned via kerberos-provider login provider is likely not the exact same user identity string returned via Spnego based kerberos authentication.  "myuserad" is a different user identity then "myaduser" and different user identity then "MyAduser" and different user identity then "myaduser@domain.com" and .etc... NiFi provides identity mapping properties which can be used to manipulate the user identity returned by different user authentication methods before the final manipulated user identity is passed over the the NiFi authorizer to check for proper authorization(s).  These are added to the nifi.properties file: Identity Mapping Properties NOTE: keep in mind that mapping patterns are checked against the user identity output during authentication in an alpha-numeric order.  First pattern (regex) to match has its value and transform applied at which time not additional mapping patterns will get evaluated.  So as your pattern regular expressions get more generic the farther down the alpha-numeric list they need to be. Please help our community thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt ... View more

@jonay__reyes  or @DeepakDonde  The issue you are encountering if caused by a code change so the the invokeHTTP would encode URLs automatically.  This issue is triggered if your URL is already encoded.  The URL encoding change will convert all '%' to '%25'. Workarounds/solutions: Remove your url encoding and allow processor to do that encoding. If your URL is not URL encoded already and happens to contain '%' in the URL you can do the following: If using Apache NiFi 1.25 verions: Upgrade to NIFi 1.26 which contains fix NIFI-12842. (now released) Try adding Apache NiFi 1.26 version of the NiFi standard nar to your 1.25 install. Downgrade to Apache NiFi 1.24 If using Apache NiFi 2.0.0-M2 Wait for upcoming release of NiFi 2.0.0-M3 and upgrade, which will contain fix. Downgrade to Apache NiFi 2.0.0-M1 Try adding  Apache NiFi 2.0.0-M1 Standard nar to your 2.0.0-M2 install and switch to using older 2.0.0-M1 invokeHTTP processor. Please help our community thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt ... View more