@Venkikancharla    You are absolutely correct that CFM 2.1.2 is also impacted by this. While CFM 2.1.2 is based off Apache NiFi 1.13.2, it includes many bug fixes that eventually went in to Apache NiFi 1.14.  One of those changes happened to be NIFI-8723. Full list of changes made on top of Apache NiFi 1.13.2 in CFM 2.1.2 can be found here: https://docs.cloudera.com/cfm/2.1.2/release-notes/topics/cfm-fixed-issues.html CFM 2.1.3 will include https://issues.apache.org/jira/browse/NIFI-8938 Thank you, Matt ... View more

@Ankit13  I would still use Cron scheduling on the PutFile processor, but rather than just having it run once at say hour 7, I'd schedule it to run every second starting at hour 7.  That may it starts putting files at hour 7 and continues to put files all the way until 07:59:59.   Then it stops executing until the next day. http://www.quartz-scheduler.org/documentation/quartz-2.3.0/tutorials/crontrigger.html Hope this helps,  Matt ... View more

if possible dynamic set parameter,  ... View more

@Yemre    The ability to dynamically fetch secrets/passwords form an external source is not something that exists currently. Doing so would require modification with the every component class that uses sensitive properties. There is some progress in this path however: https://issues.apache.org/jira/browse/NIFI-5481   This new feature handles pulling secrets from an external vault, but is a NiFi core level feature and does not extend in to individual flow component level. I recommend raising an Apache NiFi Jira with your specific request.  https://issues.apache.org/jira/projects/NIFI/ If you found this response assisted with your query, please take a moment to login and click on "Accept as Solution" below this post. Thank you, Matt ... View more

@AnnaBea  Let me make sure I am clear on your ask here: 1. You have successfully split your source file in to 3 parts (header line, body line(s), and footer line). 2. You have successfully modified all three split files as needed. 3. You are having issues re-assembling the three split files back in to one file in order of header, body, footer using MergeRecord processor? With this particular dataflow design, the MergeRecord processor is not likely what you want to use.  You probably want to be using the MergeContent processor instead with a "Merge Strategy" of  "Defragment".  But to get these three source FlowFiles merged in a specific order would require some additional work in your upstream flow.  In order to use "Defragment" your three source FlowFiles all would need o have these FlowFile Attributes: fragment.identifier All split FlowFiles produced from the same parent FlowFile will have the same randomly generated UUID added for this attribute fragment.index A one-up number that indicates the ordering of the split FlowFiles that were created from a single parent FlowFile fragment.count The number of split FlowFiles generated from the parent FlowFile   1. Add one UpdateAttribute processor before your RouteText and configure it to create the "fragement.identifier" attribute with a value of "${UUID()}" and another Attribute "Fragment.count" with a value of "3".  Each FlowFIle produced by RouteText should then have these two attribute set on it. 2. Then add one UpdateAttribute processor to each of teh 3 flow paths to set the "fragment.index" attribute uniquely per each dataflow path.  value=1 for header, value=2 for body, and value=3 for footer. 3. Now the MergeContent will have what it needs to bin these three files by the UUID and merge them in the proper order. There are often times many ways to solve the same use case using NiFi components.  Some design choices are better than others and use less resources to accomplish the end goal. While above is one solution, there are others I am sure.  Cloudera's professional services is a great resource that can help with use case designs. If you found this response assisted with your query, please take a moment to login and click on "Accept as Solution" below this post. Thank you, Matt ... View more

@Apoo  The EvaluateJsonPath processor dynamic properties do not support NiFi Expression language, so being able to pass dynamic strings to these dynamic properties from FlowFile attributes is not possible.   The dynamic properties only support NiFi parameters. You may want to raise an Apache NiFi jira requesting adding NiFi EL support to these dynamic properties or even contribute the the open source code if you so choose. Thank you, Matt ... View more

Thanks Matt, It is very helpful ... View more

Thanks heaps!, @MattWho. Much appreciated. This solved my issue. ... View more

@AA24  The easiest way to accomplish this is to use the PutDistributedMapCache processor in one flow to write the attributes values you want to share to a cache server and on your other flow use the FetchDistributedMapCache processor to retrieve those cached attributes and add them to your other FlowFiles that need them. Another option is to use the MergeContent processor. On flow one where it looks like you are extracting your session_id and job_id you would use the ModfiyBytes processor to zero out the content leaving you with a FlowFile that only has attributes and then use MergeContent to combine this FlowFile with the FlowFile in your second flow. In the MergeContent processor you would configure "Attribute Strategy" to use "Keep All Unique Attributes".   If you found this response assisted with your query, please take a moment to login and click on "Accept as Solution" below this post. Thank you, Matt ... View more