@imvn  I am finding it hard to follow your use case here. Your shared dataflow image shows: 1. ExecuteSQL (query data in source database) -->(produces a FlowFile that contains 0 or more rows) --> 2. RouteOnAttribute (evaluates the "executesql.row.count"? to see if it is "0"?  If so, routes the "0" row flowfiles to the "lines" relationship?.  Or are you auto-terminating within RouteOnAtrtibute if row count is "0" and lines relationship is used only if "executesql.row.count" is not "0"?) 3. I see the "lines" relationship is routed twice.  Once to another ExecuteSQL (deletes the data from the local database, i.e., destination) and another directly to PutDatabaseRecord (since PutDatabaseRecord has to inbound connections that will have a FlowFile, it will execute from both which is not what i think you want to happen.) Just considering the above, I think option 3 which utilizes the "FlowFile concurrency" and "outbound policy" settings on a process group would handle you timing needs.  Where your RouteOnAttribute goes in place of the ExtractText processor and you feed lines into the child Process Group. The question is what is the overall goal of this use case?  Are you trying to maintain an up-to-date copy of the source database in the destination database?   or are you trying to just copy rows added to source DB to the destination DB?  If so there are better dataflow designs for that. Please help our community thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt ... View more

@imvn  A NiFi FlowFile consists of two parts: FlowFile content - The content of a FlowFile is written to a content claim within the NiFI content repository.  A content claim is immutable. FlowFile Attributes/metadata - Attributes/metadata are written to the flowfile repository and persists in NiFi heap memory unless forced to swap due connection thresholds.  This metadata includes information about where to find the content among other things. How NiFi processor handle inbound and outbound FlowFiles is largely processor specific.  Fo processors that write output to the content of a FlowFiles, this may be handle two different ways depending on processor.  Some processor might have an "original" relationship where the original FlowFile referencing original inbound content claim gets routed while creating a new FlowFile with same attributes pointing to new content output in a different claim and routing to some other relationship like "success".   Other processor might not have an "original" relationship and instead decrement a claimant count on the original content claim and update the existing FlowFile metadata to point to the content created in the new content claim. The ExecuteSQL processor follows the latter process.   So you have a dataflow built like this if i understand correctly: ExecuteSQL (writes content to FlowFile) --> some processor/processors (extract bits from content to use for Delete) --> ExecuteSQL (performs delete but response is written as new content for the FlowFile) --> PutDatabaseRecord (has issues since original needed FlowFile content is no longer associated with the FlowFile). Option 1: Can you re-order yoru processors so you have ExecuteSQL. --> PutDatabaseRecord --> Extract content --> ExecuteSQL (Delete) This makes sure orginal content is persisted long enough to compete write to target DB. Option 2: ExecuteSQL --> ExtractContent --> Route "success" relationship twice (once to ExecuteSQL to perform delete ad second to PutDatabaseRecord to write to DB).  Similar to below example: You'll notice that the "matched" relationship has been routed twice.  When the same relationship is routed twice, NiFi clones the FlowFile (original goes to one connection and clone goes to the other.  Both FlowFiles reference the same content claim (which remember is immutable). When ExecuteSQL (delete) the executes on one of them it does not impact the content in the other one that is going to PutDatabaseRecord. If I am not clear on your use case, let me know. I was a bit confused on the "delete data from destination" part. Destination = the putDatabaseRecord configured DB destination? not clear why you would be deleting something there that has not yet been written. So if there is a dependency that the ExecuteSQL (Delete) completes before the PutDatabaseRecord executes, there is a third option that utilizes the "FlowFile concurrency" and "outbound policy" settings on a Process Group. The dataflow would look something like this: Inside the Process Group configured with "FlowFile concurrency = Single FlowFile Per Node" and "outbound policy = Batch Output" set, you would have this flow: So you Dataflow only allows 1 FlowFile to enter the PG at a time.  Within the PG, the FlowFile is cloned with one FlowFile routing to the output port and the other to the ExecuteSQL (delete).  the FlowFile queued to exit PG will not be allowed to exit PG until the FlowFile being processed by the ExecuteSQL (delete) is auto-terminated or routed to some other output port.   This makes sure that the PutDatabaseRecord processor does not process the FlowFile with the original content claim until your ExecuteSQL (delete) was executed. Please help our community thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt   ... View more

@Ashi  Potential option: What record Reader and record writers are you using in your UpdateRecord processor? What schema are you using for your records? In order to add a new field, that new field needs to be defined in the records schema.  In your case the schema must contain the field "devicename". Prior to UpdateRecord, you could use perhaps an ExtractText processor to extract the "rc01;rik2jc" value from the meHostName field to a flowfile.attribute. Then will you be able to use UdpateRecord to apply a value to that new record field in the record writer. Property: /devicename value: ${flowfile.attribute:substringAfter(';')} Please help our community thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt   ... View more

@Crags  You can not have both your NiFi-Registries linked to the same Git repository. NiFi-Registry only pushes to the git repository.  The only time NiFi-Registry would ever read from the Git Repository is on startup. So if you used two NiFi-Registries and  and were committing changes by both, you can cause issues with what is getting committed to your Git repo. What is more common is to have a single NiFi-Registry which is utilized by multiple NiFi deployments. QA NiFi builds some flow and when that flow is ready for production, it is committed to the NiFi-Registry.  That flow can the be imported from that single NiFi-Registry to the canvas of your PROD NiFi.  Now both NiFi instances are tracking to same flow in same registry.   You then start making local changes to that same version controlled Process Group (PG) in your QA NiFi.  The PG will indicate you have local changes. you then have a couple choices on how you want to use your shared NiFi-Registry: Wait until you have completed making all your changes and testing in QA before committing the next version to the shared registry.  At which time your Prod NiFi PG will indicate a newer version is now available in the shared NiFi-Registry.  You can then update your prod to that new version. Incrementally commit updated versions of the PG to the shared registry.  Your prod will show new version available, so you will want to create a process for what versions are prod ready to control when a new version is actually changed in your prod. About the UUID linkage... Your NiFi can have one or more defined registry clients and each of those defined registry clients gets an assigned UUID on the NiFi instance (will not be same UUID on every NiFi  that sets up same registry client).  NiFi stores everything on the canvas locally in the flow.json.gz file so it can be reloaded into NiFi heap on startup.  When you start version control on a PG, the flow (gets uuid) is added to a NiFi-Registry bucket (has UUID).  Locally on the NiFi within the flow.json.gz there is now a reference to a specific NiFi-Registry client (by its UUD), a specific bucket (by its UUID) and specific flow (by its UUID). Now considering scenario of a shared NiFi-Registry, the registry client on that NiFi will hav a different uuid even though it connects to same shared NiFi-Registry.  So using the registry client, you import a flow that flow from NiFi-Registry to the NiFi canvas.  Every component created from the import flow will get assigned UUIDs (will not match UUIDs assigned on other NiFi).  Those differences in UUIDs are not tracked as changes. This is why if you stop version control, you can't start version control again and  connect it back to an existing flow stored in NiFi-registry.  You also can't delete the registry client and re-create it as it too would get a different UUID (NiFi blocks removing a registry client if any PG are currently using it for version control for this reason). --------- Another option is to have a separate NiFi-Registry for each environment. When you are ready to move a flow from NiFi-Registry 1 (QA) to NiFi-Registry 2 (Prod), go into your QA NiFi-Registry, locate the flow and from the "actions" menu select export version, and select the version you want to export. You can then go to your prod NiFi-Registry and "import new flow".  Once imported you can go to yoru Prod NiFi and load that flow onto the canvas.   Later when you are ready in QA with a new version to push to prod, you can again export the prod ready version.  On prod NiFi-Registry, you can select the existing flow and from "actions" menu select "import new version".  This will allow you to add this flow as next version in Prod.  After doing so the version controlled PG(s) on your prod NiFi tracking against that flow will report a new version is available. This second option allows your have better control over what changes make it to your Prod deployment.  You could also script rest-api calls to automate these steps if you wanted. ------ Please help our community thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt ... View more

@abhinav_joshi  You should have been able to right click on the "Ghost" processor and select "change version" option.  This would have presented you with all the available versions in your NiFi installation.  Simply select the one you want to use would resolve your issue. While this work great when you only have a few ghost processor created from your dataflow, it can be annoying to follow these steps for many components. The question here is why does you deployment of NiFi have multiple versions of the same NiFi nar installed.  NiFi would not ship this way, so that means that additional nar(s) of different versions where added to your NiFi lib directory or to the NiFi extensions directory.  You should remove these duplicate nars to avoid running into this issue again. When only one version exists, dataflow imported/loaded with older versions will auto switch to version used in the NiFi in which dataflow was loaded (this may mean and older or newer version of nar classes).  Please help our community thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt ... View more

@rizalt  There is very little detail in your post. NiFi will run as whatever user is used to start it unless the "run.as" property is set in the NiFi bootstrap.conf file. If the user trying to execute the "./nifi.sh start" command is not the root user and you set the "run.as" property to "root", that user would need sudo permissions in linux to start NiFi as the root user.    The "run.as" property is ignored on Windows where the service will always be owned by user that starts it. NOTE: Starting the service as a different user then it was previously started at will not trigger a change in file ownership in NiFi directories. You would need to update file ownership manually be starting as a different issue (this includes all NiFi's repositories). While "root" user has access to all files regardless of owner, issues will exist if no root user launches app and files are owned by another user including root. Please help our community thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt ... View more