@Kilynn  I see that you have set some very high values (330 GB) for your NiFi heap space.  This is really not recommended.  Can also see from what you have shared that the particular NiFi instance has been up for ~2.5 hours and within that 2.5 hour timeframe it has been in a stop-the-world state due to Java Garbage Collection (GC) for ~20 minutes.  This means that your NiFi is spending ~13% of its uptime doing nothing by stop-the-world garbage collection.  Is there a reason you set your heap this high?  Setting large heap simply because your server has the memory available is not a good reason.  Java GC kicks in when heap usage is around 80% usage.  When an object in heap is no longer being used it is not actually cleaned out of heap and space reclaimed at that time. Space is only reclaimed via GC.  The larger the heap the longer the GC events are going to take.   Seeing as how you current heap usage is ~53%, I am guessing with each GC event a considerable amount of heap space is being released.  Long GC pauses (stop-the-world) can result in node disconnection because NiFi node heartbeats are not being sent. When NiFi seems to be restarting with no indication at all in the nifi-app.log, you may want to take a look at your systems /var/log/messages file for any indications of the Out Of Memory (OOM) killer being executed.  With your NiFi instance being the largest consumer of memory on the host, it would be the top pick process by the OS OOM Killer when available OS memory gets too low. In most case you should not need more the 16GB of heap to run most dataflows.  If you are building dataflows that utilize a lot of heap, I'd recommend taking a close look at your dataflow designs to see if there are better design choices.  This that can lead to large heap usage is creating very large FlowFile attributes (for example,  extracting large amount of a FlowFile's content in to a FlowFile attribute).  FlowFile attributes all live inside the JVM heap.  Some other dataflow designs elements that can lead to high heap usage include: - Merging a very large number of FlowFiles in a single Merge processor.  Better to use multiple merge processors in series with first merging up 10,000 to 20,000 FlowFiles and then second merging those in to even larger files - Splitting a very large file in a single split processor resulting in a lot of FlowFiles produced in a single transaction.  Better to use multiple or even look at ways of processing the dat without needing to split the file in to multiple files (look at using the available "record" based processors) - Reading in entire content of a large FlowFile in to memory to perform action on it.  Like ExtractText processor configured for entire text instead of line-by-line mode. While I 100% agree that you should be looking in to your "thread" allocation choices in your nifi.properties file.  Many of them seem unnecessarily high for a 7 node cluster.  You should understand that each node in your cluster executes independently of the others, so the settings applied pertain to each node individually.  Following the guidance in the NiFi app guide (can be found in NIFi UI under help or on Apache NiFi site) is strongly advised.  If you found this response assisted with your query, please take a moment to login and click on "Accept as Solution" below this post. Thank you, Matt ... View more

@LejlaKM    Sharing your dataflow design and processor component configurations may help get you more/better responses to your query. Things you will want to look at before and when you run this dataflow: 1. NiFi heap usage and general memory usage on the host 2. Disk I/O and Network I/O 3. NiFi Host CPU Utilization (If your flow consumes 100% of the CPU(s) during execution, this can lead to what you are observing.  Does UI functionality return once copy is complete?) 4. Your dataflow design implementation including components used, configurations, concurrent tasks etc. While most use cases can be accomplished through dataflow implementations within NiFi, not all use cases are a good fit for NiFi.  IN this case your description points at copying a large Table from One Oracle DB to another.  You made not mention of any filtering, modifying, enhancing, etc being done to the Table data between this move which is where NiFi would fit in.  If your use case is a straight forward copying from A to B, then NiFi may not be the best fit for this specific us case as it will introduce unnecessary overhead to the process.  NiFi ingest content and writes it a content_repository and creates FlowFiles with attributes/metadata about the ingested data stored in a FlowFile_repository.  Then it has to read that content as it writes ti back out to a destination.   For simple copy operations where not intermediate manipulation or routing of the DB contents needs to be done, a tool that directly streams from DB A to DB B would likely be much faster. If you found this response assisted with your query, please take a moment to login and click on "Accept as Solution" below this post. Thank you, Matt ... View more

@Neil_1992  I strongly recommend not setting your NiFi heap to 200GB. Java reserves the XMS space and grows to the XMX space as space is requested. Java Garbage Collection (GC) execution to reclaim heap no longer being used does not kick in to ~80% of heap is used. That means GC in your case would kick in at around 160+ GB of used heap.  All GC execution is stop-the-world activity which means your NiFi will stop doing anything until GC completes. This can lead to long pauses resulting node disconnections, issues with timeouts with dataflows to external services ,etc. When it comes to flow.xml.gz file, you are correct that it is uncompressed and loaded into heap memory. The flow.xml.gz contains: Everything you add via the NiFi UI to the canvas (processors, RPG, input/output ports, funnels. labels, PGs, controller services, reporting tasks, connections, etc.).  This includes all the configuration of fro each of those components. NiFi Templates are also stored in the flow.xml.gz, uncompressed and loaded in to heap as well.  Once a NiFi template is created, it should be downloaded, stored outside of NiFi, and local copy of template inside of NiFi deleted.     As far as your specific flow.xml.gz, I see a closing tag "</property>" following that indicates that some component has a property which typically consists of a "name" and "value" with the huge null laced null strings in the value field.  I'd scroll up to see which component this property belongs to and then check why this value was set.  Maybe it was a copy paste issue?  Maybe this is just part of some template that was created with this large string for some purpose?  Nothing here says with any certainty that there is a bug. If you found this response assisted with your query, please take a moment to login and click on "Accept as Solution" below this post. Thank you, Matt ... View more

@Griggsy  The "Advanced" UI capability in the UpdateAttribute processor allows you to setup rules to accomplish exactly what you are describing here in your use case.  When you right click on the UpdateAttribute processor you will see the "Advanced" button in the lower left corner. From the Advanced UI you can configure conditional rules. 1. Click "+" to the right of "Rules" to create your first rule. 2. Then click "+" to right of "Conditions" to create your boolean NiFi Expression Language (NEL) statement(s).  When using multiple statements, all statements must result in true before the corresponding "Action(s)" would be applied. 3. Then click "+" to right of "Actions" to set create one or more actions that are applied if the condition(s) for this rule are true. The thing is that your example or condition does not make sense to me. You are returning the value from a FlowFile attribute named "hostname" and checking if it contains the string "Mickey" in both NEL statements.  However, for each you want to return a different output value?  Since both would always return either "true" or "false", which output would be desired? Here is example where i check if "Hostname" attribute contains "Mickey" and if that returns true I set FlowFile attribute "hive_database" to "cartoon".  I then have a second rule that checks the "Hostname" FlowFile attribute to see if it contains "Bond" and iid that returns true, I set "hive_database" FlowFile" attribute to "movie". The "conditions" and "actions" look similar for the "Bond" rule except "Mickey" is changed to "Bond" and cartoon is set to "movie" instead of "cartoon". So above rules give you an If/then capability. Another feature of using the advanced UI is the "else" capability.  Outside the "Advanced" UI set a attribute for "hive_database": If I were to define actions that set the "hive_database" FlowFile attribute and upon evaluating a FlowFile none of the configured rules that set the "hive_database" attribute are applied, the update Attribute processor will set apply the value set outside the advanced UI.  If a rule sets the the attribute "hive_database", then value defined outside advanced UI is ignored/not set. If you found this response assisted with your query, please take a moment to login and click on "Accept as Solution" below this post. Thank you, Matt   ... View more

@Elmer    What you described here is the use case for the NiFi Parameters capability. Parameters allow you to set configurations values within NiFi components thaT are unique per environment.  Parameters can be used in both non-sensitive and sensitive property ( password) fields. Then in each unique environment you set up a parameter context that uses the parameter name, but with the unique value for that specific environment.    Then when you port over your flow definition or template, you can configure the NiFi process group in which contains your ported over flow to use the cluster specific parameter context.  No need to go to each component and reconfigure them anymore.   If you found this response assisted with your query, please take a moment to login and click on "Accept as Solution" below this post. Thank you, Matt [1]https://nifi.apache.org/docs/nifi-docs/html/user-guide.html#Parameters ... View more

@sunny_de  NiFi can only support a single "authorizer".  Your authorizers.xml file attached shows that you have two configured: 1. managed-authorizer 2. file-provider <-- (legacy and deprecated in favor of above) Which authorizer is actually being used by NiFi's core is determined by what you have configured in the following property in the nifi.properties file: nifi.security.user.authorizer=   So make sure above is set to "managed-authorizer" This provider then references the: file-access-policy-provider  This provider is responsible for generating the "authorizations.xml" file which contains all the authorizations created for your users.  It will initially seed this file with the minimum authorizations needed for the configured "initial admin identity" string value (Case sensitive), NOTE: The authorizations.xml file is only generated if it does not already exist.  If this file already exists and changes made to the file-access-policy-provider will not be reflected in that file.  The expectation is once the initial admin and node policies have been seeded, all additional authorization are granted from within the NiFi UI. The "file-access-policy-provider" then references the: file-user-group-provider This provider is designed to generate the users.xml file if it does not exist and seed it with the initial user identities (initial admin and NiFi nodes if clustered). Seeing as how you're able to see the NiFi UI, that tells me that your initial admin was successfully created and given at least "view the user interface" policy permissions.  The initial policy setup will only grant polciy permission related to the canvas if a flow.xml.gz was already present when NiFi was secured or started for the first time.  In this case, it appears that you did not have a flow.xml.gz and NiFi created one on first startup which happens after the initial admin was created and authorized. You will however have admin policies assigned for your initial admin that will allow you to setup any additional policy you need.  I can see from the "operate" panel in the left hand side of the canvas that the following icon is NOT greyed out: This means that your authenticated user is authorized to set policies on the current selected component (in this case the root process group).  Click on this icon and grant yourself at a minimum the following policies: 1. view the component 2. modify the component 3. view the data 4. modify the data 5. view provenance   After applying above, the icon across the top of the UI should no longer be greyed out since you are now authorized to make modification to the root process group.  So now you can drag and drop items to the canvas.  When you add new components (processors, input/output ports, funnels, child process groups, etc) to the canvas they will inherit there policies from the parent process group, but you can always select any component and specifically set policies per component.    In a multi-team use NiFi, it is common to create a child process group for each team or person and authorized on the desired members to each process group.  This prevents user1 from modifying user 2's process group and components within it and vice versa. For more information on setting users, groups, and policies, here is the link to the NiFi documentation that talks about this topic: https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#config-users-access-policies   If you found this response assisted with your query, please take a moment to login and click on "Accept as Solution" below this post. Thank you, Matt ... View more

@MarcioMarchiori  Depending on which NiFi Expression Language (NEL) function you are using, either of the following two NEL statements should work for you: ${path:replaceAll("\\\\",'/')}/${filename} In above you are using the replaceAll function which expects a java regular expression.  Since '\" is an escape character and NiFi is a java application, the first \ escapes the second \ and the third \ escapes the forth \.   So \\\\ results in a regex of \\ being applied against the value present in the "path" variable. or ${path:replace('\\','/')}/${filename} In above you are using the replace function which does not path the first argument to be evaluated as a java regular expression but rather takes a string literal.  And since '\' is an escape character you need \\ which tells java to treat the second \ as the literal string value to search for. If you found this response assisted with your query, please take a moment to login and click on "Accept as Solution" below this post. Thank you, Matt ... View more

@Hafiz    The ExtractText processor will evaluate a Java regular expression containing capture group against the inbound FlowFile's content. Then creates FlowFile attributes by processor dynamic property name that is assigned the value from the capture group from that Java regular expression. Above would result in FlowFiles with attributes like:   Things to keep in mind. SplitText takes the inbound FlowFile and splits it in too many FlowFiles.  If you are producing a lot of splits from a single source FlowFile, it will have an impact of NiFi's heap usage during that process.  As each Split FlowFile is created, the FlowFile attributes/metadata fro each produced FlowFile (splits) is held in heap memory.  Once all splits are created, all those produced Split FlowFiles are committed to the downstream relationship. One on the relationship, NiFi can then swap as needed out of heap memory.  NiFi does this to avoid data duplication.  Let's say you have a split that is in progress and NiFi dies.  Since nothing has been committed to a downstream relationship yet, when NiFi is brought back online, it will reprocess the original FlowFile.  You can reduce heap usage by splitting your source File multiple times if it is large (more than 20,000 - 50,000 splits).  For example, split by every 5,000 lines in first SplitText and then by every 1 line in second SplitText. While NiFi does not hold FlowFile content in heap memory (Some processor will load content in to heap to execute on that content), FlowFile attributes/metadata is held in heap memory.  So the more attributes/metadata exists on a FlowFile, the more heap that FlowFile is going to use.  FlowFiles are held in connection between processor components.  NiFi has a connection swap threshold that is applied per connection.  The default is to produce swap files that contain 10,000 FlowFiles each (these swap files are for FlowFile attributes/metadata and not content since it is not always held in heap). So swap default set in nifi.properties file is 20,000.  This means the first swap file for a connection is generated connection reaches 20,000 queued FlowFiles on one node (if multi-node NiFi cluster, swap is per node and not across all nodes) Just keep above in mind when designing dataflows where you are splitting/merging, creating a lot of FlowFile Attributes, or creating FlowFile attributes with large values. If you found this response assisted with your query, please take a moment to login and click on "Accept as Solution" below this post. Thank you, Matt ... View more

@Jarinek    As part of changing the value of a parameter, all the components that are configured to use that parameter must be stopped and restarted.  Some components like controller services have dependent components like processors. So to stop such a controller service, the dependent processor must be stopped first.  When NiFi requests that a component stop, it transition in to an intermediate stage fo "stopping" where NiFi asks the thread to exit.  The UI would reflect component as stopped or disabled even though it is still in the transition stage. But that does not always guarantee a component will ever complete the transition from "stopping" to stopped.  Some client libraries used by components and not written by NiFi developers do support interrupting running threads.  In scenarios like this the the "stopping" thread kicked off by NiFi stays active against that component until the active processor thread finally exits (if it ever does).  And as i said that specific component thread never completing is out of NiFi's core control.  Restarting NiFi is only way to kill any threads that do not gracefully exit.   Things you can try here: 1. When you try to enable or delete a controller service, I'd expect the exception to provide the component UUID and host where it claims it is running.  You could go to the cluster UI and manually "disconnect" the node where it is still running.  Then open a web browser to that specific, now disconnected" node's UI.  Use the UUID of the component to locate that component and check its status.  Try disabling it and if that does not work, try restarting that node only.  If disabling the component worked, go back to UI of another node still in cluster and via the cluster UI request that the disconnected node reconnect.  While the exception might point out one host as the issue, it may be more than one host. The UI simply returns the first exception. 2. If above is not successful, have you tried restarting the entire cluster? 3. You could also inspect the the flow.xml.gz from each host to make sure they match.  Component state (started, stopped, enabled, and disabled) is retained in the flow.xml.gz, but is not included within the flow fingerprint used when connecting a node to the cluster.  As a connecting node is suppose to inherit the state from the cluster.  You could check each flow.xml.gz for the component UUID and see what its current recorded state is in the flow.xml.gz.  If you find one or more with a state of enabled/running and other nodes with an alternate state, simply copy the flow.xml.gz with the desired state to the nodes with the undesired state.  Make sure flow.xml.gz ownership and permissions are correct and restart your NiFi cluster. As far as to why a component never transitioned from state to another, that would require examining numerous thread dumps spaced apart by at least 5 minutes.  This would allow you to identify active thread making no progress over the course of those numerous thread dumps.  Then you would have to to dig in to those threads to see what they are waiting on or blocked by.  Sometimes there are limitation within some clients that NiFi has no control over. If you found this response assisted with your query, please take a moment to login and click on "Accept as Solution" below this post. Thank you, Matt ... View more