@enam  The FetchSFTP processor only supports specifying a target directory path in which the source file on the SFTP server will be moved.  It does not support the renaming of the source file during that move. When you configure the filename in the "move destination directory" it creates that filename as a directory in which it puts the source file using the value from the FlowFiles "filename" attribute. Your option is to set "completion strategy to "Delete File" and then route the "success" relationship via two separate outbound connections.  One connection will continue down your existing dataflow path.  The second connection will feed an UpdateAttribute processor (used to change filename) and then a putSFTP processor to write it back the SFTP server in the new directory. The NiFi FlowFile routed to "success" relationship in connection 1 will retain original filename. The NiFi FlowFile routed to "success" relationship in connection 2 will have its original filename modified by the UpdateAttribute processor as follows before being written back to SFTP server in new folder via the putSFTP processor. Update Attribute configuration (click "plus" icon to add new dynamic property): Property = filename Value = ${filename:substringBeforeLast('.')}-${UUID}.${filename:substringAfterLast('.')} You can then auto-terminate the success relationship of the putSFTP processor.  terminating FlowFile down path of connection 2 will have no impact on FlowFile routed to connection 1.  Make sure that you set the target path in the putSFTP processor (path does not include filename). Please help our community thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt ... View more

@NIFI-USER  I am trying to fully understand your use case. So you have a primary Kafka and a backup Kafka for failover? This feels like an odd setup to me. So you are trying to plan for a use case where the entire primary Kafka cluster is down and NiFi will failover to sending publishing to another entirely different Kafka cluster? Option 1: I noticed that when you tried to use "retry" on the "failure" relationship, you configured it to use "Yield".  When a FlowFile fails, it is then in this case yields the processor execution for 1 sec (default yield duration from settings tab) and the FlowFile remains in priority slot 1.  So after yield duration that same FlowFile is attempted again. If it fails again a yield of 1 minute and 1 sec is applied to processor before FlowFile is processed again, and then on next retry that gets doubled to 2 mins and 2 secs, then doubled to 4 mins 4 secs, etc.  With the yield retry policy, it prevents processing of the next FlowFile in the inbound connection until the first has failed the configured number of retries.  This can really slow the process of moving FlowFiles to the connection containing the "failure" relationship.  Option 2: Instead you could try the "penalize" retry policy.  This policy applies a penalty time to the FlowFile when it fails and leaves it in inbound connection queue.  It does not yield the processor.  So the processor continues to get scheduled to execute.  Any "penalized" FlowFiles are ignored until penalty duration expires.  The default penalty duration is 30 secs configured on the settings tab (you may want to reduce this value for your use case.). It also doubles the length of penalization with each subsequent retry.  After the configured number of retries the FlowFiles is moved to the outbound connection containing the failure relationship.   Both the above strategies allow for unexpected failures to still have an opportunity to succeed to your primary Kafka with some delay. Option 3: The alternative is to stop using the "retry" capability on the failure relationship.  This means that every FlowFile that fails will immediately fail, get penalized and transferred to the "failure" relationship.  So in scenario where this is only a temp unexpected failure you'll have some FlowFiles going to your backup Kafka cluster. and other still going to yoru primary Kafka cluster.  FlowFiles routed to failure are automatically penalized also. Since you are not looping the failure relationship's connection back on the source processor, you may also want to set penalty duration to 0 sec in the settings tab so the that downstream secondary Kafka cluster processor will be able to execute on the FlowFile immediately instead of needing to wait for that penalty to expire. No matter which option above you choose to use, it is important to adjust the penalty and or yield duration settings to meet your use case needs. Resources: Relationships Tab Settings Tab Please help our community thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt ... View more

@Lorenzo  There is not enough information provided to provide a good response here. 1. Providing the full stack trace output from the nifi-app.log may provide details that are helpful here (challenging considering the NPE nature of the exception) 2. Are you sure the SMTP server supports username and password authentication and not OAuth2 based authentication instead? PutEmail Additional Details Please help our community grow. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt ... View more

@MikeH  Sounds like you are regularly ingesting a considerable number of files fro your local filesystem.   Is this a NiFi multi-node cluster or a single standalone instance of NiFi handling this use case? Both the GetFile and ListFile processors have a "Path Filter" property that takes a Java Regular expression.  You could add multiple processors each with a different regex so they each get from a subset of user sub-directories. You might consider using the ListFile along with FetchFile processors instead of the GetFile processor.   The ListFile processor produces zero byte FlowFiles (1 FlowFile for each file listed), this processor is then connected to a FetchFile processor which use attributes set on that source file to fetch the content and add it to the FlowFile.  With a NiFi cluster this design approach allows you to redistributed the 0 byte FlowFiles across all nodes in a NiFi cluster so the heavy work of reading in the content and processing each FlowFile is spread across multiple servers(NiFi cluster nodes).  With this approach you can also have many ListFile processor all feeding a single FetchFile. So perhaps you have a regex for all directories starting with A through C in one processor and another processor for D through F, etc... Please help our community thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt ... View more

@NeheikeQ  The first thing that stands out to me is that the version of NiFi-Registry you are using is not going to be compatible with the version of NiFi you are using.  NiFi introduced numerous new capabilities that would get tracked in NiFi-Registry, but that old NiFi-Registry version is not going to handle them (even if version control works, the stored flow definitions are going to be missing elements). Every time a change is made on the NiFi canvas the current flow.xml.gz ad flow.json.gz files are archived and new versions created.  So rolling back can be done by swapping to the archived flow.json.gz.   There are a few bugs that have been addressed sine 1.23 that address not disconnection and failure to rejoin related bugs.  There is not enough detail here to pinpoint an exact cause for your issue. Does your issue only happen with dataflow(s) imported from your old 0.7.0 version of NiFi-Registry?  Any particular flow design that always reproduces yoru issue?  Full error and stack traces from nifi-app.log? Any other errors or warns around same time in either nifi-app.log or nifi-user.log?  What about the nifi-request.log, what was the request made at time the exception occurs? I recommend upgrading to the latest 1.x release of Apache NiFi and to latest NIFi-Registry version to see if your issue persists.  Note: The flow.xml.gz is deprecated. It has been replaced by the flow.json.gz.  When NiFi is started it will load the flow.json.gz.  If the flow.json.gz does not exist, NiFi will load from the flow.xml.gz file at which time it will generate the flow.json.gz file from it.  Apache NiFi 1.16+ will still write out both the flow.xml.gz and flow.json.gz files whenever a changes is made on the UI.  In Apache NiFi 2.x versions, only the flow.json.gz will exist. Please help our community thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt ... View more

@Alexy  Without seeing your logs, I have no idea which NiFi classes are producing the majority of your logging. But logback is functioning exactly as you have it configured.  Each time the nifi-app.log reaches 500 MB within a single day it is compressed and rolled using an incrementing number.  I would suggest changing the log level for the base class "org.apache.nifi" from INFO to WARN.  The bulk of all NiFi classes begin with org.apache.nifi and by changing this to WARN to you will only see ERROR and WARN level log output from the bulk of the ora.apache.nifi.<XYZ...> classes. <logger name="org.apache.nifi" level="WARN"/> Unless you have a lot of exception happening within your NiFi processor components used in your dataflow(s), this should have significant impact on the amount of nifi-app.log logging being produced. Please help our community thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt ... View more