@s198  Question for you: - If you leave the putSFTP processor stopped, run your dataflow so all FlowFiles queue in front of the putSFTP processor and then start the putSFTP processor, does the issue still happen?      - Does issue only happen when the flow is an all started/running state? Answers to above can help in determining if changing the run schedule will help here. Run Schedule details: - The run schedule works in conjunction with Timer Driven scheduling strategy.  This schedule setting controls how often a component will get scheduled to execute (different from when it actually executes.  Execution depends on available threads in the NiFi Timer Driven thread pool shared by all components). By default this is set to 0 secs which means that NiFi should schedule his processor as often as possible (Basically schedule it again as soon as an available concurrent task (concurrent tasks default is 1) is available to it.  To avoid CPU saturation here, NiFi builds in a yield duration if upon scheduling of a processor there is no work to be done (inbound connection(s) are empty).    Depending on load on yoru system and dataflow, speed of network, this could happen very quick meaning it scheduled, sees only one FlowFile in the inbound connection at time of schedule and processes only that one FlowFile instead of a batch.  It then closes that thread and starts a new one for next FlowFile instead of processing multiple FlowFiles in one SFTP connection.  By changing run schedule you are allowing more time between scheduling for FlowFiles to queue on the inbound connection so they get batch processed in a single SFTP connection. Run Duration details: Another option on processors is the run duration setting.  What this adjustment does is upon scheduling of a processor the execution will not end until the configured run duration has elapsed.  So lets say at time of scheduling (run schedule) there is one FlowFile in inbound connection queue (remember we are dealing with micro seconds here, so not something you can visualize yourself via the UI).  That execution thread will execute against that FlowFile, but rather then close out the session immediately committing the FlowFile to an outbound relationship, it will check inbound connection for another FlowFile and process it in same session.  It will continue to do this until the run duration is satisfied at which time all processed FlowFiles during that execution are committed to downstream relationship(s).  So Run Duration might be another setting you try to see if it helps with your issue.  If you try run duration, i'd set run schedule to default. You may also want to look at your SFTP server logs to see what is happening when the file rename attempts are failing. Please help our community grow. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt ... View more

@shiva239  Is it the exact same error? What is the full stack trace logged to the nifi-app.log? Thanks, Matt ... View more

@Jim_Steinebrey @shiva239  I would avoid adding any additional custom jars or nars to the NiFi lib directory.  This will complicate future upgrade efforts.  Instead you should create a new directory outside NiFi's default directory tree for your custom jars that your specific dataflow(s) depend on. Make sure that the custom driver directory is both reachable and readable by your NiFi's service user.  Then as commented above configure the "Database Drivers Location(s)" property in the DBCPConnectionPool controller service to point to that new directory. Please help our community thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt ... View more

@tmarkfeld  The NiFi UI will not be avialable until you see the following lines in the nifi-app.log: 2024-03-17 16:39:59,500 INFO [main] org.apache.nifi.web.server.JettyServer NiFi has started. The UI is available at the following URLs: 2024-03-17 16:39:59,500 INFO [main] org.apache.nifi.web.server.JettyServer https://localhost:8443/nifi Up until this time NiFi is still loading. I see you mentioned it stalls at unpacking the nars but eventually completes as you also said eventually you see NiFi listening on port 8443. You can use developer tools within your browser to see what calls have been made when you try to access your NiFi,  you could then share what calls were made and which it appears to be hanging on. Hope this helps, Matt ... View more

@Ytch  All components on the NiFi canvas are executed as the NiFi service user and not as the user currently authenticated into the NiFi service.   So what you should do is from each host in your NiFi cluster (do on every host since any one of the hosts can be elected as the primary node at any given time), open a command prompt window/console window, become the user that owns the NiFi process, and manually ssh/sftp to the target SFTP server.  You will likely be prompted to add the target SFTP server to your known_hosts file for the NiFi service user.  NiFi SFTP processor has no way of doing this interactive step.  After successfully adding the SFTP to the known_hosts file for the NiFi service user, go back and try to start the GetSFTP or ListSFTP processors again to see if your issue is resolved. If not, please share your GetSFTP and ListSFTP processor component configurations. Also check the nifi-app.log for any exceptions or log output related to these processors. If no log output, you could also try enabling debug in the NiFi logback.xml for these processor classes to see what additional log output may be produced that could be useful here. classes for these processors are: org.apache.nifi.processors.standard.GetSFTP org.apache.nifi.processors.standard.ListSFTP new log lines would look like this that you would add to logback.xml: <logger name="org.apache.nifi.processors.standard.GetSFTP" level="DEBUG"/> <logger name="org.apache.nifi.processors.standard.ListSFTP" level="DEBUG"/> Simply add them in logback.xml where you see similar lines already. Please help our community thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt ... View more

@double_w  NiFi state configuration is setup in the state-management.xml file located by default in NiFi's conf directory.   When you say that state is being lost, are you specifically having issues with missing "cluster" state, "local" state, or both?  Is your NiFi clustered or a standalone (non cluster) deployment? Default cluster state is stored in zookeeper (are you using an external zookeeper or the embedded zookeeper?).   Any chance that your NiFi-2.0 is configured to use the new "kubernetes-provider" for cluster state management?  If so, did you configure the "nifi.state.management.provider.cluster.previous" property in nifi.properties with the previous "zk-provider", so that on first startup, it migrates your cluster state from zk-provider to the empty kubernetes-provider? Default local-provider stores state in ./state/local.   Have you compared the state-management.xml file configurations between both environments? Please help our community thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt ... View more

@Ytch  For the "hostname" property you only provide the sftp server's fully qualified hostname.  This hostname needs to be resolvable by the NiFi host. What version of Apache NiFi are you using? What errors or exceptions do you see when you start the processor? Have you tried using the listSFTP processor instead? Please help our community grow. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt   ... View more