The GetSFTP processor is throwing the FlowFileAccessException when trying to read a file from a remote directory. If I look at the error stack trace I can see that at StreamUtils.copy is where IOException has happened. Is it something like file was moved when actually reading it ? Can you guys please help me understand why this issue occurring and how I can avoid it ? o.a.nifi.processors.standard.GetSFTP GetSFTP[id=7d5fe4b5-7b67-3779-8de7-00e903c20c34] Unable to retrieve file FILENAME due to org.apache.nifi.processor.exception.FlowFileAccessException: Unable to create ContentClaim due to java.io.IOException: error: org.apache.nifi.processor.exception.FlowFileAccessException: Failed to import data from com.jcraft.jsch.ChannelSftp$2@6e1e8c5e for StandardFlowFileRecord[uuid =a50179f7-b631-4aa8-9e74-a2dca44d89bd,claim=,offset=0,name=a50179f7-b631-4aa8-9e74-a2dca44d89bd,size=0] due to org.apache.nifi.processor.exception.FlowFileAccessException: Unable to create ContentClaim due to java.io.IOException: errororg.apache.nifi.processor.exception.FlowFileAccessException: Failed to import data from com.jcraft.jsch.ChannelSftp$2@6e1e8c5e for StandardFlowFileRecord[uuid=a50179f7- b631-4aa8-9e74-a2dca44d89bd,claim=,offset=0,name=a50179f7-b631-4aa8-9e74-a2dca44d89bd,size=0] due to org.apache.nifi.processor.exception.FlowFileAccessException: Unable to create ContentClaim due to java.io.IOException: error at org.apache.nifi.controller.repository.StandardProcessSession.importFrom(StandardProcessSession.java:3045) at org.apache.nifi.processors.standard.GetFileTransfer.onTrigger(GetFileTransfer.java:194) at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27) at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1162) at org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:205) at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:117) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: org.apache.nifi.processor.exception.FlowFileAccessException: Unable to create ContentClaim due to java.io.IOException: error at org.apache.nifi.controller.repository.StandardProcessSession.importFrom(StandardProcessSession.java:3038) ... 12 common frames omitted Caused by: java.io.IOException: error at com.jcraft.jsch.ChannelSftp$2.close(ChannelSftp.java:1532) at com.jcraft.jsch.ChannelSftp$2.read(ChannelSftp.java:1444) at com.jcraft.jsch.ChannelSftp$2.read(ChannelSftp.java:1364) at org.apache.nifi.controller.repository.io.TaskTerminationInputStream.read(TaskTerminationInputStream.java:62) at org.apache.nifi.stream.io.StreamUtils.copy(StreamUtils.java:35) at org.apache.nifi.controller.repository.FileSystemRepository.importFrom(FileSystemRepository.java:744) at org.apache.nifi.controller.repository.StandardProcessSession.importFrom(StandardProcessSession.java:3035) ... 12 common frames omitted ... View more

I am using MiNiFi(0.5.0) to pick and transfer files from a Linux machine to NiFi(1.9.1). In few cases i can see duplicate files being transferred to NiFi. Flow is setup as below GetFile ->LogAttribute -> PutFile(archive) -> RemoteProcessGroup Log 1: minifi-app.log:2020-09-13 06:09:00,760 INFO [Timer-Driven Process Thread-4] o.a.nifi.remote.StandardRemoteGroupPort RemoteGroupPort[name=Input_Port_GES,targets=https://nifi1.myorg.com:7071/nifi] Successfully sent [StandardFlowFileRecord[uuid=345d9b6d-e9f7-4dd8-ad9a-a9d66fdfd902,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1599937413340-1, container=default, section=1], offset=1073154, length=237],offset=0,name=RandomFile1154.txt,size=237]] (237 bytes) to nifi://nifi1.myorg.com:7074 in 48 milliseconds at a rate of 4.74 KB/sec   Log 2 : minifi-app.log:2020-09-13 06:09:01,910 INFO [Timer-Driven Process Thread-5] o.a.nifi.remote.StandardRemoteGroupPort RemoteGroupPort[name=Input_Port_GES,targets=https://nifi1.myorg.com:7071/nifi] Successfully sent [StandardFlowFileRecord[uuid=f74eb941-a233-4f9e-86ff-07723940f012,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1599937413340-1, container=default, section=1], offset=1109014, length=237],offset=0,name=RandomFile1154.txt,size=237], StandardFlowFileRecord[uuid=522a4350-4cab-476c-a087-a3793101412e,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1599937413340-1, container=default, section=1], offset=1074338, length=235],offset=0,name=RandomFile1346.txt,size=235]] (472 bytes) to nifi://nifi1.myorg.com:7074 in 30 milliseconds at a rate of 14.9 KB/sec   In the above log its seen that RandomFile1154.txt file is transferred once at 2020-09-13 06:09:00,760 and then again at 2020-09-13 06:09:01,910 along with RandomFile1346.txt.   I went through the StandardRemoteGroupPort code and i can see that once the transfer is successful, session is committed and it should not be available for next transfer. I have added the logs to see if my GetFile picked the file twice, but this is not the case, the log printed only once. Please share your thoughts on this ... View more

We use NiFi mainly for file transfer, processing these files etc. The requirement is to pull reports from NiFi which has the statistics such as how many files were processed from NiFi, how many of them are successfully transferred to destination, how many are failed and waiting in different queues. The NiFi rest api documentation(https://nifi.apache.org/docs/nifi-docs/rest-api/index.html) has the API's to pull this information, but i did not find any info on how to use these API's. Is the API's exposed out of the box or should it be enabled somehow. Please share your suggestions on this. If there is any other way to extract the data statistics from NiFi please share them as well. ... View more

We want to store our sensitive information such as passwords, private keys in Vault and retrieve it from Vault on need basis, kindly share your inputs which is much needed. ... View more