Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

MiNiFi is sending duplicate files to NiFi

avatar
Explorer

I am using MiNiFi(0.5.0) to pick and transfer files from a Linux machine to NiFi(1.9.1). In few cases i can see duplicate files being transferred to NiFi.

Flow is setup as below

GetFile ->LogAttribute -> PutFile(archive) -> RemoteProcessGroup

Log 1:
minifi-app.log:2020-09-13 06:09:00,760 INFO [Timer-Driven Process Thread-4] o.a.nifi.remote.StandardRemoteGroupPort RemoteGroupPort[name=Input_Port_GES,targets=https://nifi1.myorg.com:7071/nifi] Successfully sent [StandardFlowFileRecord[uuid=345d9b6d-e9f7-4dd8-ad9a-a9d66fdfd902,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1599937413340-1, container=default, section=1], offset=1073154, length=237],offset=0,name=RandomFile1154.txt,size=237]] (237 bytes) to nifi://nifi1.myorg.com:7074 in 48 milliseconds at a rate of 4.74 KB/sec

 

Log 2 :

minifi-app.log:2020-09-13 06:09:01,910 INFO [Timer-Driven Process Thread-5] o.a.nifi.remote.StandardRemoteGroupPort RemoteGroupPort[name=Input_Port_GES,targets=https://nifi1.myorg.com:7071/nifi] Successfully sent [StandardFlowFileRecord[uuid=f74eb941-a233-4f9e-86ff-07723940f012,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1599937413340-1, container=default, section=1], offset=1109014, length=237],offset=0,name=RandomFile1154.txt,size=237],
StandardFlowFileRecord[uuid=522a4350-4cab-476c-a087-a3793101412e,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1599937413340-1, container=default, section=1], offset=1074338, length=235],offset=0,name=RandomFile1346.txt,size=235]] (472 bytes) to nifi://nifi1.myorg.com:7074 in 30 milliseconds at a rate of 14.9 KB/sec

 

In the above log its seen that RandomFile1154.txt file is transferred once at 2020-09-13 06:09:00,760 and then again at 2020-09-13 06:09:01,910 along with RandomFile1346.txt.

 

I went through the StandardRemoteGroupPort code and i can see that once the transfer is successful, session is committed and it should not be available for next transfer.

I have added the logs to see if my GetFile picked the file twice, but this is not the case, the log printed only once.

Please share your thoughts on this

1 ACCEPTED SOLUTION

avatar
Super Mentor
hide-solution

This problem has been solved!

Want to get a detailed solution you have to login/registered on the community

Register/Login
1 REPLY 1

avatar
Super Mentor
hide-solution

This problem has been solved!

Want to get a detailed solution you have to login/registered on the community

Register/Login