Skip to main content
thangarajanp
thangarajanp
Gold ⭐️
March 16, 2017
Solved

Fixing PutS3Object Overwriting Existing S3 Files: Shifting from GetFTP to ListFTP in Apache NiFi

Hi All,

In mynifi flow i have two processor one is GetFTP and PutS3Object . consider i have one file in FTP

a.txt .after the data get into the S3 the a.txt's timestamp is 12:00:00 in S3 after sometime again one file 'b.txt' is put into ftp now the S3 have two files as below but the timestamp in the S3 is changed for both a.txt and b.txt

a.txt 12:01:00

b.txt 12:01:00

MattWho
MattWhoAnswer
Titanium ⭐️⭐️⭐️
March 16, 2017

@Thangarajan Pannerselvam

If your GetFTP processor is configured with "delete original" set to false, every time this processor runs it will pull all th the files it finds including those pulled in the last run of the getFP processor.

The ListFTP processor maintains state unlike the GetFTP processor. so if you replace your GetFTP with both ListFTP and FetchFTP processors, you will not see the same files pulled twice unless the timestamp on the files on the FTP server are updated.

Thanks,

Matt

View in thread

1 reply

MattWho
Titanium ⭐️⭐️⭐️
MattWhoAnswer
Titanium ⭐️⭐️⭐️
March 16, 2017

@Thangarajan Pannerselvam

If your GetFTP processor is configured with "delete original" set to false, every time this processor runs it will pull all th the files it finds including those pulled in the last run of the getFP processor.

The ListFTP processor maintains state unlike the GetFTP processor. so if you replace your GetFTP with both ListFTP and FetchFTP processors, you will not see the same files pulled twice unless the timestamp on the files on the FTP server are updated.

Thanks,

Matt

Translate
    Terms & ConditionsAccessibility statement