Support Questions

Find answers, ask questions, and share your expertise
Announcements
Check out our newest addition to the community, the Cloudera Data Analytics (CDA) group hub.

How to use Conflict Resolution Strategy in PutS3Object ?

Hi All,

In mynifi flow i have two processor one is GetFTP and PutS3Object . consider i have one file in FTP

a.txt .after the data get into the S3 the a.txt's timestamp is 12:00:00 in S3 after sometime again one file 'b.txt' is put into ftp now the S3 have two files as below but the timestamp in the S3 is changed for both a.txt and b.txt

a.txt 12:01:00

b.txt 12:01:00

1 ACCEPTED SOLUTION

Mentor

@Thangarajan Pannerselvam

If your GetFTP processor is configured with "delete original" set to false, every time this processor runs it will pull all th the files it finds including those pulled in the last run of the getFP processor.

The ListFTP processor maintains state unlike the GetFTP processor. so if you replace your GetFTP with both ListFTP and FetchFTP processors, you will not see the same files pulled twice unless the timestamp on the files on the FTP server are updated.

Thanks,

Matt

View solution in original post

1 REPLY 1

Mentor

@Thangarajan Pannerselvam

If your GetFTP processor is configured with "delete original" set to false, every time this processor runs it will pull all th the files it finds including those pulled in the last run of the getFP processor.

The ListFTP processor maintains state unlike the GetFTP processor. so if you replace your GetFTP with both ListFTP and FetchFTP processors, you will not see the same files pulled twice unless the timestamp on the files on the FTP server are updated.

Thanks,

Matt

Take a Tour of the Community
Don't have an account?
Your experience may be limited. Sign in to explore more.