Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

How to use Conflict Resolution Strategy in PutS3Object ?

Solved Go to solution

How to use Conflict Resolution Strategy in PutS3Object ?

Hi All,

In mynifi flow i have two processor one is GetFTP and PutS3Object . consider i have one file in FTP

a.txt .after the data get into the S3 the a.txt's timestamp is 12:00:00 in S3 after sometime again one file 'b.txt' is put into ftp now the S3 have two files as below but the timestamp in the S3 is changed for both a.txt and b.txt

a.txt 12:01:00

b.txt 12:01:00

1 ACCEPTED SOLUTION

Accepted Solutions
Highlighted

Re: How to use Conflict Resolution Strategy in PutS3Object ?

Master Guru

@Thangarajan Pannerselvam

If your GetFTP processor is configured with "delete original" set to false, every time this processor runs it will pull all th the files it finds including those pulled in the last run of the getFP processor.

The ListFTP processor maintains state unlike the GetFTP processor. so if you replace your GetFTP with both ListFTP and FetchFTP processors, you will not see the same files pulled twice unless the timestamp on the files on the FTP server are updated.

Thanks,

Matt

View solution in original post

1 REPLY 1
Highlighted

Re: How to use Conflict Resolution Strategy in PutS3Object ?

Master Guru

@Thangarajan Pannerselvam

If your GetFTP processor is configured with "delete original" set to false, every time this processor runs it will pull all th the files it finds including those pulled in the last run of the getFP processor.

The ListFTP processor maintains state unlike the GetFTP processor. so if you replace your GetFTP with both ListFTP and FetchFTP processors, you will not see the same files pulled twice unless the timestamp on the files on the FTP server are updated.

Thanks,

Matt

View solution in original post

Don't have an account?
Coming from Hortonworks? Activate your account here