Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

HDF 3.4.1 Nifi 1.9 listftp and fetchftp - pulls files again if nifi service is restarted otherwise no dups

HDF 3.4.1 Nifi 1.9 listftp and fetchftp - pulls files again if nifi service is restarted otherwise no dups

Contributor

I know we have this issue in nifi 1.5 thought fixed in 1.9 . not sure in which version this is fixed?

 

Any thoughts? instead of a workaround , interested to know if this is handled in any newer versions?

2 REPLIES 2

Re: HDF 3.4.1 Nifi 1.9 listftp and fetchftp - pulls files again if nifi service is restarted otherwise no dups

Expert Contributor

I believe this is fixed in 1.9 (and up)

 

Based on : http://mail-archives.apache.org/mod_mbox/nifi-issues/201811.mbox/%3CJIRA.13201249.1543470758000.5433...


If this answer helped, please mark it as 'solved' and/or if it is valuable for future readers please apply 'kudos'.
Highlighted

Re: HDF 3.4.1 Nifi 1.9 listftp and fetchftp - pulls files again if nifi service is restarted otherwise no dups

Master Guru

@DennisJaheruddi 

 

The bug jira https://issues.apache.org/jira/browse/NIFI-5849 addressed listSFTP and ListFTP losing state on a NiFi restart.  This bug fix is included in Apache NiFi 1.9.0+ releases.

Since HDF 3.4.1.1 is based off Apache NiFi 1.9, you will have this fix.

Another possible reason for what you are seeing may be the result of using the "Tracking Entities" listing strategy in the ListFTP processor.    This strategy requires that you have also configured a "Entity Tracking State Cache".  If you use the DistributedMapCache embedded in your NiFi and you have not configured a "Persistence Directory" for the DistributedMapCacheServer, then all cache entries will be lost on a NIFi restart.  This can lead to files being listed again on restart as the DistributedMapCacheServer in heap memory cache is populated all over again.  If you setup the "Persistence Directory", then previously cached entries will be loaded back in to heap memory on NiFi restart.

Hope this helps,

Matt

 

Don't have an account?
Coming from Hortonworks? Activate your account here