@jirungaray  Welcome to the community. The DetectDuplicate processor does not store anything in NiFi state providers (local state directory or cluster state in zookeeper).  The DetectDuplicate processor utilizes a DistributedMapCache Service to store cached items.  Depending on the cache service used, those cache service may offer retention configurations for number of cache entries and cache entry persistence. Any NiFi component that retains state will indicate such in its documentation under the "State Management" section. The "Age Off Duration" configuration will age off cache entries that may still exist when that duration is reached, but it can not control the number of cache entries the end service will retain.  So the cache service may still be evicting cache entries prior to that configured Age of Duration is reached. Since you mention that your Cache Entries are not being preserved on NiFi restart, I assume you have configured your DetectDuplicate to use the DistributedMapCacheClientService.    The DistributedMapCacheClientService is dependent on the existence of a running DistributedMapCacheServer.  This DistributedMapCacheServer does in fact hold cache entries within NiFi's Heap memory and unless you have configured a "Persistence Directory", will lose all cache entries on NiFi service stop.  The DistributedMapCacheServer also has configuration thresholds for the max number of cache entries it will hod before evicting cache entires based on the eviction strategy configured.  This configuration established an upper boundary.  Keep in mind the higher the Max Cache Entries setting, the more NiFi heap memory is used which could lead to NiFi experiencing OutOfMemory (OOM) exceptions.  Since it sounds like you want to retain a very large amount of cached entries, I'd recommend against using the NiFi internal DistributedMapCacheClientService considering the high heap memory usage it would require and the high likelihood that will impact your NiFi's stability and performance.   NOTE: The DistributedMapCacheClientService and DistributedMapCacheServer do NOT offer any form of High Availability. The DistributeMapCacheClientService can only be configured with a single server hostname. While the DistributedMapCacheServer when started does create a running Cache server on all hosts within the NiFi cluster, the cached entries are not shared or replicated across all of them. ONLY the cache server hostname configured in the DistributedMapCacheClientService is used. For HA, you should be using a more robust external to NiFi cache service like Redis.     Please help our community grow and thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt ... View more

@dsender  Apache NiFi is a data agnostic service.  It can move any data format through a dataflow because the content is treated as just bytes  inside a FlowFile.  The only time the content needs to be read is if there is need to manipulate it, extract from it, etc.   Then you would need to use a processor that understand the data format.  While it does not appear that Cloudera Flow Management offers any SAS specific processor components.  So some custom processor would need to be developed or perhaps you can use one of the available scripting processors?  You would still need to write a custom script to ingest and/or process the SAS files.  So this starts with the question of how would you pull these SAS files from command line outside of using NiFi?  Then figure out how to turn that success into a custom script or processor that does the same thing. You could also reach out to your Cloudera Account owner and discuss possible professional service offering that maybe able to help you here with your custom needs. Please help our community grow and thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt ... View more

@fs_nifioneks  Welcome to the community. The logic behind why the encrypt-config tools were removed in Apache NiFi 2.0 is well explained in the jira NIFI-13414 you mentioned.  I am sure that the Apache community will eventually implement other more robust options for password security. That being said, Cloudera's Cloudera Flow Management 4.x product line is  based off Apache NiFi 2.0, but will still include the encrypt-config utility in its code base to persist the existing password encryption option until more robust options replace it.  In addition, Cloudera Flow Management also keeps many (not all) Apache NiFi components processors and controller services deprecated in the Apache NiFi 2.0 releases, as well as, includes additional components only available through Cloudera only for even more dataflow design capability and connection options.  Cloudera Flow management 4.0 is available to our licensed users as a technical preview, but a full production ready release will be coming in the near future.  You can view the Cloudera Flow Management documentation here for the Tech Preview release: https://docs.cloudera.com/cfm/4.0.0/index.html These Tech preview docs do not provide a component list, but the production ready release docs will. Please help our community grow and thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt ... View more

@David07  Welcome to the community. NiFi-Registry would allow you to version control NiFi Process Group dataflows.  You can then connect multiple NiFi instances/cluster to the same NiFi Registry which would allow those other NiFi's access to these version controlled flow definitions (if authorized correctly).    From within NiFi, you can also download process groups as flow definition json files.  You can use these to create offline catalog of these flow definitions for ease of reuse in other NiFi instances/clusters.  You can easily import a flow definitions to the canvas of a NiFi instance.  For more details here, read the "Building a Dataflow" section of the NiFi User Guide.  Importing and downloading flow definitions is covered in the "Process Group" section. Tip: Building your dataflows using parameter contexts for properties that may have unique values per environments (URL, passwords, usernames, etc.) makes the process of sharing or moving flow definitions between NiFi deployments.  Each environment may have different values assigned to the parameter contexts referenced in NiFi processors. Cloudera offers a unique option for rapid multiple NiFi instance deployments through Cloudera Edge Flow Manager.   This option requires a license with Cloudera to download this product.  This tool allows you to construct a dataflows just how you would in NiFi and then deploy that dataflow to one or more agents (MiNiFi instances).  This provides you with a central management location for managing multiple unique dataflow deployments to unique agents. Please help our community grow and trhive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt ... View more