@Johny Travolta I don't understand how not having a shared token necessarily means you do not have a cluster. You will get better traction form the community if you move this to a new question. I am not a NiFi developer myself, so I could not comment on the complexity of implementing a shared LDAP token response across all nodes in a NiFi cluster. But I am sure if you open a new question around this topic, you will get a response from someone who can answer for you. Thanks, Matt ... View more

@Thierry Vernhet With number 3, I am assuming that every file has a unique filename from which to determine if the same filename has ever been listed more then once. If that is not the case, then you would need to use detectDuplicate after fetching the actual data (less desirable since you will have wasted the resources to potential fetch the same files twice before deleting the duplicate. Let assume every file has a unique filename. If so the detect duplicate flow would look like this: with the DetectDuplicate configured as follows: You will also need to add two controller services to your NiFi: - DistributedMapCacheServer - DistributedMapCacheClientService The value associated to the "filename" attribute on the FlowFile is checked against entries in the DistributedMapCacheServer. If filename does not exist, it is added. If it exists already then FlowFile is routed to duplicate relationship. In scenario 2 where filenames may be reused we need to detect if the content after fetch is a duplicate or not. IN this case the flow may look like this: After fetching the content of a FlowFile, the "HashContent" processor is used to create a hash of the content and write it to a FlowFile attribute (default is hash.value). The detectDuplicate processor then configured to look for FlowFile with the same hash.value to determine if they are duplicates. FlowFiles where the content hash already exist in the distributedMapCacheServer, those FlowFile are routed to duplicate where you can delete them if you like. If you found this answer addressed your original question, please mark it as accepted by clicking under the answer. Thanks, Matt ... View more

@Anoop Shet Sorry for the late response, but i don't get pinged unless you tag me in your response. The ListSFTP processor retains state on Files that have been listed. My guess here is the state is preventing these new filter form returning anything. Try clearing the state and see if it then lists the files based on your new filter or add a new ListSFTP processor using that different file filter. You can right click on the processor and select "View state". In the state UI for this processor you will see a link to "Clear state". If you found my answer addressed your original question, please mark as accepted to close out thsi thread. Thanks, Matt ... View more

@Mahmoud Shash HDF 2.1.3 is a bad release. You are running into the exact Controller Service UI bug that resulted in HDF 2.1.3 being pulled and replaced with HDF 2.1.4. You can upgrade you HDF 2.1.3 to HDF 2.1.4 to fix this issue. Then you will be able to enable, disable, configure, and delete the HiveConnectionPool Controller service. Matt ... View more

@Narasimma varman Sorry for late response, but I don't get pinged unless you add a comment to my response or tag me in your new answer. The dynamic properties expect the the "value" to be a valid NiFI expression language (EL) statement. Otherwise it is treated as a literal value. So i expect what you are seeing is that exact string passed in the nested header or some kind of session rollback, etc... Also not sure how you are pulling data using a "POST" method? Shouldn't you be using "GET"? Thanks, Matt ... View more

@forest lin NiFi at is core has no issues working with very large files. Often times, when you run into OOM it is because of what you are trying to do with those very large files after they are in NiFi. In the majority of the cases OOM can be avoided via dataflow design and tweaks to the heap size allocated to the NiFi JVM. The content of a FlowFile does not live in heap memory space, but the FlowFile attributes do (*** except when swapped out to disk in large queues). So avoid extracting large amounts of the content into FlowFile attributes, avoid trying to split very large files in to large numbers of small FlowFiles using a single processor, avoid trying to merge a very large number of FlowFiles in to a single FlowFile, etc... You can still do these types of things but may need to do it in two stages rather then one. For example Splitting large files by every 5000 lines first and then split 5000 line FlowFiles by every line (Huge difference in heap usage). If you found this answer addressed your question, please mark it as accepted to close out this thread. Thanks, Matt ... View more