Hi all, I am having problems with DetectDuplicate. It is not working as expected or I am not knowing how to configure it - Am I missing something? Imagine this simple Json list: [ { "ID": 101 }, { "ID": 102 }, { "ID": 103 }, { "ID": 104 }, { "ID": 105 }, { "ID": 106 }, { "ID": 107 }, { "ID": 108 }, { "ID": 109 }, { "ID": 110 } ] Looking at the above Json list we expect that every item will be a non duplicate item in Distributed Map Cache Server. But it is not what is happening. Here is the Detect Duplicate Propertie configuration: When I start the process flow look what happens: Only the first ID is detected as a non duplicate as you can see in the LogAttribute - Non Duplicate data provenance: What Am I doing wrong? Am I missing setting some configuration? Here is the template: detect-duplicate.xml Any help will be much appreciated! Thank you in advance. ... View more

Hi all! NiFi newbie attacks again! Today I have a question about using DistributedMapCacheServer. We have the following scenario: - We will have a lot of incoming data but we don't want to process two (or more) identifiers at the same time.If a second object with the same identifier come from data flow (FlowFile), we will have to discard it until the other identifier runs through the whole process. Then we find the DetectDuplicate Processor and it's working perfect (as you can see in the image below and the template is here -> detect-duplicate-v1.xml). But the problem is that after the whole process execute, we will have to free the identifier from DistributedMapCacheServer. We know that the DetectDuplicate Processor has the propertie to clear the cache (Age Off Duration), but it uses time instead of an event to clear the cache, then that propertie doesn't suit for our use case. Now we are trying to finding a way to Remove the identifier from DistributedMapCacheServer like the flow below. I searched in NiFi docs and internet and I don't found any processor to remove a cached identifier. The only thing that I find was @Matt Burgess article https://community.hortonworks.com/articles/71837/working-with-a-nifi-distributedmapcache.html but it uses a script and I don't know how to use it. I am missing something? Any help will be much appreciate! ... View more

Hi all, First of all I am a NiFi newbie! 🙂 We are STUCK. Our initial desire is to get a attribute from a flowfile and asks the SQLServer if this attribute exists. If the answer is YES, then the flowfile continue his way - but we didn't find anything like this. Then we had an idea: Query the database and put the Attributes in the SELECT. But, again, we didn't find a processor that create a SQL Query dynamically. The flowfile attribute looks like this: [ { "ID" : 14733307, "DeviceID" : "40d71f9776cbdca1a39b6f8c5493219f16ce5e8ac77a537f3936d5b81c367320", "LogDate" : "2017-07-03 12:27:26.327", "Latitude" : -29.98709106445312, "Longitude" : -51.1904296875 } ] And we are trying to get this attributes and create the query like the one below: SELECT '${ID}' as ID ,'${DeviceID}' as DeviceID ,'${LogDate}' as LogDate ,'${Latitude}' as Latitude ,'${Longitude}' as Longitude ,'OK' as OK FROM [Device] WHERE DeviceID = '${DeviceID}' Is there any Processor that do this? Or we will have to create one? 😄 Cause the examples that we find searching in internet only demonstrate how to replace text and using a lot of processors. Like the two below: https://stackoverflow.com/questions/41057483/how-to-pass-values-dynamically-in-apache-nifi-from-executesql-to-selecthiveql https://community.hortonworks.com/questions/70562/executesql-dynamic-query.html This is working: What we want: Any help will be appreciated! 😄 ... View more