Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

DetectDuplicate is not working

Solved Go to solution
Highlighted

DetectDuplicate is not working

Explorer

help.PNG

hello everyone ....please help me out ...i m working on one system that system received alarm from client ...we generally received 10000 alarm ......we want to configure our web with nifi for some other update in alarm ...for this we have used invoked http processor that get all the alarm from web app and then we used split json processor that split one alarm in one flowfile then we used evaluate jsonpath that extract id attribute which will be use in detectduplicate processor ....actually the invoked http processor give repeated alarm but we want only one copy of one alarm ...every 10 min the http processor get alarm from web app but those alarm should not be duplicate ....is it possible to do this .....if you have any other option for this work please suggest me.

and in detectduplicate ----

CacheEntryIndentifier - ${id}

age of duration - 15 min 

distributed cache service - DistributedMapCacheClientService 

and one more thing i have used  DistributedMapCacheClientService --

server hostname - localhost 

server port - 4557

thanks in advance 

 

2 ACCEPTED SOLUTIONS

Accepted Solutions
Highlighted

Re: DetectDuplicate is not working

@renuu The solution you are looking for is to add the DistributedMapCacheServer from the controller services tab.  Set this to the same port as the DisitributedMapCachClient.   Enable. Test flow again.

 

 

If this answer resolves your issue or allows you to move forward, please choose to ACCEPT this solution and close this topic. If you have further dialogue on this topic please comment here or feel free to private message me. If you have new questions related to your Use Case please create separate topic and feel free to tag me in your post.  

 

Thanks,


Steven @ DFHZ

 


 


If this answer resolves your issue or allows you to move forward, please choose to ACCEPT this solution and close this topic. If you have further dialogue on this topic please comment here or feel free to private message me. If you have new questions related to your Use Case please create separate topic and feel free to tag me in your post.  


 


Thanks,



Steven

View solution in original post

Highlighted

Re: DetectDuplicate is not working

After you posted this new issue, I created a sample like yours, and I had the same error you have above.  Then I added the DistributedMapCacheServer, enabled it, and re-tested my flow.

 

I created a template for DetectDuplicate which you can find here:

 

https://github.com/steven-dfheinz/NiFi-Templates/blob/master/DetectDuplicate_DistributedMapCache_Dem...

 

If you install the template, you still have to + ADD the Map Cache Server in the Template's Process Group Controller Services.  The NiFI Template system does not include the server (known bug), it only includes the client.

 

 


 


If this answer resolves your issue or allows you to move forward, please choose to ACCEPT this solution and close this topic. If you have further dialogue on this topic please comment here or feel free to private message me. If you have new questions related to your Use Case please create separate topic and feel free to tag me in your post.  


 


Thanks,



Steven

View solution in original post

3 REPLIES 3
Highlighted

Re: DetectDuplicate is not working

@renuu The solution you are looking for is to add the DistributedMapCacheServer from the controller services tab.  Set this to the same port as the DisitributedMapCachClient.   Enable. Test flow again.

 

 

If this answer resolves your issue or allows you to move forward, please choose to ACCEPT this solution and close this topic. If you have further dialogue on this topic please comment here or feel free to private message me. If you have new questions related to your Use Case please create separate topic and feel free to tag me in your post.  

 

Thanks,


Steven @ DFHZ

 


 


If this answer resolves your issue or allows you to move forward, please choose to ACCEPT this solution and close this topic. If you have further dialogue on this topic please comment here or feel free to private message me. If you have new questions related to your Use Case please create separate topic and feel free to tag me in your post.  


 


Thanks,



Steven

View solution in original post

Re: DetectDuplicate is not working

Explorer

It is already added ....but not work 

Highlighted

Re: DetectDuplicate is not working

After you posted this new issue, I created a sample like yours, and I had the same error you have above.  Then I added the DistributedMapCacheServer, enabled it, and re-tested my flow.

 

I created a template for DetectDuplicate which you can find here:

 

https://github.com/steven-dfheinz/NiFi-Templates/blob/master/DetectDuplicate_DistributedMapCache_Dem...

 

If you install the template, you still have to + ADD the Map Cache Server in the Template's Process Group Controller Services.  The NiFI Template system does not include the server (known bug), it only includes the client.

 

 


 


If this answer resolves your issue or allows you to move forward, please choose to ACCEPT this solution and close this topic. If you have further dialogue on this topic please comment here or feel free to private message me. If you have new questions related to your Use Case please create separate topic and feel free to tag me in your post.  


 


Thanks,



Steven

View solution in original post

Don't have an account?
Coming from Hortonworks? Activate your account here