Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

is it possible to get details from web service in every 5min but not duplicate in nifi / and what are the configuration of DetectDuplicate processor

Solved Go to solution
Highlighted

is it possible to get details from web service in every 5min but not duplicate in nifi / and what are the configuration of DetectDuplicate processor

Explorer

@stevenmatison is it possible to apply filter condition on invoke HTTP processor that give all the alarm from web app in every 5 min but will not fetch the duplicate alarm that already fetched before ? or else is it possible when the alarm will come in to my web app then workflow in NIFI will trigger and gives recent alarm which came just in 10 min ...i have tried with DetectDuplicate processor ...but not getting any result ..i think i made mistake in configuration in detecduplicate processor.. please direct me ....and if you have any other route for this please provide me the whole way(configuration an all) ..i'm new in nifi ..don't know much more about that 

 

B81vu.png

detectduplicate processor4o59W.png

when i clicked in arrow in side of RedisDistributedMapCacheClintService ...this below config has open(RedisConnectionPoolService)..and in connection string option i have put web address with port number is it wrong ..or else we have to put nifi localhost address with port no.??

 

L2Dv9.png

 

1 ACCEPTED SOLUTION

Accepted Solutions
Highlighted

Re: is it possible to get details from web service in every 5min but not duplicate in nifi / and what are the configuration of DetectDuplicate processor

Master Collaborator

@renuu  Try the DistributedMapCacheClient Controller Service,  not Redis (unless you have a redis system you can use).  If you use the other one you can set it to localhost and work within NiFi.   After creating the  DistributedMapCacheClient be sure to click + and add DistributedMapCacheServer to run on the same port.    Here are some screen shots of my sample:

 

DetectDuplicate:

Screen Shot 2020-05-28 at 9.14.58 AM.png

 

DistributedMapCacheClient:

Screen Shot 2020-05-28 at 9.15.42 AM.png

 

DistributedMapCacheServer:

Screen Shot 2020-05-28 at 4.05.34 PM.png

 

 

 

 


Additionally,  when working a demo flow,  be sure to always route the other relationships during testing (detectDuplicate).  You do this so when its running you can see the duplicate or non-duplicate outputs.   If you run the flow, and it doesn't work as expected you will see where it went, then can adjust.    For example, below is how I use an output port to hold all the routes for InvokeHttp.  Later when I know it works, I will auto terminate what I don't need, and route what I do need.

 

Screen Shot 2020-05-28 at 8.42.10 AM.png

 

Last but not least, spend some time in the community doing research to learn from other examples for your Use Case by searching the community:

 

https://community.cloudera.com/t5/forums/searchpage/tab/message?advanced=false&allow_punctuation=fal...

 

 

If this answer resolves your issue or allows you to move forward, please choose to ACCEPT this solution and close this topic. If you have further dialogue on this topic please comment here or feel free to private message me. If you have new questions related to your Use Case please create separate topic and feel free to tag me in your post.  

 

Thanks,


Steven @ DFHZ

 


 


If this answer resolves your issue or allows you to move forward, please choose to ACCEPT this solution and close this topic. If you have further dialogue on this topic please comment here or feel free to private message me. If you have new questions related to your Use Case please create separate topic and feel free to tag me in your post.  


 


Thanks,



Steven

View solution in original post

2 REPLIES 2
Highlighted

Re: is it possible to get details from web service in every 5min but not duplicate in nifi / and what are the configuration of DetectDuplicate processor

Master Collaborator

@renuu  Try the DistributedMapCacheClient Controller Service,  not Redis (unless you have a redis system you can use).  If you use the other one you can set it to localhost and work within NiFi.   After creating the  DistributedMapCacheClient be sure to click + and add DistributedMapCacheServer to run on the same port.    Here are some screen shots of my sample:

 

DetectDuplicate:

Screen Shot 2020-05-28 at 9.14.58 AM.png

 

DistributedMapCacheClient:

Screen Shot 2020-05-28 at 9.15.42 AM.png

 

DistributedMapCacheServer:

Screen Shot 2020-05-28 at 4.05.34 PM.png

 

 

 

 


Additionally,  when working a demo flow,  be sure to always route the other relationships during testing (detectDuplicate).  You do this so when its running you can see the duplicate or non-duplicate outputs.   If you run the flow, and it doesn't work as expected you will see where it went, then can adjust.    For example, below is how I use an output port to hold all the routes for InvokeHttp.  Later when I know it works, I will auto terminate what I don't need, and route what I do need.

 

Screen Shot 2020-05-28 at 8.42.10 AM.png

 

Last but not least, spend some time in the community doing research to learn from other examples for your Use Case by searching the community:

 

https://community.cloudera.com/t5/forums/searchpage/tab/message?advanced=false&allow_punctuation=fal...

 

 

If this answer resolves your issue or allows you to move forward, please choose to ACCEPT this solution and close this topic. If you have further dialogue on this topic please comment here or feel free to private message me. If you have new questions related to your Use Case please create separate topic and feel free to tag me in your post.  

 

Thanks,


Steven @ DFHZ

 


 


If this answer resolves your issue or allows you to move forward, please choose to ACCEPT this solution and close this topic. If you have further dialogue on this topic please comment here or feel free to private message me. If you have new questions related to your Use Case please create separate topic and feel free to tag me in your post.  


 


Thanks,



Steven

View solution in original post

Re: is it possible to get details from web service in every 5min but not duplicate in nifi / and what are the configuration of DetectDuplicate processor

Explorer

renuu_0-1590675600228.png

 

@stevenmatison  thank you for your reply ...now i got this error ...i have gone through your reply  

Don't have an account?
Coming from Hortonworks? Activate your account here