Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

is it possible to get details from web service in every 5min but not duplicate in nifi / and what are the configuration of DetectDuplicate processor

avatar
Contributor

@stevenmatison is it possible to apply filter condition on invoke HTTP processor that give all the alarm from web app in every 5 min but will not fetch the duplicate alarm that already fetched before ? or else is it possible when the alarm will come in to my web app then workflow in NIFI will trigger and gives recent alarm which came just in 10 min ...i have tried with DetectDuplicate processor ...but not getting any result ..i think i made mistake in configuration in detecduplicate processor.. please direct me ....and if you have any other route for this please provide me the whole way(configuration an all) ..i'm new in nifi ..don't know much more about that 

 

B81vu.png

detectduplicate processor4o59W.png

when i clicked in arrow in side of RedisDistributedMapCacheClintService ...this below config has open(RedisConnectionPoolService)..and in connection string option i have put web address with port number is it wrong ..or else we have to put nifi localhost address with port no.??

 

L2Dv9.png

 

1 ACCEPTED SOLUTION

avatar
Super Guru

@renuu  Try the DistributedMapCacheClient Controller Service,  not Redis (unless you have a redis system you can use).  If you use the other one you can set it to localhost and work within NiFi.   After creating the  DistributedMapCacheClient be sure to click + and add DistributedMapCacheServer to run on the same port.    Here are some screen shots of my sample:

 

DetectDuplicate:

Screen Shot 2020-05-28 at 9.14.58 AM.png

 

DistributedMapCacheClient:

Screen Shot 2020-05-28 at 9.15.42 AM.png

 

DistributedMapCacheServer:

Screen Shot 2020-05-28 at 4.05.34 PM.png

 

 

 

 


Additionally,  when working a demo flow,  be sure to always route the other relationships during testing (detectDuplicate).  You do this so when its running you can see the duplicate or non-duplicate outputs.   If you run the flow, and it doesn't work as expected you will see where it went, then can adjust.    For example, below is how I use an output port to hold all the routes for InvokeHttp.  Later when I know it works, I will auto terminate what I don't need, and route what I do need.

 

Screen Shot 2020-05-28 at 8.42.10 AM.png

 

Last but not least, spend some time in the community doing research to learn from other examples for your Use Case by searching the community:

 

https://community.cloudera.com/t5/forums/searchpage/tab/message?advanced=false&allow_punctuation=fal...

 

 

If this answer resolves your issue or allows you to move forward, please choose to ACCEPT this solution and close this topic. If you have further dialogue on this topic please comment here or feel free to private message me. If you have new questions related to your Use Case please create separate topic and feel free to tag me in your post.  

 

Thanks,


Steven @ DFHZ

View solution in original post

2 REPLIES 2

avatar
Super Guru

@renuu  Try the DistributedMapCacheClient Controller Service,  not Redis (unless you have a redis system you can use).  If you use the other one you can set it to localhost and work within NiFi.   After creating the  DistributedMapCacheClient be sure to click + and add DistributedMapCacheServer to run on the same port.    Here are some screen shots of my sample:

 

DetectDuplicate:

Screen Shot 2020-05-28 at 9.14.58 AM.png

 

DistributedMapCacheClient:

Screen Shot 2020-05-28 at 9.15.42 AM.png

 

DistributedMapCacheServer:

Screen Shot 2020-05-28 at 4.05.34 PM.png

 

 

 

 


Additionally,  when working a demo flow,  be sure to always route the other relationships during testing (detectDuplicate).  You do this so when its running you can see the duplicate or non-duplicate outputs.   If you run the flow, and it doesn't work as expected you will see where it went, then can adjust.    For example, below is how I use an output port to hold all the routes for InvokeHttp.  Later when I know it works, I will auto terminate what I don't need, and route what I do need.

 

Screen Shot 2020-05-28 at 8.42.10 AM.png

 

Last but not least, spend some time in the community doing research to learn from other examples for your Use Case by searching the community:

 

https://community.cloudera.com/t5/forums/searchpage/tab/message?advanced=false&allow_punctuation=fal...

 

 

If this answer resolves your issue or allows you to move forward, please choose to ACCEPT this solution and close this topic. If you have further dialogue on this topic please comment here or feel free to private message me. If you have new questions related to your Use Case please create separate topic and feel free to tag me in your post.  

 

Thanks,


Steven @ DFHZ

avatar
Contributor

renuu_0-1590675600228.png

 

@stevenmatison  thank you for your reply ...now i got this error ...i have gone through your reply