Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Reset of Target Signal Count in Wait Processor

Solved Go to solution
Highlighted

Reset of Target Signal Count in Wait Processor

New Contributor

I have a flow in which I wait for 5 files to come out from each of the respective executeProcessors, after which I start another processor. I achieved this using Wait and Notify processors by target signal count given as 5. I stop and restart the flow, it works as expected. Now, my question is when and how would this counter be reset? If the executeProcessors are scheduled to run for every 30 min, do signals from notify keep increasing the counter. If so, how does my wait processor ever match the signal count of 5 that I gave and proceed to next processor every time.

Thanks,

John

1 ACCEPTED SOLUTION

Accepted Solutions

Re: Reset of Target Signal Count in Wait Processor

@John Carter

Assuming you have the flow designed correctly.

You don't have to reset the counter. That is why the Wait/Notify use the DistributedMapCacheClientService. It will keep track of the signal count.

Would you be willing to provide a snapshot of your flow? Also, a snapshot of how the DistributedMapCacheServer controller service is configured.

11 REPLIES 11

Re: Reset of Target Signal Count in Wait Processor

@John Carter

What does the configuration of the Wait processor look like?

Re: Reset of Target Signal Count in Wait Processor

New Contributor

Hi Wyner,

wait-processor.jpeg

You can see that am waiting for 10 signals here.

Re: Reset of Target Signal Count in Wait Processor

You'd probably want to have a signal identifier using a unique flow file attribute (such as uuid). If you could provide more details around what you want to achieve, that'd be helpful.

Re: Reset of Target Signal Count in Wait Processor

New Contributor

I am using file name as my signal identifier. Filenames before notify and wait are all renamed to a specific value and the count of these signals is what I am checking.

Re: Reset of Target Signal Count in Wait Processor

@John Carter

Assuming you have the flow designed correctly.

You don't have to reset the counter. That is why the Wait/Notify use the DistributedMapCacheClientService. It will keep track of the signal count.

Would you be willing to provide a snapshot of your flow? Also, a snapshot of how the DistributedMapCacheServer controller service is configured.

Re: Reset of Target Signal Count in Wait Processor

@John Carter

Are you not seeing the behavior that you expect? You don't need to stop and start the flow, that has no effect on the count of the flow files the Wait processor is waiting for.

Would you be willing to provide a snapshot of your flow?

Re: Reset of Target Signal Count in Wait Processor

New Contributor

Thanks for your reply, Wyner. Here is the screen shot of the flow that I simulated from the actual ones that we have. Also, the configurations of wait and notify that I am using are shown here:

1. Generate flow file generates a flow file every 5 seconds

2. Flow files are renamed to 'release_signal' (as this is what I am using as release signal identifier in wait and notify processors)

3. When there are 5 such signals, I want wait processor to push all the 5 flow files to the downstream success relation. Schedule on wait processor is 10 sec

Given the above scenario, I expect the success relation from wait processor should get flow files in the steps 5.. (like 5,10,15 ..) But I do not see something like that. You may see it in the screen shot too. There are 13 files in total that are pushed to success relation from wait. What I observed is, sometimes 5 flow files are pushed at once. But sometimes even one flow file is pushed to success from wait which I don't understand why that happens.

Since I could not get this working as expected, I switched to using "Merge Content" and it works for the use case that I have at hand.

Flow:

39554-pic1.jpeg

Update Attribute:

39555-update-attr.jpeg

Notify:

39556-notify.jpeg

Wait:

39557-wait.jpeg

Re: Reset of Target Signal Count in Wait Processor

@John Carter

I am using a similar setup for my testing and it works just like you think it should.

For your test, make sure and clear any residual file counts, by stopping and starting the DistributedMapCacheServer and the DistributedMapCacheClientService controller services. Then restart the test.

39558-screen-shot-2017-09-26-at-30519-pm.png

UpdateAttribute

39559-screen-shot-2017-09-26-at-30727-pm.png

Notify

39560-screen-shot-2017-09-26-at-30905-pm.png

Wait

39561-screen-shot-2017-09-26-at-30939-pm.png

Required controller services

39562-screen-shot-2017-09-26-at-31032-pm.png

Re: Reset of Target Signal Count in Wait Processor

@John Carter

Did you get chance to look at my flow? Also, will you post a snapshot of your DistributedMapCacheClientService configuration? Your flow should work like you expect.