Process Group level counter

[#1]

Hi,

I'm looking for a way to implement a process group level counter that I can read and update either with a processor, including ExecuteScript / ExecuteGroovyScript.

I've looked at and tried:

1. Counters: I can create one, set the value and increment it, but not read the value within a processor, that I've found as of yet.

2. Parameter Context parameters: I can read the value, but not increment or reset it. It seems to be static once it's defined.

Is there something I've missed with the above, or does anyone have another suggestion please?

Thank you!

[#2 · Accepted solution]

@Emery 

You are correct that there is only an option for adjusting a counter either up or down.  There is no process session for retrieving the value from a counter.

https://github.com/apache/nifi/blob/master/nifi-api/src/main/java/org/apache/nifi/processor/ProcessS...

You could make a NiFi rest-api call to get the complete list of counters.  You would need to then filter out the specific counter(s) you want to get the value you are looking for from the json response.

Please help our community grow and thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped.

Thank you,
Matt

[#3]

@Emery 

1. Counters: NiFi counters only exist in NiFi heap, so values are lost if NiFi is restarted. 
2. Parameter Contexts are static defined values intended use across many components thus along a single place to update when needed.

I don't know what your use case is for these counters, but perhaps you could use the following processors to maintain a count and be able to fetch the value when needed in your dataflows:

• PutDistributedMapCache 1.28.0
• FetchDistributedMapCache 1.28.0

So you can Fetch current value from cache, update it, and put replace value in cache. 

Please help our community grow and thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped.

Thank you,
Matt

[#4]

Hi,

Thanks for your reply.

It's ok for me if the Counter values are lost when NiFi is restarted. I can deal with that. Can I read their values in ExecuteScript / ExecuteGroovyScript?

I used this to increment it, but I wasn't able to read the value itself.

session.adjustCounter("my_counter", 1, true)

I'll look into Distributed Map Cache. I'm unfamiliar with it. Thank you!

[#5 · Accepted solution]

@Emery 

You are correct that there is only an option for adjusting a counter either up or down.  There is no process session for retrieving the value from a counter.

https://github.com/apache/nifi/blob/master/nifi-api/src/main/java/org/apache/nifi/processor/ProcessS...

You could make a NiFi rest-api call to get the complete list of counters.  You would need to then filter out the specific counter(s) you want to get the value you are looking for from the json response.

Please help our community grow and thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped.

Thank you,
Matt

[#6]

I've read up a bit onDistributedMapCacheServer and setting up and accessing ports, and its purpose really feels larger than what I'm looking for.  😞 

I'd just like the ability to maintain a simple counter within a process group that I can read, increment and reset, best within a groovy script.

Parameters and Counters, or something else if it exists, would totally suit the purpose. Is there a way I can read counter values in ExecuteScript / ExecuteGroovyScript that I'm missing?

Thanks!

[#7]

I read about that possibility. Also with that, making a rest api call for a simple "global variable" within a process group or across a single process flow uses more processing resources and time than I would think is optimal, but I may be wrong.

[#8]

@Emery 

No argument with you there.  A rest-api call fetches a json of all counters which you then need to parse to extract the specific counter value you are looking for.  

The NiFi counters where never intended to be used for anything more then simple counts during flow development testing.

Matt

[#9]

That's unfortunate. All I require is a simple counter. 😞

[#10 · Accepted solution]

With the help of the information @MattWho provided in

https://community.cloudera.com/t5/Support-Questions/Using-MapCacheServer-and-MapCacheClientService-i...

I've been able to implement a counter as I required. Not with 

PutDistributedMapCache / FetchDistributedMapCache, but more flexibly with ExecuteScript.

Thanks for your help. I really appreciate it.