question Re: Get list of keys in NiFi DistributedMapCache in Support Questions

Get list of keys in NiFi DistributedMapCache

julie_nyikos — Wed, 15 Aug 2018 06:13:46 GMT

Here is my use case -- I would like to use the DistributedMapCache to cache about 30k records that I read from an http service. Once per day, I'd like to update the cache, which includes removing items which are no longer returned by the http service. I had wanted to implement this by writing a custom processor to retrieve the list of keys from the DistributedMapCache, and remove any item which did not get returned by the http service. However, when I looked at the interface for DistributedMapCacheClient, the method to return the set of keys (keySet) throws the UnsupportedOperationException. Does anyone have any suggestions on how else I can implement this? I'm a newbie with NiFi -- any suggestions you have would be greatly appreciated! Thanks!

Re: Get list of keys in NiFi DistributedMapCache

mburgess — Wed, 15 Aug 2018 06:37:51 GMT

The default implementation is to throw UnsupportedOperationException, but many/most/all of the subclasses override it. Check the reference impl, and since you are removing things, also check removeAndGet(), removeByPattern(), and removeByPatternAndGet() to see if they would help, they might save you the trouble of fetching all keys to remove some of them.

Re: Get list of keys in NiFi DistributedMapCache

julie_nyikos — Wed, 15 Aug 2018 06:52:11 GMT

Ah yes thank you! Indeed the reference impl does implement the keySet method. I found your dcache groovy script and attempted to modify it to call the keySet method to return the set of keys, but I don't know groovy and so far it's not working... any chance I can get you to add a "getkeys" option to your groovy script if it's not too difficult? Thank you so much for your help! 🙂

Re: Get list of keys in NiFi DistributedMapCache

mburgess — Thu, 16 Aug 2018 03:10:57 GMT

dcachegroovy.txt Here you go! The command is called "keys", prints each key on its own line.

Re: Get list of keys in NiFi DistributedMapCache

julie_nyikos — Thu, 16 Aug 2018 22:11:02 GMT

You are awesome! Thank you!!!

Re: Get list of keys in NiFi DistributedMapCache

Dave0x1 — Wed, 13 Mar 2024 11:06:42 GMT

Hi there, the script looks good and exactly what we are looking for. Just a quick question, when we try to implement it, we get the following error:

We are using the invokescriptprocessor and configure like this

What are we missing here 😉

Thank you.

Re: Get list of keys in NiFi DistributedMapCache

DianaTorres — Wed, 13 Mar 2024 12:23:01 GMT

@Dave0x1 As this is an older post, you would have a better chance of receiving a resolution by starting a new thread. This will also be an opportunity to provide details specific to your environment that could aid others in assisting you with a more accurate answer to your question. You can link this thread as a reference in your new post. Thanks.

Re: Get list of keys in NiFi DistributedMapCache

Dave0x1 — Wed, 13 Mar 2024 12:57:31 GMT

Hi Diana, good idea. I'll create a new thread. Thank you.