Get list of keys in NiFi DistributedMapCache

[#1]

Here is my use case -- I would like to use the DistributedMapCache to cache about 30k records that I read from an http service. Once per day, I'd like to update the cache, which includes removing items which are no longer returned by the http service. I had wanted to implement this by writing a custom processor to retrieve the list of keys from the DistributedMapCache, and remove any item which did not get returned by the http service. However, when I looked at the interface for DistributedMapCacheClient, the method to return the set of keys (keySet) throws the UnsupportedOperationException. Does anyone have any suggestions on how else I can implement this? I'm a newbie with NiFi -- any suggestions you have would be greatly appreciated! Thanks!

[#2 · Accepted solution]

The default implementation is to throw UnsupportedOperationException, but many/most/all of the subclasses override it. Check the reference impl, and since you are removing things, also check removeAndGet(), removeByPattern(), and removeByPatternAndGet() to see if they would help, they might save you the trouble of fetching all keys to remove some of them.

[#3]

Ah yes thank you! Indeed the reference impl does implement the keySet method. I found your dcache groovy script and attempted to modify it to call the keySet method to return the set of keys, but I don't know groovy and so far it's not working... any chance I can get you to add a "getkeys" option to your groovy script if it's not too difficult? Thank you so much for your help! 🙂

[#4]

dcachegroovy.txt Here you go! The command is called "keys", prints each key on its own line.

[#5]

You are awesome! Thank you!!!

[#6]

Hi there, the script looks good and exactly what we are looking for. Just a quick question, when we try to implement it, we get the following error:

We are using the invokescriptprocessor and configure like this

What are we missing here 😉 

Thank you.

[#7]

@Dave0x1 As this is an older post, you would have a better chance of receiving a resolution by starting a new thread. This will also be an opportunity to provide details specific to your environment that could aid others in assisting you with a more accurate answer to your question. You can link this thread as a reference in your new post. Thanks.

[#8]

Hi Diana, good idea. I'll create a new thread. Thank you.