Member since
10-23-2024
16
Posts
5
Kudos Received
1
Solution
My Accepted Solutions
Title | Views | Posted |
---|---|---|
2080 | 03-05-2025 05:29 AM |
03-05-2025
05:29 AM
With the help of the information @MattWho provided in https://community.cloudera.com/t5/Support-Questions/Using-MapCacheServer-and-MapCacheClientService-in-NiFi/td-p/403402 I've been able to implement a counter as I required. Not with PutDistributedMapCache / FetchDistributedMapCache, but more flexibly with ExecuteScript. Thanks for your help. I really appreciate it.
... View more
03-04-2025
06:38 AM
@MattWho Hi Matt, I made changes in my test based on the description you provided and I've been able to get the Groovy code working in ExecuteScript also. Thanks very much for your help! Back to the counter now. 🙂
... View more
03-03-2025
11:51 AM
Ah! Thank you for the explanation! I was always clicking on the three stacked dots and never saw the option to pick my existing instance! Umm, any thoughts about what I can do to access the map cache via ExecuteScript as described in https://community.cloudera.com/t5/Community-Articles/ExecuteScript-Cookbook-part-3/ta-p/249148 ? 🙂
... View more
03-02-2025
01:13 PM
Hi, I'm still trying to get this to work: https://community.cloudera.com/t5/Support-Questions/Process-Group-level-counter/td-p/402648 I have versions NiFi 2.0.0 and 2.2.0 at my disposal, and have tried to get simple examples working that I can then implement in my flow. I'm having difficulties as all the examples I find are with older versions of NiFi. First, Distributed Server and Client Services have been renamed: DistributedMapCacheServer has been renamed to MapCacheServer DistributedSetCacheServer has been renamed to SetCacheServer DistributedMapCacheClientService has been renamed to MapCacheClientService DistributedSetCacheClientService has been renamed to SetCacheClientService I've mostly gotten PutDistributedMapCache and FetchDistributedMapCache working as per the older example here: https://community.cloudera.com/t5/Support-Questions/how-to-put-data-in-PutDistributedMapCache/td-p/171803 and https://www.youtube.com/watch?v=wjis28NrF90 In the Controller Settings under Management Controller Services, I create a MapCacheServer and a MapCacheClientService set to use "localhost" as the Server Hostname. One thing I've noticed is that it's not possible to select the MapCacheClientService I created. In PutDistributedMapCache and FetchDistributedMapCache I have to create new MapCacheClientServices for each. Is this how it's supposed to work? Next I tried to recreate the ExecuteScript example, "Recipe: Get the value of a property stored in a DistributedMapCacheServer" here: https://community.cloudera.com/t5/Community-Articles/ExecuteScript-Cookbook-part-3/ta-p/249148 The Groovy code is import org.apache.nifi.distributed.cache.client.DistributedMapCacheClient import org.apache.nifi.distributed.cache.client.Serializer import org.apache.nifi.distributed.cache.client.Deserializer import java.nio.charset.StandardCharsets def StringSerializer = {value, out -> out.write(value.getBytes(StandardCharsets.UTF_8))} as Serializer<String> def StringDeserializer = { bytes -> new String(bytes) } as Deserializer<String> def myDistClient = clientServiceId.asControllerService(DistributedMapCacheClient) def result = myDistClient.get('a', StringSerializer, StringDeserializer) log.info("Result = $result") I received the error: ERROR ExecuteScript[id=587556de-0195-1000-c86f-dca652b949c6] Error Occured,java.lang.NullPointerException: Cannot invoke method get() on null object: {} I did create a new property in the ExecuteScript processor and use the Id from the MapCacheClientService in Controler Settings as per the directions. Then I replaced DistributedMapCacheClient in the code with MapCacheClientService and I received the following error: ERROR ExecuteScript[id=587556de-0195-1000-c86f-dca652b949c6] Processing failed: org.apache.nifi.processor.exception.ProcessException: javax.script.ScriptException: org.codehaus.groovy.control.MultipleCompilationErrorsException: startup failed: Script4.groovy: 13: unable to resolve class org.apache.nifi.distributed.cache.client.MapCacheClientService @ line 13, column 1. import org.apache.nifi.distributed.cache.client.MapCacheClientService ^ I found the nifi-distributed-cache-client-service-2.2.0.jar, and org.apache.nifi.distributed.cache.client.MapCacheClientService is a class within it, but I receive further errors when I placed the jar file in the Nifi lib directory. Is there a way to get the code in ExecuteScript-Cookbook-part-3 working? PutDistributedMapCache and FetchDistributedMapCache alone are not enough for my needs. Thanks for you help!
... View more
Labels:
- Labels:
-
Apache NiFi
02-21-2025
06:37 AM
I read about that possibility. Also with that, making a rest api call for a simple "global variable" within a process group or across a single process flow uses more processing resources and time than I would think is optimal, but I may be wrong.
... View more
02-21-2025
05:53 AM
I've read up a bit onDistributedMapCacheServer and setting up and accessing ports, and its purpose really feels larger than what I'm looking for. 😞 I'd just like the ability to maintain a simple counter within a process group that I can read, increment and reset, best within a groovy script. Parameters and Counters, or something else if it exists, would totally suit the purpose. Is there a way I can read counter values in ExecuteScript / ExecuteGroovyScript that I'm missing? Thanks!
... View more
02-20-2025
01:13 PM
Hi, Thanks for your reply. It's ok for me if the Counter values are lost when NiFi is restarted. I can deal with that. Can I read their values in ExecuteScript / ExecuteGroovyScript? I used this to increment it, but I wasn't able to read the value itself. session.adjustCounter("my_counter", 1, true) I'll look into Distributed Map Cache. I'm unfamiliar with it. Thank you!
... View more
02-20-2025
08:27 AM
Hi, I'm looking for a way to implement a process group level counter that I can read and update either with a processor, including ExecuteScript / ExecuteGroovyScript. I've looked at and tried: 1. Counters: I can create one, set the value and increment it, but not read the value within a processor, that I've found as of yet. 2. Parameter Context parameters: I can read the value, but not increment or reset it. It seems to be static once it's defined. Is there something I've missed with the above, or does anyone have another suggestion please? Thank you!
... View more
Labels:
- Labels:
-
Apache NiFi
01-22-2025
09:13 AM
To get the result I wanted this is the roundabout process I used. I would imagine there's a better way.
... View more