Using MapCacheServer and MapCacheClientService in NiFi versions 2.0.0 / 2.2.0

[#1]

Hi,

I'm still trying to get this to work:

https://community.cloudera.com/t5/Support-Questions/Process-Group-level-counter/td-p/402648

I have versions NiFi 2.0.0 and 2.2.0 at my disposal, and have tried to get simple examples working that I can then implement in my flow. I'm having difficulties as all the examples I find are with older versions of NiFi.

First, Distributed Server and Client Services have been renamed:

DistributedMapCacheServer has been renamed to MapCacheServer

DistributedSetCacheServer has been renamed to SetCacheServer

DistributedMapCacheClientService has been renamed to MapCacheClientService

DistributedSetCacheClientService has been renamed to SetCacheClientService

I've mostly gotten PutDistributedMapCache and FetchDistributedMapCache working as per the older example here:

https://community.cloudera.com/t5/Support-Questions/how-to-put-data-in-PutDistributedMapCache/td-p/1...

and

https://www.youtube.com/watch?v=wjis28NrF90

In the Controller Settings under Management Controller Services, I create a MapCacheServer and a MapCacheClientService set to use "localhost" as the Server Hostname.

One thing I've noticed is that it's not possible to select the MapCacheClientService I created. In PutDistributedMapCache and FetchDistributedMapCache I have to create new MapCacheClientServices for each.

Is this how it's supposed to work?

Next I tried to recreate the ExecuteScript example, "Recipe: Get the value of a property stored in a DistributedMapCacheServer" here:

https://community.cloudera.com/t5/Community-Articles/ExecuteScript-Cookbook-part-3/ta-p/249148

The Groovy code is

import org.apache.nifi.distributed.cache.client.DistributedMapCacheClient

import org.apache.nifi.distributed.cache.client.Serializer

import org.apache.nifi.distributed.cache.client.Deserializer

import java.nio.charset.StandardCharsets

def StringSerializer = {value, out -> out.write(value.getBytes(StandardCharsets.UTF_8))} as Serializer<String>

def StringDeserializer = { bytes -> new String(bytes) } as Deserializer<String>

def myDistClient = clientServiceId.asControllerService(DistributedMapCacheClient)

def result = myDistClient.get('a', StringSerializer, StringDeserializer)

log.info("Result = $result")

I received the error:

ERROR

ExecuteScript[id=587556de-0195-1000-c86f-dca652b949c6] Error Occured,java.lang.NullPointerException: Cannot invoke method get() on null object: {}

I did create a new property in the ExecuteScript processor and use the Id from the MapCacheClientService in Controler Settings as per the directions.

Then I replaced DistributedMapCacheClient in the code with MapCacheClientService and I received the following error:

ERROR

ExecuteScript[id=587556de-0195-1000-c86f-dca652b949c6] Processing failed: org.apache.nifi.processor.exception.ProcessException: javax.script.ScriptException: org.codehaus.groovy.control.MultipleCompilationErrorsException: startup failed:

Script4.groovy: 13: unable to resolve class org.apache.nifi.distributed.cache.client.MapCacheClientService

@ line 13, column 1.

import org.apache.nifi.distributed.cache.client.MapCacheClientService

^

I found the nifi-distributed-cache-client-service-2.2.0.jar, and org.apache.nifi.distributed.cache.client.MapCacheClientService is a class within it, but I receive further errors when I placed the jar file in the Nifi lib directory.

Is there a way to get the code in ExecuteScript-Cookbook-part-3 working? PutDistributedMapCache and FetchDistributedMapCache alone are not enough for my needs.

Thanks for you help!

[#2 · Accepted solution]

@Emery 

Your first query:

One thing I've noticed is that it's not possible to select the MapCacheClientService I created.

Components added to the NiFi canvas only have access to the controller services created on Process Groups (PG) on the canvas.  Even when you are looking at the canvas presented when you first login, you are looking at the root PG.  From the root PG you can create many child PGs.  

When you use the Global Menu in the UI to access the "Controller Settings", you have the ability to create "Management Controller services".  Controller services created here are for use by Reporting tasks and Registry Clients created from with the same "Controller Settings" UI.  They are not directly going to be referenced by the components on the dataflow canvas.   This is why the MapCacheClientService you created was not see by your PutMapCache and FetchMapCache processors.  

From within the processor component, you have the option to select existing supported controller service that exist already within the current PG level or any parent level PG (assuming user has proper permissions at those parent levels).  It is important to understand the Child PGs inherit policies from parent PGs unless an explicit policy is defined on the child PG.  You also have the option to "Create New Service", which you can select even if an available controller service already exists..

If a supported controller service exists, it will be presented in a selectable list when you click on the processor field, so it is NOT necessary to create a separate controller service for each processor.

To create a new service you must click on the three stacked dots instead of clicking on the field.

Please help our community grow. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped.

Thank you,
Matt

[#3]

Ah! Thank you for the explanation! I was always clicking on the three stacked dots and never saw the option to pick my existing instance!

Umm, any thoughts about what I can do to access the map cache via ExecuteScript as described in https://community.cloudera.com/t5/Community-Articles/ExecuteScript-Cookbook-part-3/ta-p/249148 ? 🙂

[#4]

@MattWho  Hi Matt, I made changes in my test based on the description you provided and I've been able to get the Groovy code working in ExecuteScript also.

Thanks very much for your help! Back to the counter now. 🙂