question Re: EvaulateJSONPath processor configuration in Support Questions

EvaulateJSONPath processor configuration

manish_parab — Fri, 18 Jan 2019 07:15:54 GMT

hello everyone,

I have a query regarding EvaulateJSONPath processor configuration.

whats the significance of Execution in Scheduling of EvaulateJSONPath.

I have observed when I select Primary node instead of All nodes, Queue builds up as shown in image and stays the same, dosen't decrease , but if I select All nodes Queue gradually becomes 0.

Please suggest.

screen-shot-2019-01-17-at-60906-pm.png

Re: EvaulateJSONPath processor configuration

Shu_ashu — Fri, 18 Jan 2019 08:32:43 GMT

@Manish Parab

Your GetMongo processor is running on All nodes that means same data is pulled on all nodes..
If you select EvaluateJsonPath processor to run on Only primary node then all the other nodes flowfiles will be left queue before EvaluateJsonPath processor, Because you are not processing flowfiles that are pulled from all other nodes except of PrimaryNode.

Run GetMongoProcessor only on primary node and keep EvaluateJsonPath Processor to run on all nodes, Reason to keep EvaluateJsonPath processor on all nodes if NiFi primary node changed then EvaluateJsonPath processor not going to processor the flowfile that are listed on old PrimaryNode.

Re: EvaulateJSONPath processor configuration

manish_parab — Fri, 18 Jan 2019 09:41:10 GMT

Thats a really good answer, I tried doing that and it works, thanks @Shu , can you please give me more insight on "same data is pulled on all nodes "

Re: EvaulateJSONPath processor configuration

Shu_ashu — Fri, 18 Jan 2019 10:43:14 GMT

@Manish Parab

Sure, In NiFi processors that triggers the flow(scheduled to run in cron) we need to run the processors on primary node only and running on all nodes means we are triggering n times the same processor on each node.
That means NiFi each node works with data specifically that receives, in case of Getmongo processor(triggers the flow in this case) when running on all nodes will pull same data.

Run GetMongo(source processor) to run on primary node then distribute the load using RemoteProcessorGroups (or) connectionloadbalancing across the cluster.

Re: EvaulateJSONPath processor configuration

manish_parab — Fri, 18 Jan 2019 11:46:50 GMT

makes sense now, why I was getting duplicate data from mongo... when I was running GetMongo on all nodes. thanks again @Shu