​How to scale SplitJson queues?

[#1]

I have processes that capture data from a SGDB => Converts to AvroJSON => SpliteJSON => Publish in Google PUBSUB

But it is accumulating and I would like to escalate the queues during the Split (putting 3 processors) and lasts the publish in Google it's possible?

[#2 · Accepted solution]

@Bruno Gomes de Souza

Make use of Record oriented processors to do your split on json array,

Try with the below approach

Once you feed the success relation to SplitRecord processor then you need to define RecordReader Controller service to read the contents of flowfile and Record Writer as JsonRecordSetWriter.

Mention the Records per split property value as 1 and feed only the splits relationship from SplitRecord processor to PublishGCPubsub processor.

If you find any OOM issues then it's better to use Series of SplitRecord processors to Make Records Per split to 1 message into each flowfile.

Refer to this and this links regarding usage of series of split processors.

Refer to this link regarding configuring Record Reader/Writer controller services.

-

[#3]

Thanks very much @Shu