https://community.cloudera.com/t5/Support-Questions/How-to-scale-SplitJson-queues/m-p/182433#M144599 <A rel="user" href="https://community.cloudera.com/users/87243/bruno.html" nodeid="87243" target="_blank">@Bruno Gomes de Souza</A><P>Make use of Record oriented processors to do your split on json array, </P><P>Try with the below approach</P><P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="79457-flow.png" style="width: 1351px;"><img src="https://community.cloudera.com/t5/image/serverpage/image-id/19166i30C4723BB04515FC/image-size/medium?v=v2&amp;px=400" role="button" title="79457-flow.png" alt="79457-flow.png" /></span></P><P>Once you feed the success relation to <STRONG>SplitRecord </STRONG>processor then you need to define <STRONG>RecordReader </STRONG>Controller service to read the contents of flowfile and Record Writer as <STRONG>JsonRecordSetWriter</STRONG>.</P><P>Mention the Records per split property value as 1 and <STRONG>feed only the splits relationship</STRONG> from SplitRecord processor to PublishGCPubsub processor.</P><P>If you find any OOM issues then it's better to use Series of SplitRecord processors to Make Records Per split to 1 message into each flowfile.</P><P>Refer to <A href="https://community.hortonworks.com/content/kbentry/144771/ingesting-a-big-csv-file-into-kafka-using-a-multi.html" target="_blank" rel="nofollow noopener noreferrer">this</A> and <A href="https://community.hortonworks.com/questions/122858/nifi-splittext-big-file.html" target="_blank" rel="nofollow noopener noreferrer">this</A> links regarding usage of series of split processors.</P><P>Refer to <A href="https://community.hortonworks.com/articles/115311/convert-csv-to-json-avro-xml-using-convertrecord-p.html" target="_blank" rel="nofollow noopener noreferrer">this</A> link regarding configuring Record Reader/Writer controller services.</P><P>-</P> Sun, 18 Aug 2019 08:54:37 GMT Shu_ashu 2019-08-18T08:54:37Z https://community.cloudera.com/t5/Support-Questions/How-to-scale-SplitJson-queues/m-p/182432#M144598 <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="80478-nifi-test-template-scale.png" style="width: 717px;"><img src="https://community.cloudera.com/t5/image/serverpage/image-id/19167i44459EFEB0DEA06A/image-size/medium?v=v2&amp;px=400" role="button" title="80478-nifi-test-template-scale.png" alt="80478-nifi-test-template-scale.png" /></span></P><P>I have processes that capture data from a SGDB =&gt; Converts to AvroJSON =&gt; SpliteJSON =&gt; Publish in Google PUBSUB </P><P>But it is accumulating and I would like to escalate the queues during the Split (putting 3 processors) and lasts the publish in Google it's possible?</P> Sun, 18 Aug 2019 08:54:45 GMT https://community.cloudera.com/t5/Support-Questions/How-to-scale-SplitJson-queues/m-p/182432#M144598 bruno1 2019-08-18T08:54:45Z https://community.cloudera.com/t5/Support-Questions/How-to-scale-SplitJson-queues/m-p/182433#M144599 <A rel="user" href="https://community.cloudera.com/users/87243/bruno.html" nodeid="87243" target="_blank">@Bruno Gomes de Souza</A><P>Make use of Record oriented processors to do your split on json array, </P><P>Try with the below approach</P><P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="79457-flow.png" style="width: 1351px;"><img src="https://community.cloudera.com/t5/image/serverpage/image-id/19166i30C4723BB04515FC/image-size/medium?v=v2&amp;px=400" role="button" title="79457-flow.png" alt="79457-flow.png" /></span></P><P>Once you feed the success relation to <STRONG>SplitRecord </STRONG>processor then you need to define <STRONG>RecordReader </STRONG>Controller service to read the contents of flowfile and Record Writer as <STRONG>JsonRecordSetWriter</STRONG>.</P><P>Mention the Records per split property value as 1 and <STRONG>feed only the splits relationship</STRONG> from SplitRecord processor to PublishGCPubsub processor.</P><P>If you find any OOM issues then it's better to use Series of SplitRecord processors to Make Records Per split to 1 message into each flowfile.</P><P>Refer to <A href="https://community.hortonworks.com/content/kbentry/144771/ingesting-a-big-csv-file-into-kafka-using-a-multi.html" target="_blank" rel="nofollow noopener noreferrer">this</A> and <A href="https://community.hortonworks.com/questions/122858/nifi-splittext-big-file.html" target="_blank" rel="nofollow noopener noreferrer">this</A> links regarding usage of series of split processors.</P><P>Refer to <A href="https://community.hortonworks.com/articles/115311/convert-csv-to-json-avro-xml-using-convertrecord-p.html" target="_blank" rel="nofollow noopener noreferrer">this</A> link regarding configuring Record Reader/Writer controller services.</P><P>-</P> Sun, 18 Aug 2019 08:54:37 GMT https://community.cloudera.com/t5/Support-Questions/How-to-scale-SplitJson-queues/m-p/182433#M144599 Shu_ashu 2019-08-18T08:54:37Z https://community.cloudera.com/t5/Support-Questions/How-to-scale-SplitJson-queues/m-p/182434#M144600 <P>Thanks very much <A rel="user" href="https://community.cloudera.com/users/18929/yaswanthmuppireddy.html" nodeid="18929">@Shu</A></P> Fri, 13 Jul 2018 03:04:21 GMT https://community.cloudera.com/t5/Support-Questions/How-to-scale-SplitJson-queues/m-p/182434#M144600 bruno1 2018-07-13T03:04:21Z