We have a nifi two node cluster on vm ,and need to process the data to cosmosDb graph api from azure sql server,i have written a custom processors to get the data from sql server and import into graph api, I have 6 connection stirngs for each user that means 6 flowfiles :
Flow like:first populate the users,items,and items they have bought into graphDb ,like this ,everything is fine fetching data but problem is while inserting into graph db ,each flowfile size 12mb to 30 mb data .
even we have 2 node cluster ,but processing the data only in one node not processing by both ,so its taking time and hanging and getting heap out of memory exception ,not able to insert all users data
Could you plz explain the cluster how to configure two process the data by two nodes .
@siva karna It appears that your processor is configured to run only on 'primary node'. Please check the scheduling tab in the processor configuration, and if it is configured to run only on 'Primary Node' configure it to run on 'All nodes'. Check screenshot.