Hi,
Getting started with Nifi, and i have a table thats about 2 million records. Here is my data flow screen-shot-2018-08-17-at-35302-pm.png
Have a querydatabaserecord, which uses the tables updated_at date for syncing. I limited the fetch size to 100, and the schedule of the job every minute, thinking that it would take its time and sync over a few days. I ran this with another smaller table and its fine, and it syncs data over to my redshift db.
I've tried twice now, and it seems like when i start this, the nifi interface freezes, and then if i refresh browser i can't get to the nifi interface. and then it takes an hour of restarting all the services and nifi services over and over again. Once it comes back up, had an error about jvm stack memory overflow. The table is about 3GB.
My cluster is a 3 node cluster. So i have ambari on its own server, 4 nifi nodes, one which is the manager, and 3 which are the "workers", i guess. Each server has 30gb ram, and are aws ec2 r5.xlarge.
Any thougths or ideas? I think its using all of the memory, but if someone could help me with how to check or not, maybe thats the problem.
Thanks,
Ron
