Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Nifi HDF initial table sync crashing interface/server for querydatabasetable

avatar
Contributor

Hi,

Getting started with Nifi, and i have a table thats about 2 million records. Here is my data flow screen-shot-2018-08-17-at-35302-pm.png

Have a querydatabaserecord, which uses the tables updated_at date for syncing. I limited the fetch size to 100, and the schedule of the job every minute, thinking that it would take its time and sync over a few days. I ran this with another smaller table and its fine, and it syncs data over to my redshift db.

I've tried twice now, and it seems like when i start this, the nifi interface freezes, and then if i refresh browser i can't get to the nifi interface. and then it takes an hour of restarting all the services and nifi services over and over again. Once it comes back up, had an error about jvm stack memory overflow. The table is about 3GB.

My cluster is a 3 node cluster. So i have ambari on its own server, 4 nifi nodes, one which is the manager, and 3 which are the "workers", i guess. Each server has 30gb ram, and are aws ec2 r5.xlarge.

Any thougths or ideas? I think its using all of the memory, but if someone could help me with how to check or not, maybe thats the problem.

Thanks,

Ron

1 ACCEPTED SOLUTION

avatar
Contributor

Think I found that is was just the initial memory allocation only being at the default 512mb. upped it to 25gb and seems to be running fine.

View solution in original post

1 REPLY 1

avatar
Contributor

Think I found that is was just the initial memory allocation only being at the default 512mb. upped it to 25gb and seems to be running fine.