NiFi JVM settings for large files.

[#1]

Hi,

Our current nifi jvm settings are

java.arg.2=-Xmx16g

java.arg.3=-Xms16g

i need to read a huge JSON file 22GB , mainly to replace white spaces from it.

i am planning to use the list-->fetch-->splittext-->replacetext-->mergecontent approach which i used earlier for similar use cases. but since the file now is bigger than the JVM , i am thikning i will get outofmemory errors since NiFi needs to read the file before it splits it. am i correct.?

i can change the jvm settings to use 32 or 48 gb , but just want to get expert opinion on this.??

Regards,

Sai

[#2]

@Bryan Bende I think that is what happening there. what options do I have now.?

[#3]

but even if it is treating as one line, I have 16gb for java heap and the file is only 7.5 gb.it seems like I am getting that error after ReplaceText reads 2gb.(if you look at the screen shot above)

[#4]

@Bryan Bende

looks like I have an option to use SegmentContent and MergeContent after I use ReplaceText on each segment.

I tried it with 10mb segment size on a 120 mb file and it worked..now will try on the bigger file.