Support Questions
Find answers, ask questions, and share your expertise

NiFi Database Extract Table

Highlighted

NiFi Database Extract Table

Contributor

Hi,

I would like to extract a big table (MySQL, more than 3 millions rows) and to write it as a file in HDFS.

What would be the best way to do it ?

I tried the following processors :

- ExecuteSQL : error : pb memory

- QueryDatabaseTable : error : pb memory

- GenerateTableFetch : error : failed to invoke @OnScheduled method due to java.lang.RuntimeException

I have 20 Go of memory.

What would be the best way to do it ? Can I set up parameters so that I generate more than 1 DataFlow, then merge in NiFi before loading to HDFS ?

Thank you.

5 REPLIES 5
Highlighted

Re: NiFi Database Extract Table

Hi @Raphaël MARY

What are your JVM memory settings? The standard is 512MB, which will likely result in OOM with a large query result set. Best to give NIFI as much memory as possible if you plan to do a lot of in memory workload, like working with large result sets in this case.

Re: NiFi Database Extract Table

Contributor

@Ward Bekker

yes settings are 512MB in NiFi for nifi.initial_mem and nifi.max_mem.

Is there a way to set the best values for these parameters ? Like 1/2 * Amout of RAM ?

Highlighted

Re: NiFi Database Extract Table

Ah, 512MB is probably to low for your use case. If you don't have a lot of other services running on your node, I would suggest to start with 80% of the node memory.

Highlighted

Re: NiFi Database Extract Table

Contributor

Thank you @Ward Bekker

Highlighted

Re: NiFi Database Extract Table

Super Guru

I like 12 - 16 GB for NiFi. That's a nice chunk of RAM.