Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

NiFi Database Extract Table

Highlighted

NiFi Database Extract Table

Contributor

Hi,

I would like to extract a big table (MySQL, more than 3 millions rows) and to write it as a file in HDFS.

What would be the best way to do it ?

I tried the following processors :

- ExecuteSQL : error : pb memory

- QueryDatabaseTable : error : pb memory

- GenerateTableFetch : error : failed to invoke @OnScheduled method due to java.lang.RuntimeException

I have 20 Go of memory.

What would be the best way to do it ? Can I set up parameters so that I generate more than 1 DataFlow, then merge in NiFi before loading to HDFS ?

Thank you.

5 REPLIES 5

Re: NiFi Database Extract Table

Hi @Raphaël MARY

What are your JVM memory settings? The standard is 512MB, which will likely result in OOM with a large query result set. Best to give NIFI as much memory as possible if you plan to do a lot of in memory workload, like working with large result sets in this case.

Highlighted

Re: NiFi Database Extract Table

Contributor

@Ward Bekker

yes settings are 512MB in NiFi for nifi.initial_mem and nifi.max_mem.

Is there a way to set the best values for these parameters ? Like 1/2 * Amout of RAM ?

Highlighted

Re: NiFi Database Extract Table

Ah, 512MB is probably to low for your use case. If you don't have a lot of other services running on your node, I would suggest to start with 80% of the node memory.

Highlighted

Re: NiFi Database Extract Table

Contributor

Thank you @Ward Bekker

Highlighted

Re: NiFi Database Extract Table

Super Guru

I like 12 - 16 GB for NiFi. That's a nice chunk of RAM.

Don't have an account?
Coming from Hortonworks? Activate your account here