Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

How can I improve performace of ExecuteSQL processor in NIFI

Highlighted

How can I improve performace of ExecuteSQL processor in NIFI

New Contributor

In my project, I am extracting data from Oracle DB using ExecuteSQL (using Select query joining 4 tables ) and ingesting data to POSTGRESDB using PutDataBaserecord processor

Issue : Execute SQL is taking 4-6 minutes to extract 50K records from DB. So for 1 million records it will take over 1 hour. At the same time PutDBrecordprocessor is inserting records in few seconds

Is there any other way to extract data from DB?

Execute SQL : Attributes given is like below MaxWaittime :10seconds Use Avro logical format :true Compression Format :None Max rows per flow file : 50000 (I have tried with 10K as well) Outputbatch size : 50000

Nifi is running in kubernetes with single cluster

Don't have an account?
Coming from Hortonworks? Activate your account here