Support Questions
Find answers, ask questions, and share your expertise

Ingest data from MYSQL(MariaDB) to Hive.


I've put all the required configuration in the processors though it gave me couple of exceptions which I've solved and now everything looks fine but querydatabasetable processor not pushing any data into queue. Attached all the configurations. I'm a NIFI beginner please suggest me a solution for this.


Super Guru

What version of NiFi are you using? Does QueryDatabaseTable generate any errors? How often is it scheduled to run? Without a Maximum Value Column set, it should repeatedly fetch the whole table, so you could also try ExecuteSQL with a query of "SELECT * from flights" if you want that behavior. If you want it to fetch incrementally you will have to specify a Maximum Value column so the processor can keep track of the largest value (for that column) it's seen so far.

@Matt Burgess

Apache NiFi - Version

QueryDatabaseTable generated previously but I solved it. Now no errors.

I want to incrementally fetch the data. I dont know what's the Maximum value of column. Is it something like no.of records in that table. Please help me in figuring out maximum value.

Below attached how flow looks when I tried to start. It neither says started no popping of data nor an error.


Super Guru

To do incremental fetching with QueryDatabaseTable, you'll need a column in your table that always increases in value, such as an integer ID column (that keeps increasing with each row added) or a "last-updated" column that stores timestamps, or something of that nature.

For true Change Data Capture (CDC) of MySQL tables, consider using the CaptureChangeMySQL processor. I'm not sure if it works 100% with MariaDB but it is often a better solution for CDC use cases as you don't need to alter your table if you don't have a column that is "right" for incremental fetch.

@Matt Burgess

May I know what's the problem with the flow I've developed. Initially if I figure that atleast I can proceed for this incremental CDC.

Super Guru

I can't tell from your configuration why it is not grabbing rows from the table, since you have no errors and there is no Maximum-Value Column specified. Are there definitely rows in that table (you can verify with ExecuteSQL)?