Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Ingest data from MYSQL(MariaDB) to Hive.

Highlighted

Ingest data from MYSQL(MariaDB) to Hive.

New Contributor

Hello,

I've put all the required configuration in the processors though it gave me couple of exceptions which I've solved and now everything looks fine but querydatabasetable processor not pushing any data into queue. Attached all the configurations. I'm a NIFI beginner please suggest me a solution for this.


puthivestreaming.jpgquerydatabasetable.jpgdbcp-connection-pool.jpg
5 REPLIES 5

Re: Ingest data from MYSQL(MariaDB) to Hive.

What version of NiFi are you using? Does QueryDatabaseTable generate any errors? How often is it scheduled to run? Without a Maximum Value Column set, it should repeatedly fetch the whole table, so you could also try ExecuteSQL with a query of "SELECT * from flights" if you want that behavior. If you want it to fetch incrementally you will have to specify a Maximum Value column so the processor can keep track of the largest value (for that column) it's seen so far.

Re: Ingest data from MYSQL(MariaDB) to Hive.

New Contributor

@Matt Burgess

Apache NiFi - Version 1.5.0.3.1.0.0-564

QueryDatabaseTable generated previously but I solved it. Now no errors.

I want to incrementally fetch the data. I dont know what's the Maximum value of column. Is it something like no.of records in that table. Please help me in figuring out maximum value.

Below attached how flow looks when I tried to start. It neither says started no popping of data nor an error.

78478-flow.jpg

Re: Ingest data from MYSQL(MariaDB) to Hive.

To do incremental fetching with QueryDatabaseTable, you'll need a column in your table that always increases in value, such as an integer ID column (that keeps increasing with each row added) or a "last-updated" column that stores timestamps, or something of that nature.

For true Change Data Capture (CDC) of MySQL tables, consider using the CaptureChangeMySQL processor. I'm not sure if it works 100% with MariaDB but it is often a better solution for CDC use cases as you don't need to alter your table if you don't have a column that is "right" for incremental fetch.

Re: Ingest data from MYSQL(MariaDB) to Hive.

New Contributor

@Matt Burgess

May I know what's the problem with the flow I've developed. Initially if I figure that atleast I can proceed for this incremental CDC.

Re: Ingest data from MYSQL(MariaDB) to Hive.

I can't tell from your configuration why it is not grabbing rows from the table, since you have no errors and there is no Maximum-Value Column specified. Are there definitely rows in that table (you can verify with ExecuteSQL)?

Don't have an account?
Coming from Hortonworks? Activate your account here