- Subscribe to RSS Feed
- Mark Question as New
- Mark Question as Read
- Float this Question for Current User
- Bookmark
- Subscribe
- Mute
- Printer Friendly Page
How to display the delta changes for a db using nifi
- Labels:
-
Apache NiFi
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
I have a requirement like, I will be using mysql database and cassandra database.So I should be able to read data from both the databases based on hourly,weekly,monthly or specified date range. When we read the table from database it should display the delta changes for that table. So which nifi processors I should use to show delta changes for both mysql and cassandra db
Created on ‎01-16-2020 03:30 AM - edited ‎01-16-2020 04:59 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
You can use "QueryDatabaseTable" as input processor, but you can´t use it twice in one flow (Im searching for a solution that I can get a delta in one flow for 2 Fields btw. :D).
You have to set the Property "Maximum-value Columns". It can be a date, id or something else to have the maximum of a value.
At "view state" you can see the value of "Maximum-value Columns" field you set.
To set up the runtime you can go to "Schedule" at the processor and change the "Run Schedule"
If you need more input, please let me know
Created ‎02-16-2020 11:26 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Thanks @Faerballert , as you suggested I did the following
Set the Maximum-Value-Columns with modified_time, later I checked the View state for the modified_time column and able to see the modified column which I updated.
I want to know How I can get the changed data over the period of time, for example if I have modified the column in last 1hr/1 day/1 week then I should be able to see all the modified data.
Created ‎02-17-2020 05:23 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Go to the Processor and select "Scheduling".
As Scheduling strategy you can use "Cron driven"
and at the run schedule there you can set, when the Processor should be check for a Delta.
Examples:
Every 2 hours
0 0 0/2 1/1 * ? *
Daily at 1 a.m.
0 0 1 1/1 * ? *
Weekly at 3 p.m. on every monday
0 0 15 ? * MON *
Created ‎03-04-2020 09:46 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi @Faerballert ,
How can I achieve the same delta changes on cassandra. could u please share some processors which I can use and if I'm using QueryDatabaseTable its asking for jdbc driver and jar details. So please share your thought
Created on
‎01-16-2020
06:27 AM
- last edited on
‎03-04-2020
01:46 PM
by
ask_bill_brooks
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
- Download HDP Sandbox
- MySQL database (Should already be present in the sandbox)
- NiFi 0.6 or later (Download and install a new version of NIFI or use Ambari to install NIFI in the sandbox)
Use this code:
insert into cdc_test values (3, 'cdc3', null, null);
insert into cdc_test values (4, 'cdc3', null, null);
insert into cdc_test values (5, 'cdc3', null, null);
insert into cdc_test values (6, 'cdc3', null, null);
insert into cdc_test values (7, 'cdc3', null, null);
insert into cdc_test values (8, 'cdc3', null, null);
insert into cdc_test values (9, 'cdc3', null, null);
insert into cdc_test values (10, 'cdc3', null, null);
insert into cdc_test values (11, 'cdc3', null, null);
insert into cdc_test values (12, 'cdc3', null, null);
insert into cdc_test values (13, 'cdc3', null, null);
I hope this helps!
Regards,
Lewis
