Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

How to display the delta changes for a db using nifi

avatar
Explorer

I have a requirement like, I will be using mysql database and cassandra database.So I should be able to read data from both the databases based on hourly,weekly,monthly or specified date range. When we read the table from database it should display the delta changes for that table. So which nifi processors I should use to show delta changes for both mysql and cassandra db

5 REPLIES 5

avatar
Expert Contributor

You can use "QueryDatabaseTable" as input processor, but you can´t use it twice in one flow (Im searching for a solution that I can get a delta in one flow for 2 Fields btw. :D).

You have to set the Property "Maximum-value Columns". It can be a date, id or something else to have the maximum of a value.

At "view state" you can see the value of "Maximum-value Columns" field you set.

To set up the runtime you can go to "Schedule" at the processor and change the "Run Schedule"

 

If you need more input, please let me know

avatar
Explorer

Thanks @Faerballert , as you suggested I did the following 

maximum-value-column.JPG

Set the Maximum-Value-Columns with modified_time, later I checked the View state for the modified_time column and able to see the modified column which I updated.

View-State.JPG

I want to know How I can get the changed data over the period of time, for example if I have modified the column in last 1hr/1 day/1 week then I should be able to see all the modified data.

avatar
Expert Contributor

Go to the Processor and select "Scheduling".

As Scheduling strategy you can use "Cron driven"

and at the run schedule there you can set, when the Processor should be check for a Delta.

 

Examples:

Every 2 hours

0 0 0/2 1/1 * ? *

Daily at 1 a.m.

0 0 1 1/1 * ? *

Weekly at 3 p.m. on every monday

0 0 15 ? * MON *

avatar
Explorer

Hi @Faerballert ,

How can I achieve the same delta changes on cassandra. could u please share some processors which I can use and if I'm using QueryDatabaseTable its asking for jdbc driver and jar details. So please share your thought

avatar
Explorer
  1. Download HDP Sandbox
  2. MySQL database (Should already be present in the sandbox)
  3. NiFi 0.6 or later (Download and install a new version of NIFI or use Ambari to install NIFI in the sandbox)

Use this code:

 

 



insert into cdc_test values (3, 'cdc3', null, null);

insert into cdc_test values (4, 'cdc3', null, null);

insert into cdc_test values (5, 'cdc3', null, null);

insert into cdc_test values (6, 'cdc3', null, null);

insert into cdc_test values (7, 'cdc3', null, null);

insert into cdc_test values (8, 'cdc3', null, null);

insert into cdc_test values (9, 'cdc3', null, null);

insert into cdc_test values (10, 'cdc3', null, null);

insert into cdc_test values (11, 'cdc3', null, null);

insert into cdc_test values (12, 'cdc3', null, null);

insert into cdc_test values (13, 'cdc3', null, null);

 

 

I hope this helps!

Regards,

Lewis