Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

How to display the delta changes for a db using nifi

Highlighted

How to display the delta changes for a db using nifi

Explorer

I have a requirement like, I will be using mysql database and cassandra database.So I should be able to read data from both the databases based on hourly,weekly,monthly or specified date range. When we read the table from database it should display the delta changes for that table. So which nifi processors I should use to show delta changes for both mysql and cassandra db

5 REPLIES 5
Highlighted

Re: How to display the delta changes for a db using nifi

Contributor

You can use "QueryDatabaseTable" as input processor, but you can´t use it twice in one flow (Im searching for a solution that I can get a delta in one flow for 2 Fields btw. :D).

You have to set the Property "Maximum-value Columns". It can be a date, id or something else to have the maximum of a value.

At "view state" you can see the value of "Maximum-value Columns" field you set.

To set up the runtime you can go to "Schedule" at the processor and change the "Run Schedule"

 

If you need more input, please let me know

Highlighted

Re: How to display the delta changes for a db using nifi

Explorer

Thanks @Faerballert , as you suggested I did the following 

maximum-value-column.JPG

Set the Maximum-Value-Columns with modified_time, later I checked the View state for the modified_time column and able to see the modified column which I updated.

View-State.JPG

I want to know How I can get the changed data over the period of time, for example if I have modified the column in last 1hr/1 day/1 week then I should be able to see all the modified data.

Highlighted

Re: How to display the delta changes for a db using nifi

Contributor

Go to the Processor and select "Scheduling".

As Scheduling strategy you can use "Cron driven"

and at the run schedule there you can set, when the Processor should be check for a Delta.

 

Examples:

Every 2 hours

0 0 0/2 1/1 * ? *

Daily at 1 a.m.

0 0 1 1/1 * ? *

Weekly at 3 p.m. on every monday

0 0 15 ? * MON *

Re: How to display the delta changes for a db using nifi

Explorer

Hi @Faerballert ,

How can I achieve the same delta changes on cassandra. could u please share some processors which I can use and if I'm using QueryDatabaseTable its asking for jdbc driver and jar details. So please share your thought

Highlighted

Re: How to display the delta changes for a db using nifi

Explorer
  1. Download HDP Sandbox
  2. MySQL database (Should already be present in the sandbox)
  3. NiFi 0.6 or later (Download and install a new version of NIFI or use Ambari to install NIFI in the sandbox)

Use this code:

 

 



insert into cdc_test values (3, 'cdc3', null, null);

insert into cdc_test values (4, 'cdc3', null, null);

insert into cdc_test values (5, 'cdc3', null, null);

insert into cdc_test values (6, 'cdc3', null, null);

insert into cdc_test values (7, 'cdc3', null, null);

insert into cdc_test values (8, 'cdc3', null, null);

insert into cdc_test values (9, 'cdc3', null, null);

insert into cdc_test values (10, 'cdc3', null, null);

insert into cdc_test values (11, 'cdc3', null, null);

insert into cdc_test values (12, 'cdc3', null, null);

insert into cdc_test values (13, 'cdc3', null, null);

 

 

I hope this helps!

Regards,

Lewis

Don't have an account?
Coming from Hortonworks? Activate your account here