question Re: How to stop QueryDatabaseTable after processing all the data? in Support Questions

How to stop QueryDatabaseTable after processing all the data?

karthikprasad44 — Wed, 19 Oct 2016 17:10:45 GMT

I am trying move a table from Oracle to HDFS. I used QueryDatabaseTable -> putHdfs processors and configured them. I can see data coming to hdfs.. but the process is running continuously and records are being added again and again. Am I doing anything wrong or missing something?

Re: How to stop QueryDatabaseTable after processing all the data?

bbende — Wed, 19 Oct 2016 19:25:57 GMT

How did you schedule QueryDatabaseTable?

If you didn't change anything on the scheduling tab of the processor, then the run schedule is 0 seconds which means as fast as possible. You most likely want to run this on some kind of timer or cron scheduling.

Re: How to stop QueryDatabaseTable after processing all the data?

karthikprasad44 — Wed, 19 Oct 2016 20:41:56 GMT

I did not change anything on scheduling tab as I wanted it to run asap. The problem is that the processor is running continuously. So If I have 10 records in actual table, I see the count on hdfs keep on increasing (dumping these 10 again and again). I expect it to stop after moving just these initial 10 records onto hdfs.

Re: How to stop QueryDatabaseTable after processing all the data?

bbende — Wed, 19 Oct 2016 20:55:48 GMT

If you use timer scheduling it will still execute right away, so if you set 30 seconds it will run right away then wait 30 seconds before running again.

Can you provide all of the configuration you entered for the processor?

You would need to provide the "Maximum-Value Columns" in order for it to track where it left off and pick up there on next execution.

Re: How to stop QueryDatabaseTable after processing all the data?

karthikprasad44 — Thu, 20 Oct 2016 18:13:41 GMT

Thanks you for the information. So, if I set "Run Schedule" to default i.e ) sec, It will run tasks one after other again and again. And If I want it to execute only once, I would make it to some huge value or use event or CRON timer.

Re: How to stop QueryDatabaseTable after processing all the data?

bbende — Thu, 20 Oct 2016 19:52:50 GMT

Correct. If you truly only want to run it once, then make the timer schedule larger and just manually start and stop the processor.

Re: How to stop QueryDatabaseTable after processing all the data?

srikanth_gunupu — Sat, 04 Mar 2017 22:14:56 GMT

@Bryan Bende

Hi Bryan,

Small question regarding your earlier reply.

In my scenario, the processor should run once when i started and it should run again when there is any update. I'm not sure that when the Database will get updated. So, i can't use Timer here. I have tried by specifying the "Maximun-value-columns", but no luck.

Can you please help me in finding the way to do this.

Thanks,

Srikanth.

Re: How to stop QueryDatabaseTable after processing all the data?

srikanth_gunupu — Mon, 06 Mar 2017 01:47:32 GMT

Thanks. Its worked after giving the schedule time.

Re: How to stop QueryDatabaseTable after processing all the data?

mburgess — Mon, 06 Mar 2017 23:42:00 GMT

What does your table look like? Is there a column that is guaranteed to be "strictly increasing" for each added/updated row? Sometimes this is the ID column (if using an autoincrementing integer that doesn't roll over), or perhaps a timestamp column such as "Last Updated". If you have no such column, then you will want to follow Bryan's advice on scheduling and start/stop.