question Re: Retrieving Impala data using SQL, only data records since last poll using the Date/Timestamp in Support Questions

Retrieving Impala data using SQL, only data records since last poll using the Date/Timestamp

StuartM — Wed, 18 Jan 2023 10:56:54 GMT

I want to poll an Impala table using SQL queries and only want to retrieve data records added since I last polled. This would be using a Date/Timestamp column in the table. Does anyone know how to achieve this?

I am happy to hold the last Date/Timestamp retrieved, in my source system

Re: Retrieving Impala data using SQL, only data records since last poll using the Date/Timestamp

mszurap — Thu, 19 Jan 2023 16:20:38 GMT

Hi @StuartM , I know it's not a direct answer, but this requirement sounds more like a good call for Kafka - which inherently supports the idea of "consumer offsets".

Re: Retrieving Impala data using SQL, only data records since last poll using the Date/Timestamp

VidyaSargur — Thu, 02 Feb 2023 08:08:13 GMT

@StuartM, Has the reply helped resolve your issue? If so, please mark the appropriate reply as the solution, as it will make it easier for others to find the answer in the future.

Re: Retrieving Impala data using SQL, only data records since last poll using the Date/Timestamp

FlorianHaas — Mon, 29 Apr 2024 18:51:49 GMT

To retrieve data records after crm enrichment added since the last time you polled an Impala table using a Date/Timestamp column, you can use a SQL query with a WHERE clause filtering for records with timestamps greater than the last timestamp you retrieved. Here's a basic example assuming your timestamp column is named timestamp_column:

SELECT *
FROM your_table
WHERE timestamp_column > 'last_poll_timestamp';

Replace 'last_poll_timestamp' with the actual timestamp value you stored from your last poll. Make sure the timestamp format matches the format stored in your table.

Here's a step-by-step guide:

Store the timestamp of the last poll in your source system.
Use this timestamp to construct your SQL query, ensuring you're retrieving records with timestamps greater than the last poll timestamp.
Execute the SQL query against your Impala table to retrieve the new records.