question NIFI: SelectHiveQL vs QueryDatabaseTable in Archives of Support Questions (Read Only)

NIFI: SelectHiveQL vs QueryDatabaseTable

TimothySpann — Tue, 28 Feb 2017 22:30:52 GMT

I want to query Hive for only changed records ala QueryDatabaseTable.

Should I use QueryDatabaseTable or is there a way to make SelectHiveQL maintain current state?

Re: NIFI: SelectHiveQL vs QueryDatabaseTable

sunile_manjee — Tue, 28 Feb 2017 22:42:57 GMT

I would go with querydatabasetable. this will provide you state and also another important feature to break up return records into flow files. for example if 1000 records are expected to be a output of query, you can set Max Rows Per Flow File to x, and process data in smaller chunks. if you use selecthiveql, then build your query using update attribute and use a state via distributed map cache (DMC). maintain the last state in DMC and use that state in your updateattribute to run query in selecthiveql.

Flow

DMC fetch (state field) --> update attribute (build query) --> selecthiveql

You will have to set dmc to initial state value or set in logic via update attribute.

Re: NIFI: SelectHiveQL vs QueryDatabaseTable

Kumar78 — Mon, 12 Oct 2020 21:11:10 GMT

Hi,

Could you let me know if you have managed to query HIVE table using QueryDatabaseTableRecord processor as I am having issue in doing same please.

Re: NIFI: SelectHiveQL vs QueryDatabaseTable

VidyaSargur — Tue, 13 Oct 2020 05:43:07 GMT

Hi @Kumar78, as this is an older post, you would have a better chance of receiving a resolution by starting a new thread. This will also be an opportunity to provide details specific to your environment that could aid others in assisting you with a more accurate answer to your question.