NIFI: SelectHiveQL vs QueryDatabaseTable

[#1]

I want to query Hive for only changed records ala QueryDatabaseTable.

Should I use QueryDatabaseTable or is there a way to make SelectHiveQL maintain current state?

[#2 · Accepted solution]

I would go with querydatabasetable. this will provide you state and also another important feature to break up return records into flow files. for example if 1000 records are expected to be a output of query, you can set Max Rows Per Flow File to x, and process data in smaller chunks. if you use selecthiveql, then build your query using update attribute and use a state via distributed map cache (DMC). maintain the last state in DMC and use that state in your updateattribute to run query in selecthiveql.

Flow

DMC fetch (state field) --> update attribute (build query) --> selecthiveql

You will have to set dmc to initial state value or set in logic via update attribute.

[#3]

Hi, 

Could you let me know if you have managed to query HIVE table using QueryDatabaseTableRecord processor as I am having issue in doing same please.

[#4]

Hi @Kumar78, as this is an older post, you would have a better chance of receiving a resolution by starting a new thread. This will also be an opportunity to provide details specific to your environment that could aid others in assisting you with a more accurate answer to your question.