question Impala view based on UDF in Support Questions

Impala view based on UDF

ChineduLB — Sat, 05 Oct 2019 14:27:51 GMT

we have a beeline process that inserts data into a hive table

we can only access the table via impala views

so when the beeline job is done,

we want to refresh the tables and select rows from the table via impala views

could we write a user defined function that

1. refreshes the table in impala

2. selects * from the refreshed table

call that UDF from an impala view

Re: Impala view based on UDF

ChineduLB — Sat, 05 Oct 2019 19:07:43 GMT

So basically a situation where Hive is updating a table and impala clients are querying same table.

Sometimes the impala queries throw missing hdfs files exception.

How do we handle this

Re: Impala view based on UDF

tmater — Thu, 17 Oct 2019 08:26:30 GMT

Hi @ChineduLB,

UDFs let you code your own application logic for processing column values during an Impala query. Adding a refresh/invalidate to it could cause unexpected behavior during value processing.

A general recommendation for Invalidate metadata/Refresh is to execute it after the ingestion finished. This way the Impala user does not have to worry about the staleness of the metadata. There is a blogpost on how to handle "Fast Data" and make it available to Impala in batches:

https://blog.cloudera.com/how-to-ingest-and-query-fast-data-with-impala-without-kudu/

Additionally, just wanted to mention that the Invalidate metadata/Refresh can be executed from beeline as well, just need to connect from beeline to Impala, this blogpost has the details:

https://www.ericlin.me/2017/04/how-to-use-beeline-to-connect-to-impala/