01-26-2017 03:25 AM
Is there a way (a virtual column or function) to get the record's filename in impala queries just similar to INPUT__FILE__NAME in hive.
There is a jira about that, however there is nothing done by anybody with the jira.
01-27-2017 08:16 AM
You could try partitioning your table and using the "show partitions" command to narrow down the set of files a row might be in. Can you give more context on the usecase without sharing private information?
01-27-2017 08:19 AM
That's a good point. I updated the JIRA description to provide that additional motivation.
As an open-source project, we're somewhat dependent on people finding time to pick up new features like this that are nice-to-have but not critical for many users.
01-29-2017 05:33 AM
Yes, I thought partitioning on the pattern of filename. For one type of data I have, it is acceptable since there are only 10-15 different types of file name. However; in another type of data, there is a unique "id" field on filename and this field's frequency is very high. So, there would be the threat of creation of lots of partitions which could be compelling for the catalog server.