Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Impala support for INPUT__FILE__NAME in hive

avatar
New Contributor

Hi,

 

Is there a way (a virtual column or function) to get the record's filename in impala queries just similar to INPUT__FILE__NAME in hive.

There is a jira about that, however there is nothing done by anybody with the jira.

 

Thanks

5 REPLIES 5

avatar
Hi hakki,
We haven't implemented that yet. Currently I think the only way to get
that info is to run "Show files".

- Tim

avatar
New Contributor
With show files, I dont know the record corresponds to which file.

avatar
Super Collaborator

Hi hakki,

 

You could try partitioning your table and using the "show partitions" command to narrow down the set of files a row might be in. Can you give more context on the usecase without sharing private information?

 

Cheers, Lars

avatar
New Contributor

Yes, I thought partitioning on the pattern of filename. For one type of data I have, it is acceptable since there are only 10-15 different types of file name. However; in another type of data, there is a unique "id" field on filename and this field's frequency is very high. So, there would be the threat of creation of lots of partitions which could be compelling for the catalog server.

avatar

That's a good point. I updated the JIRA description to provide that additional motivation.

 

As an open-source project, we're somewhat dependent on people finding time to pick up new features like this that are nice-to-have but not critical for many users.