Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Impala support for INPUT__FILE__NAME in hive

Impala support for INPUT__FILE__NAME in hive

New Contributor

Hi,

 

Is there a way (a virtual column or function) to get the record's filename in impala queries just similar to INPUT__FILE__NAME in hive.

There is a jira about that, however there is nothing done by anybody with the jira.

 

Thanks

5 REPLIES 5

Re: Impala support for INPUT__FILE__NAME in hive

Master Collaborator
Hi hakki,
We haven't implemented that yet. Currently I think the only way to get
that info is to run "Show files".

- Tim

Re: Impala support for INPUT__FILE__NAME in hive

New Contributor
With show files, I dont know the record corresponds to which file.

Re: Impala support for INPUT__FILE__NAME in hive

Expert Contributor

Hi hakki,

 

You could try partitioning your table and using the "show partitions" command to narrow down the set of files a row might be in. Can you give more context on the usecase without sharing private information?

 

Cheers, Lars

Highlighted

Re: Impala support for INPUT__FILE__NAME in hive

New Contributor

Yes, I thought partitioning on the pattern of filename. For one type of data I have, it is acceptable since there are only 10-15 different types of file name. However; in another type of data, there is a unique "id" field on filename and this field's frequency is very high. So, there would be the threat of creation of lots of partitions which could be compelling for the catalog server.

Re: Impala support for INPUT__FILE__NAME in hive

Master Collaborator

That's a good point. I updated the JIRA description to provide that additional motivation.

 

As an open-source project, we're somewhat dependent on people finding time to pick up new features like this that are nice-to-have but not critical for many users.