question Re: what is the best practice on where to store hive external tables in HDFS? in Support Questions

what is the best practice on where to store hive external tables in HDFS?

ryu — Tue, 16 Feb 2021 19:37:18 GMT

Hi,

I was just wondering what is the best practice on where to store hive external tables in HDFS?

Currently they are being stored in the /tmp directory in HDFS but sometimes someone accidentally deletes the files under /tmp and right now I am trying to add permissions to ranger to prevent this from happening.

Thanks,

Re: what is the best practice on where to store hive external tables in HDFS?

tusharkathpal — Thu, 18 Feb 2021 18:38:58 GMT

Hello @ryu

There is no such path as best path but obviously not /tmp location.

You can create some path under /user/external_tables and further create the tables here. Again it totally depends upon you how you are designing and your use case.

Re: what is the best practice on where to store hive external tables in HDFS?

ryu — Thu, 18 Feb 2021 23:59:38 GMT

Thanks @tusharkathpal for the response.

Can you please let me know what are some of the use cases to storing the external tables at a certain location?

Thanks,

Re: what is the best practice on where to store hive external tables in HDFS?

tusharkathpal — Fri, 19 Feb 2021 06:18:09 GMT

Well @ryu, My understanding is when you are storing things on HDFS and that too things related to hive, it is best to use managed table considering in mind that CDP is now coming up with compaction features where in small file issue would automatically get addressed.

Compaction will not happen on external tables.

one would prefer to choose external tables if the data is stored outside HDFS like S3. This is my understanding, but again it could vary on customer to customer based on their use cases.

Re: what is the best practice on where to store hive external tables in HDFS?

Bender — Tue, 23 Mar 2021 09:28:55 GMT

To see more on Hive Managed and External tables, please see our public documentation for CDP Hive.