Support Questions

Find answers, ask questions, and share your expertise
Celebrating as our community reaches 100,000 members! Thank you!

Is there a way to define the HDFS maximum file size per table using hive (no. of files will be derived automatically from the definition)?


Super Guru

@Abraham Abraham

Do you mean the maximum size of each file while the table may have multiple files or maximum size of the table?

For maximum file size, you cannot do much except for a block size (each file having multiple blocks). There is no limit to a file size.

However, you can limit the size of data in each directory using HDFS disk quota. So assume you have external table at /user/mytable

Now you can the quota for this directory to be 1TB for example. In this case you will limit the table size to be 1TB. But you can of course have multiple files.