Knowing size of Hive tables

ManuN — Thu, 13 Aug 2020 05:58:06 GMT

Hello all,

I want to see the size of each table in Hive residing in multiple databases. There are around 3000 tables, so it is difficult to do it one by one for each table. How can I do it at one go?

Regards,

Manu

Re: Knowing size of Hive tables

stevenmatison — Thu, 13 Aug 2020 12:11:04 GMT

@ManuN Anyway you go about this task, you are going to have to execute the commands against the tables to get sizes. With a large number of tables this should be a script, program, or process.

The common methods are to query the table with hive:

-- gives all properties
show tblproperties yourTableName

-- show just the raw data size
show tblproperties yourTableName("rawDataSize")

Or the most accurate is to look at the table location in HDFS:

hdfs dfs -du -s -h /path/to/table

There are also some methods to try and get this data directly from the Hive Metastore, assuming the table is an internal Hive table.

In the past I have completed this with a basic bash/shell script. I have also done similar in NiFI and prefer to do it like this without coding.

If this answer resolves your issue or allows you to move forward, please choose to ACCEPT this solution and close this topic. If you have further dialogue on this topic please comment here or feel free to private message me. If you have new questions related to your Use Case please create separate topic and feel free to tag me in your post.

Thanks,

Steven @ DFHZ

question Knowing size of Hive tables in Support Questions

Knowing size of Hive tables

Re: Knowing size of Hive tables