Number of ORC files effect on namnode?

[#1]

Hi,

For very large datasets in PB range does it help creating large ORC files?

I understand they should be greater than block size.

So lets say I have a block size of 256 mb and am creating 1 GB ORC files for a hive table of total size 3 TB.

So would it help to create bigger file sizes say of 2 GB?

Keep in mind I will be using ORC index to query only 1 file per partition and that data output would be in kb.

Thanks

[#2 · Accepted solution]

As a general rule, you should be creating the largest files you can within a partition.

Check out @David Streever's excellent answer to this question for more details.

[#3]

https://community.hortonworks.com/questions/3037/what-size-of-tables-make-the-best-out-of-orc-forma....

Large ORC files with large stripes should be best performance.

Look at this Yahoo article on Hive and ORC at scale

http://www.slideshare.net/Hadoop_Summit/hive-at-yahoo-letters-from-the-trenches