question Re: what is huge datasets for Hive ? in Archives of Support Questions (Read Only)

what is huge datasets for Hive ?

sebastien_frack — Thu, 02 Nov 2017 22:42:34 GMT

I read lot of articles advising about fastest solutions to compute datasets.

I saw that Hive / TEZ is 100x faster than Hive / MapReduce, but Spark is 100x faster than Hive (TEZ or MR not mentionned ;-)), and finally, "it depends if you compute huge datasets or not".

My first question is : from what size can I consider a "huge" datasets ? I presume the number of rows and columns is significant...

My second question is : what if I am querying few partitions from a large dataset ? I think it comes to querying a small dataset ?

Re: what is huge datasets for Hive ?

sebastien_frack — Thu, 02 Nov 2017 23:04:19 GMT

...and, I always wondered how benchmarks are performed, is it just a timing of an execution on a "clear" plateform ?

Re: what is huge datasets for Hive ?

SQLShaw — Fri, 03 Nov 2017 00:29:28 GMT

Hi @Sebastien F Hive has been documented at running on 300+ PB of raw storage at Facebook. The largest cluster is 4,500+ nodes at Yahoo. Yahoo Japan was able to run 100,000 queries per hour and LLAP ran 100 million rows/s per node.

Hive\Tez scales to 100's of PB. LLAP is meant for smaller data sets (1-10 TB) which are typical for standard BI type workloads. With that being said, LLAP allows you to utilize SSD for cache so you can extend this to 100's TB (if you can afford that much SSD storage).

Hope this helps!

Re: what is huge datasets for Hive ?

sebastien_frack — Fri, 03 Nov 2017 15:03:36 GMT

Hi @Scott Shaw ; it helps 🙂 thanks a lot.