https://community.cloudera.com/t5/Support-Questions/HDFS-caching-to-hive-performance/m-p/150522#M113024 <P>HDFS caching helps, however it helps only a bit since you are saving only the cost of moving bytes off disk and are still paying the cost of de-serialization, don't get JVM JIT etc. So, with technologies like Hive LLAP (coming in hive-2) you will get significantly better performance because LLAP caches de-serialized vectors in memory-efficient formats (2 bits for certain integer ranges - rather than 4 bytes), cpu-efficient filters (vectorized query processing via filters etc.) removes JVM startup cost for tasks (100s of ms), provides JIT-enhanced CPU performance etc. Rather excited about it!</P> Sat, 02 Apr 2016 05:27:36 GMT acm 2016-04-02T05:27:36Z https://community.cloudera.com/t5/Support-Questions/HDFS-caching-to-hive-performance/m-p/150519#M113021 <P>One of the customer is exploring ways to improve their hive query performance - and they are wondering about HDFS caching. They wanted to check if this is something we recommend.</P> Thu, 31 Mar 2016 08:45:55 GMT https://community.cloudera.com/t5/Support-Questions/HDFS-caching-to-hive-performance/m-p/150519#M113021 hduraiswamy 2016-03-31T08:45:55Z https://community.cloudera.com/t5/Support-Questions/HDFS-caching-to-hive-performance/m-p/150520#M113022 <P>HDFS memory as a storage is in technical preview (link <A href="https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.0/bk_hdfs_admin_tools/content/ch08.html">here</A>). I recommend review hive performance tuning <A href="http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.4.0/bk_performance_tuning/content/ch_hive_hi_perf_best_practices.html">here</A>.</P> Thu, 31 Mar 2016 08:50:32 GMT https://community.cloudera.com/t5/Support-Questions/HDFS-caching-to-hive-performance/m-p/150520#M113022 sunile_manjee 2016-03-31T08:50:32Z https://community.cloudera.com/t5/Support-Questions/HDFS-caching-to-hive-performance/m-p/150521#M113023 <P>Thanks.. exactly what I was looking for.</P> Thu, 31 Mar 2016 09:15:36 GMT https://community.cloudera.com/t5/Support-Questions/HDFS-caching-to-hive-performance/m-p/150521#M113023 hduraiswamy 2016-03-31T09:15:36Z https://community.cloudera.com/t5/Support-Questions/HDFS-caching-to-hive-performance/m-p/150522#M113024 <P>HDFS caching helps, however it helps only a bit since you are saving only the cost of moving bytes off disk and are still paying the cost of de-serialization, don't get JVM JIT etc. So, with technologies like Hive LLAP (coming in hive-2) you will get significantly better performance because LLAP caches de-serialized vectors in memory-efficient formats (2 bits for certain integer ranges - rather than 4 bytes), cpu-efficient filters (vectorized query processing via filters etc.) removes JVM startup cost for tasks (100s of ms), provides JIT-enhanced CPU performance etc. Rather excited about it!</P> Sat, 02 Apr 2016 05:27:36 GMT https://community.cloudera.com/t5/Support-Questions/HDFS-caching-to-hive-performance/m-p/150522#M113024 acm 2016-04-02T05:27:36Z https://community.cloudera.com/t5/Support-Questions/HDFS-caching-to-hive-performance/m-p/150523#M113025 <P>Thank you! I am excited about LLAP too. Do we yet have a timeline on which HDP release the new hive-2 will be packaged in?</P> Sun, 03 Apr 2016 06:52:03 GMT https://community.cloudera.com/t5/Support-Questions/HDFS-caching-to-hive-performance/m-p/150523#M113025 hduraiswamy 2016-04-03T06:52:03Z https://community.cloudera.com/t5/Support-Questions/HDFS-caching-to-hive-performance/m-p/150524#M113026 <P>Watch the Tim Hall townhall from this Friday.</P> Sun, 03 Apr 2016 06:57:36 GMT https://community.cloudera.com/t5/Support-Questions/HDFS-caching-to-hive-performance/m-p/150524#M113026 aervits 2016-04-03T06:57:36Z