https://community.cloudera.com/t5/Archives-of-Support-Questions/HIve-on-Tez-or-HIve-query-using-Spark-SQL/m-p/133997#M39603 <P>@<A href="https://community.hortonworks.com/users/11008/chandramoulimuthukumaran.html">chandramouli muthukumaran</A></P><P>Just to clarify, SparlSQL does not access or use Hive engine. It just consumes the metadata of Hive data structures.</P><P> Assuming that both can execute the query functionally (SparkSQL is quite limited functionally compared with Hive), but the query will need to churn through 40 TB of data, then I would say likely Hive on Tez is your optimal choice. That is also driven by the cost associated with your Spark cluster RAM additional to Hive's requirements because I assume that you will still have some cases where running Hive is needed. I noticed that if the amount of data is less than 1 TB, SparkSQL outperforms Hive on Tez. </P><P>Anyhow, be aware, that with HDP 2.5 LLAP is in Tech Preview and soon will be GA. If you were asking Hive on LLAP vs. SparkSQL, I would say without hesitation for most of the queries, Hive on LLAP. Again, for some sofisticated queries with limited amount of data, and limited function, SparkSQL may be a winner, but in the big picture is too expensive to maintain both approaches and I would still consider Hive on Tez and LLAP over SparkSQL for most of the cases that deal with BIG DATA. Otherwise, 1 TB does not need Hadoop for fast queries.</P><P>Read more about Hive on LLAP here:</P><P><A href="http://hortonworks.com/blog/llap-enables-sub-second-sql-hadoop/" target="_blank">http://hortonworks.com/blog/llap-enables-sub-second-sql-hadoop/</A></P><P>Give LLAP a shot before deciding to use SparkSQL, especially, if you already have the queries written in HiveQL.</P><P>If this response or any response in this thread was helpful, please don't forget to vote/accept it as the best answer.</P> Sat, 03 Sep 2016 03:07:34 GMT cstanca 2016-09-03T03:07:34Z https://community.cloudera.com/t5/Archives-of-Support-Questions/HIve-on-Tez-or-HIve-query-using-Spark-SQL/m-p/133996#M39602 <P>Hi,</P><P>Can you please let me know which one is faster -Hive on Tez or accessing Hive using Spark SQL. </P><P>Thanks,</P><P>Chandra</P> Sat, 03 Sep 2016 02:44:01 GMT https://community.cloudera.com/t5/Archives-of-Support-Questions/HIve-on-Tez-or-HIve-query-using-Spark-SQL/m-p/133996#M39602 Chandra 2016-09-03T02:44:01Z https://community.cloudera.com/t5/Archives-of-Support-Questions/HIve-on-Tez-or-HIve-query-using-Spark-SQL/m-p/133997#M39603 <P>@<A href="https://community.hortonworks.com/users/11008/chandramoulimuthukumaran.html">chandramouli muthukumaran</A></P><P>Just to clarify, SparlSQL does not access or use Hive engine. It just consumes the metadata of Hive data structures.</P><P> Assuming that both can execute the query functionally (SparkSQL is quite limited functionally compared with Hive), but the query will need to churn through 40 TB of data, then I would say likely Hive on Tez is your optimal choice. That is also driven by the cost associated with your Spark cluster RAM additional to Hive's requirements because I assume that you will still have some cases where running Hive is needed. I noticed that if the amount of data is less than 1 TB, SparkSQL outperforms Hive on Tez. </P><P>Anyhow, be aware, that with HDP 2.5 LLAP is in Tech Preview and soon will be GA. If you were asking Hive on LLAP vs. SparkSQL, I would say without hesitation for most of the queries, Hive on LLAP. Again, for some sofisticated queries with limited amount of data, and limited function, SparkSQL may be a winner, but in the big picture is too expensive to maintain both approaches and I would still consider Hive on Tez and LLAP over SparkSQL for most of the cases that deal with BIG DATA. Otherwise, 1 TB does not need Hadoop for fast queries.</P><P>Read more about Hive on LLAP here:</P><P><A href="http://hortonworks.com/blog/llap-enables-sub-second-sql-hadoop/" target="_blank">http://hortonworks.com/blog/llap-enables-sub-second-sql-hadoop/</A></P><P>Give LLAP a shot before deciding to use SparkSQL, especially, if you already have the queries written in HiveQL.</P><P>If this response or any response in this thread was helpful, please don't forget to vote/accept it as the best answer.</P> Sat, 03 Sep 2016 03:07:34 GMT https://community.cloudera.com/t5/Archives-of-Support-Questions/HIve-on-Tez-or-HIve-query-using-Spark-SQL/m-p/133997#M39603 cstanca 2016-09-03T03:07:34Z https://community.cloudera.com/t5/Archives-of-Support-Questions/HIve-on-Tez-or-HIve-query-using-Spark-SQL/m-p/133998#M39604 <P>Thanks for your valuable information. So your recommendation is to go for Hive on LLAP rather than SparkSQL. Please correct me if I am wrong. </P> Sat, 03 Sep 2016 03:16:55 GMT https://community.cloudera.com/t5/Archives-of-Support-Questions/HIve-on-Tez-or-HIve-query-using-Spark-SQL/m-p/133998#M39604 Chandra 2016-09-03T03:16:55Z https://community.cloudera.com/t5/Archives-of-Support-Questions/HIve-on-Tez-or-HIve-query-using-Spark-SQL/m-p/133999#M39605 <P>Also what is the need to run Hive queries on SparkSql when Hive on Tez can run much faster....</P> Sat, 03 Sep 2016 04:19:45 GMT https://community.cloudera.com/t5/Archives-of-Support-Questions/HIve-on-Tez-or-HIve-query-using-Spark-SQL/m-p/133999#M39605 Chandra 2016-09-03T04:19:45Z