question Using RDD in hive in Archives of Support Questions (Read Only)

Using RDD in hive

balavignesh_nag — Sun, 19 Mar 2017 01:57:45 GMT

Hi,

Can we use RDD cache in hive?

Like can I create a dataframe which picks data from a hive table. Then create an external hive table in top the data frame which is in the cache? Is it compatible? Does enabling execution engine as 'Spark' in hive will allow me to use RDD cache? My question might be silly but still i wanted to know whether it is really possible as I have less knowledge in spark. If possible throw some light on how I can make use of RDD cache in hive.

Re: Using RDD in hive

namaheshwari — Sun, 19 Mar 2017 04:05:24 GMT

Below is a good read:

https://hortonworks.com/hadoop-tutorial/using-hive-with-orc-from-apache-spark/

Re: Using RDD in hive

tli — Tue, 28 Mar 2017 05:07:05 GMT

Hi @Bala Vignesh N V Specifying the hive.execution.engine to spark will result in kicking off Spark jobs for the SQL query. But that's not supported by Hortonworks. The better way is to use Spark thrift server plus beeline to run queries:

https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.4.0/bk_installing_manually_book/content/starting_sts.html

You can create hive tables, execute a query (by submitting a spark job under the hood) and the query result set is generated based on SparkContext. Is that what you need?