Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Using RDD in hive

avatar

Hi,

Can we use RDD cache in hive?

Like can I create a dataframe which picks data from a hive table. Then create an external hive table in top the data frame which is in the cache? Is it compatible? Does enabling execution engine as 'Spark' in hive will allow me to use RDD cache? My question might be silly but still i wanted to know whether it is really possible as I have less knowledge in spark. If possible throw some light on how I can make use of RDD cache in hive.

1 ACCEPTED SOLUTION
2 REPLIES 2

avatar

avatar
Explorer

Hi @Bala Vignesh N V Specifying the hive.execution.engine to spark will result in kicking off Spark jobs for the SQL query. But that's not supported by Hortonworks. The better way is to use Spark thrift server plus beeline to run queries:

https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.4.0/bk_installing_manually_book/content/startin...

You can create hive tables, execute a query (by submitting a spark job under the hood) and the query result set is generated based on SparkContext. Is that what you need?