Support Questions
Find answers, ask questions, and share your expertise
Announcements
Check out our newest addition to the community, the Cloudera Innovation Accelerator group hub.

Using RDD in hive

Hi,

Can we use RDD cache in hive?

Like can I create a dataframe which picks data from a hive table. Then create an external hive table in top the data frame which is in the cache? Is it compatible? Does enabling execution engine as 'Spark' in hive will allow me to use RDD cache? My question might be silly but still i wanted to know whether it is really possible as I have less knowledge in spark. If possible throw some light on how I can make use of RDD cache in hive.

1 ACCEPTED SOLUTION

2 REPLIES 2

Explorer

Hi @Bala Vignesh N V Specifying the hive.execution.engine to spark will result in kicking off Spark jobs for the SQL query. But that's not supported by Hortonworks. The better way is to use Spark thrift server plus beeline to run queries:

https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.4.0/bk_installing_manually_book/content/startin...

You can create hive tables, execute a query (by submitting a spark job under the hood) and the query result set is generated based on SparkContext. Is that what you need?