About akasper

akasper · ‎12-20-2016

Found another way of achieving this which also works for PySpark in an Oozie Spark action. Add this to the <spark-opts> tag in the action definition: --conf spark.yarn.appMasterEnv.hive.metastore.uris=thrift://<your-hive-metastore>:9083 This will add the metastore URI to the application master environment and should allow successful connection to Hive for using tables inside a PySpark script.

akasper · ‎05-17-2016

I found a solution, even though it is not the prettiest. In the Spark job, before creating the SparkContext, you need to set a system property for the Hive metastore URI like so: System.setProperty("hive.metastore.uris", "thrift://<your metastore host>:9083"); I have tried setting this through the Oozie configuration but to no avail. So far, this was the only way to make it work.

akasper · ‎05-13-2016

I'm facing exactly the same issue. Trying to run a Spark job that is using a HiveContext from an Oozie Spark action results in the job failing to connect to the Hive metastore. I also tried adding the hive-site.xml in the various mentioned places to no avail. So where would be the right place to configure the Oozie Spark action to play nicely with Hive?

Online	Offline
Last Visited	‎12-20-2016 09:23 AM

Member Since	‎05-13-2016 05:51 AM
Last Visited	‎12-20-2016 09:23 AM
Posts	3
Kudos received	5

Cloudera Community

Re: Oozie - Spark Action - Hive

Re: Oozie - Spark Action - Hive

Re: Oozie - Spark Action - Hive

Re: Oozie - Spark Action - Hive