Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Why is PySpark not recognising new changes to hive-site.xml made through Ambari?

Why is PySpark not recognising new changes to hive-site.xml made through Ambari?

Contributor

Hi,

When I run a pyspark command to access a hive table I have to explicitly set my hive configurations first (e.g. mapreduce.input.fileinputformat.input.dir.recursive=true) otherwise the command fails. But I have already set these in hive-site through ambari.

Its as if spark is reading an old version ?

Any ideas?

Mike

1 REPLY 1
Highlighted

Re: Why is PySpark not recognising new changes to hive-site.xml made through Ambari?

Contributor

In /etc/spark/conf/hive-site.xml it only specifies the following:

<property><name>hive.metastore.uris</name><value>thrift://<domainname>:9083</value></property<property>

Yet in the hive-site.xml configured through the hive ambari UI I have many other properties defined. Why does ambari not reflect the hive-site.xml configuration settings in /etc/spark/hive-site.xml? so that PySpark can set these when creating the sqlContext (aka hiveContext).

Thanks,

Mike

Don't have an account?
Coming from Hortonworks? Activate your account here