Member since
10-01-2017
4
Posts
0
Kudos Received
1
Solution
My Accepted Solutions
Title | Views | Posted |
---|---|---|
2716 | 10-22-2018 01:41 PM |
10-22-2018
01:41 PM
The issue is that Spark couldn't read hive-site.xml and hence providing the properties in SparkSession builder fixed the issue. spark_session = (SparkSession
.builder
.config("hive.metastore.uris", "thrift://server.country.cloud.company.com:9083")
.config("hive.metastore.client.socket.timeout", "300")
.config("hive.metastore.warehouse.dir", "/user/hive/warehouse")
.config("hive.warehouse.subdir.inherit.perms", "true")
.config("hive.execution.engine", "mr")
.config("hive.metastore.execute.setugi", "true")
.config("hive.support.concurrency", "true")
.config("hive.zookeeper.quorum", "<<REDACTED>>")
.config("hive.zookeeper.client.port", "2181")
.config("hive.zookeeper.namespace", "hive_zookeeper_namespace_hive")
.config("hive.cluster.delegation.token.store.class", "org.apache.hadoop.hive.thrift.MemoryTokenStore")
.config("hive.server2.enable.doAs", "false")
.config("hive.metastore.sasl.enabled", "true")
.config("hive.server2.authentication", "kerberos")
.config("hive.metastore.kerberos.principal", "<<REDACTED>>")
.config("hive.server2.authentication.kerberos.principal", "<<REDACTED>>")
.config("hive.server2.use.SSL", "true")
.config("hive.exec.dynamic.partition", "true")
.config("hive.exec.dynamic.partition.mode", "nonstrict")
.enableHiveSupport()
.appName('app_name')
.getOrCreate())
... View more