I am using the HDP 2.6.5 Sandbox on Docker/Windows 10. Using the Hive 2 view, I created a database and table. Let's call them "sampledb" with a table called "sampletable" in HDFS location /sampledb/sampletable/. I stored data in the form of an ORC file in the appropriate directory, and invoked `msck repair sampledb.sampletable`. From Ambari Hive View and Hive View 2.0 I am able to successfully read data from sampletable. In Zeppelin I wrote a Pyspark script that uses the Spark SQL interface to read data from sampletable. It works, no problem. Note, this is Spark 2, not Spark 1.6. We do not want to run Zeppelin scripts in production, so I converted the script to standard Python for use with spark-submit. The Spark initialization code is below: settings = [ ("hive.exec.dynamic.partition", "true"), ("hive.exec.dynamic.partition.mode", "nonstrict"), ("spark.sql.orc.filterPushdown", "true"), ("hive.msck.path.validation", "ignore"), ("spark.sql.caseSensitive", "true"), ("spark.speculation", "false"), ] spark_conf = SparkConf().setAppName("sampleApp").setAll(settings) self.spark = SparkSession.builder. \ enableHiveSupport(). \ config(conf = spark_conf). \ getOrCreate() Python dependencies are zipped into spark.zip and the script is called sample.py. After copying the script to my /home/ dir, I attempt run it as follows: # spark-submit --py-files ./spark.zip sample.py The relevant error output, so far as I can tell, are the following console outputs: SharedState:54 - Warehouse path is 'file:/home/raj_ops/spark-warehouse'. ... HiveClientImpl:54 - Warehouse location for Hive client (version 1.2.2) is file:/home/raj_ops/spark-warehouse ... Failed to get database sampledb, returning NoSuchObjectException u"Table or view not found: `sampledb`.`sampletable` The only thing I had to show for my effort was a new /spark-warehouse sub-directory within /home/raj_ops/. So I tried adding the following line to the Python script: ("spark.sql.warehouse.dir","/apps/hive/warehouse") So now the full configuration is as follows: settings = [ ("hive.exec.dynamic.partition", "true"), ("hive.exec.dynamic.partition.mode", "nonstrict"), ("spark.sql.orc.filterPushdown", "true"), ("hive.msck.path.validation", "ignore"), ("spark.sql.caseSensitive", "true"), ("spark.speculation", "false"), ("spark.sql.warehouse.dir","/apps/hive/warehouse") ] spark_conf = SparkConf().setAppName("sampleApp").setAll(settings) The invocation of spark-submit was the same. The outcome was similarly unsuccessful. What else do I need to do to get the queries from Pyspark scripts to access the Hive database and tables successfully? Thanks in advance! Chris Falter ... View more