Hi Guys I have a HDP 2.4 cluster. We are working with spark and we use spark-submmit with --deploy-mode client and the cluster run well. We try to use the --deploy-mode cluster and we cannot execute Hive DDL as we made use client deploy mode. I made a simple test case in order you could reproduce the problem. In hive shell I create the database and a partitioned table CREATE DATABASE TEST
CREATE TABLE test.testcase (
FIELD STRING
) partitioned by (p_kpart date)
STORED AS ORC
LOCATION "hdfs://rsicluster01/tmp/delete/test.db/testcase" We create a python file named testcase.py #!/usr/bin/env Python
# -*- coding: utf-8 -*-
from pyspark import SparkContext, SparkConf
from pyspark.sql import HiveContext
conf = SparkConf()
sc = SparkContext(conf=conf)
sqlContext = HiveContext(sc)
ori_query_p = "ALTER TABLE TEST.TESTCASE ADD IF NOT EXISTS PARTITION (p_kpart= '{0}')".format("2017-10-30")
print ("ori_query_p:", ori_query_p)
addpart = sqlContext.sql(ori_query_p)
print ("Alter OK")
We run this code use spark-submit --deploy-mode cluster --master yarn testcase.py Generated the following log, I cut a lot of lines ('ori_query_p:', "ALTER TABLE TEST.TESTCASE ADD IF NOT EXISTS PARTITION (p_kpart= '2017-10-30')")
Traceback (most recent call last):
File "testcase.py", line 12, in <module>
addpart = sqlContext.sql(ori_query_p)
File "/tmp/hadoop/yarn/local/usercache/hdfs/appcache/application_1509376396249_0059/container_e09_1509376396249_0059_02_000001/pyspark.zip/pyspark/sql/context.py", line 583, in sql
File "/tmp/hadoop/yarn/local/usercache/hdfs/appcache/application_1509376396249_0059/container_e09_1509376396249_0059_02_000001/pyspark.zip/pyspark/sql/context.py", line 691, in _ssql_ctx
Exception: ("You must build Spark with Hive. Export 'SPARK_HIVE=true' and run build/sbt assembly", Py4JJavaError(u'An error occurred while calling None.org.apache.spark.sql.hive.HiveContext.\n', JavaObject id=o50))
How I could config the HDP 2.4 Cluster in order that I could use the --deploy-mode cluster for Hive DDL
... View more
