Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Yarn-error; custom install of Spark 2.0 on HDP 2.4

Highlighted

Yarn-error; custom install of Spark 2.0 on HDP 2.4

New Contributor

Hi,

I am trying to manually add Spark 2.0.0 to an existing HDP 2.4 installation (on HDInsight 3.4). The necessary environment variables (such as HADOOP_HOME) are set correctly. The spark-defaults.conf file is identical to the existing Spark 1.6 installation, except for the parameter spark.yarn.jars which I updated. I was able to fix some Azure-specific dependencies (i.e. using hadoop-azure-2.7.2.jar). However, when I run pyspark I run into the following Yarn-error:

---------

16/11/14 12:33:18 ERROR SparkContext: Error initializing SparkContext.

org.apache.spark.SparkException: Yarn application has already ended! It might have been killed or unable to launch application master.

at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.waitForApplication(YarnClientSchedulerBackend.scala:85)

[...]

--------

Does anyone have experience with this issue / any tips or pointers? Thanks!

- Michel

2 REPLIES 2

Re: Yarn-error; custom install of Spark 2.0 on HDP 2.4

seems it is falling back to spark 1.x as it is trying to initialize sparkcontext not sparksession.

Re: Yarn-error; custom install of Spark 2.0 on HDP 2.4

New Contributor

Thanks for you answer. However, actually the error message originates from the python/pyspark/shell.py script included in Spark 2.0, which is trying to create a SparkSession:

SparkContext._ensure_initialized()

try:

SparkContext._jvm.org.apache.hadoop.hive.conf.HiveConf()

spark = SparkSession.builder\

.enableHiveSupport()\

.getOrCreate()

except py4j.protocol.Py4JError:

[...]

Don't have an account?
Coming from Hortonworks? Activate your account here