Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Spark2 Exception after upgrade to CDH 5.14.2 "when registering SparkListener"

Spark2 Exception after upgrade to CDH 5.14.2 "when registering SparkListener"

New Contributor

Hi All,

 

Earlier I upgraded my development cluster from 5.13.0 to 5.14.2

 

All is working now apart from Spark2

 

When we try to run spark2-shell the following error occurs:

 

ERROR spark.SparkContext: Error initializing SparkContext.
org.apache.spark.SparkException: Exception when registering SparkListener
at org.apache.spark.SparkContext.setupAndStartListenerBus(SparkContext.scala:2364)
at org.apache.spark.SparkContext.<init>(SparkContext.scala:553)
at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2486)
at org.apache.spark.sql.SparkSession$Builder$$anonfun$7.apply(SparkSession.scala:930)
at org.apache.spark.sql.SparkSession$Builder$$anonfun$7.apply(SparkSession.scala:921)
at scala.Option.getOrElse(Option.scala:121)
at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:921)
at org.apache.spark.repl.Main$.createSparkSession(Main.scala:103)
[...]
Caused by: java.io.FileNotFoundException: Lineage directory /var/log/spark2/lineage doesn't exist or is not writable.
at com.cloudera.spark.lineage.LineageWriter$.checkLineageConfig(LineageWriter.scala:158)
at com.cloudera.spark.lineage.NavigatorAppListener.<init>(ClouderaNavigatorListener.scala:30)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
[...]

 

 

Anyone able to help?

Thanks

5 REPLIES 5
Highlighted

Re: Spark2 Exception after upgrade to CDH 5.14.2 "when registering SparkListener"

Rising Star

Hi,

 

A bit trivial.. but

1/ Do you have the folder /var/log/spark2/lineage present on the Gateway instance of Spark2?

2/ is spark:spark listed as owner of said folder if it exist?

 

I have a similar issue (CDH 5.14.1 / Spark 2.3 - Issue appaered after Spark 2.3 got enabled) though Yarn  & workbench for which a case is open

In the meantime, I disabled Navigator Lineage from cloudera Manager (spark2 Configuration / config.navigator.lineage_enabled) which allowed my colleagues to work.

 

Re: Spark2 Exception after upgrade to CDH 5.14.2 "when registering SparkListener"

New Contributor

@ChrisV,

Where exactly did you disable Navigator Lineage in Cloudera Manager, I can't find that property under the configs tab in Spark. I'm using Cloudera express version 5.14.1 with a custom Spark install (Version-2.3).

Best,

Hobie

Re: Spark2 Exception after upgrade to CDH 5.14.2 "when registering SparkListener"

New Contributor
Lineage directory /var/log/spark2/lineage doesn't exist or is not writable.

Hi, 

As it says you can fix that by creating the formentioned folder and then doing 

 

# chown spark:spark /var/log/spark2/lineage

Re: Spark2 Exception after upgrade to CDH 5.14.2 "when registering SparkListener"

New Contributor
I was getting the same error. I solved it by creating the directory on all the machines. (In my case a 3 nodes cluster). So, please make sure that you have "/var/log/spark2/lineage" directory present on all the machines.

Also, the ownership should be as follows:
chown spark:spark /var/log/spark2/lineage

Re: Spark2 Exception after upgrade to CDH 5.14.2 "when registering SparkListener"

Explorer

 

Even easier, disable the listeners which generate the lineage files, by setting the following spark properties:

 

--conf spark.sql.queryExecutionListeners=""
--conf spark.extraListeners=""