Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

SparkListenerBus has already stopped!

Highlighted

SparkListenerBus has already stopped!

New Contributor

Issue:

My Spark Application to read data from Kafka and write to HDFS is failing intermittently.

The same code worked totally fine earlier and started failing today onwards. Also, it ran successfully once today.

This error even comes when I am trying to just read data from Kafka and count the items read using already existing batch consumer.

 

Error Message:

 

Traceback (most recent call last):
File "/home/bddpetl/Projects/datapipeline-consumer/LDAPConsumer.py", line 352, in <module>
main()
File "/home/bddpetl/Projects/datapipeline-consumer/LDAPConsumer.py", line 287, in main
all_data_viewDF.show()
File "/opt/cloudera/parcels/SPARK2-2.1.0.cloudera1-1.cdh5.7.0.p0.120904/lib/spark2/python/lib/pyspark.zip/pyspark/sql/dataframe.py", line 318, in show
File "/opt/cloudera/parcels/SPARK2-2.1.0.cloudera1-1.cdh5.7.0.p0.120904/lib/spark2/python/lib/py4j-0.10.4-src.zip/py4j/java_gateway.py", line 1131, in __call__
File "/opt/cloudera/parcels/SPARK2-2.1.0.cloudera1-1.cdh5.7.0.p0.120904/lib/spark2/python/lib/py4j-0.10.4-src.zip/py4j/java_gateway.py", line 883, in send_command
File "/opt/cloudera/parcels/SPARK2-2.1.0.cloudera1-1.cdh5.7.0.p0.120904/lib/spark2/python/lib/py4j-0.10.4-src.zip/py4j/java_gateway.py", line 1028, in send_command
File "/opt/cloudera/parcels/Anaconda-4.2.0/lib/python2.7/socket.py", line 451, in readline
data = self._sock.recv(self._rbufsize)
File "/opt/cloudera/parcels/SPARK2-2.1.0.cloudera1-1.cdh5.7.0.p0.120904/lib/spark2/python/lib/pyspark.zip/pyspark/context.py", line 239, in signal_handler
KeyboardInterrupt
18/03/06 23:15:35 ERROR scheduler.LiveListenerBus: SparkListenerBus has already stopped! Dropping event SparkListenerStageCompleted(org.apache.spark.scheduler.StageInfo@e9c3e2e)
18/03/06 23:15:35 ERROR scheduler.LiveListenerBus: SparkListenerBus has already stopped! Dropping event SparkListenerJobEnd(0,1520378135123,JobFailed(org.apache.spark.SparkException: Job 0 cancelled as part of cancellation of all jobs))
18/03/06 23:15:35 ERROR scheduler.LiveListenerBus: SparkListenerBus has already stopped! Dropping event SparkListenerSQLExecutionEnd(0,1520378135125)

Any help will be appreciated.