Reply
New Contributor
Posts: 3
Registered: ‎09-14-2017

SparkListenerBus has already stopped!

Issue:

My Spark Application to read data from Kafka and write to HDFS is failing intermittently.

The same code worked totally fine earlier and started failing today onwards. Also, it ran successfully once today.

This error even comes when I am trying to just read data from Kafka and count the items read using already existing batch consumer.

 

Error Message:

 

Traceback (most recent call last):
File "/home/bddpetl/Projects/datapipeline-consumer/LDAPConsumer.py", line 352, in <module>
main()
File "/home/bddpetl/Projects/datapipeline-consumer/LDAPConsumer.py", line 287, in main
all_data_viewDF.show()
File "/opt/cloudera/parcels/SPARK2-2.1.0.cloudera1-1.cdh5.7.0.p0.120904/lib/spark2/python/lib/pyspark.zip/pyspark/sql/dataframe.py", line 318, in show
File "/opt/cloudera/parcels/SPARK2-2.1.0.cloudera1-1.cdh5.7.0.p0.120904/lib/spark2/python/lib/py4j-0.10.4-src.zip/py4j/java_gateway.py", line 1131, in __call__
File "/opt/cloudera/parcels/SPARK2-2.1.0.cloudera1-1.cdh5.7.0.p0.120904/lib/spark2/python/lib/py4j-0.10.4-src.zip/py4j/java_gateway.py", line 883, in send_command
File "/opt/cloudera/parcels/SPARK2-2.1.0.cloudera1-1.cdh5.7.0.p0.120904/lib/spark2/python/lib/py4j-0.10.4-src.zip/py4j/java_gateway.py", line 1028, in send_command
File "/opt/cloudera/parcels/Anaconda-4.2.0/lib/python2.7/socket.py", line 451, in readline
data = self._sock.recv(self._rbufsize)
File "/opt/cloudera/parcels/SPARK2-2.1.0.cloudera1-1.cdh5.7.0.p0.120904/lib/spark2/python/lib/pyspark.zip/pyspark/context.py", line 239, in signal_handler
KeyboardInterrupt
18/03/06 23:15:35 ERROR scheduler.LiveListenerBus: SparkListenerBus has already stopped! Dropping event SparkListenerStageCompleted(org.apache.spark.scheduler.StageInfo@e9c3e2e)
18/03/06 23:15:35 ERROR scheduler.LiveListenerBus: SparkListenerBus has already stopped! Dropping event SparkListenerJobEnd(0,1520378135123,JobFailed(org.apache.spark.SparkException: Job 0 cancelled as part of cancellation of all jobs))
18/03/06 23:15:35 ERROR scheduler.LiveListenerBus: SparkListenerBus has already stopped! Dropping event SparkListenerSQLExecutionEnd(0,1520378135125)

Any help will be appreciated.

 

Announcements
New solutions