Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

How to restart spark streaming

avatar

Hello There,

 

Can anyone please let me know how can i restart spark streaming. 

 

Below code for stop the streaming 

 

ssc.stop(stopSparkContext = false, stopGracefully = true)

 

and again when i try to start it . it is giving me bellow error 

java.lang.IllegalStateException: Adding new inputs, transformations, and output operations after stopping a context is not supported
	at org.apache.spark.streaming.dstream.DStream.validateAtInit(DStream.scala:224) ~[spark-streaming_2.11-1.4.1.jar:1.4.1]
	at org.apache.spark.streaming.dstream.DStream.<init>(DStream.scala:64) ~[spark-streaming_2.11-1.4.1.jar:1.4.1]
	at org.apache.spark.streaming.dstream.InputDStream.<init>(InputDStream.scala:41) ~[spark-streaming_2.11-1.4.1.jar:1.4.1]
	at org.apache.spark.streaming.dstream.ReceiverInputDStream.<init>(ReceiverInputDStream.scala:41) ~[spark-streaming_2.11-1.4.1.jar:1.4.1]
	at org.apache.spark.streaming.twitter.TwitterInputDStream.<init>(TwitterInputDStream.scala:46) ~[spark-streaming-twitter_2.11-1.4.1.jar:1.4.1]
	at org.apache.spark.streaming.twitter.TwitterUtils$.createStream(TwitterUtils.scala:44) ~[spark-streaming-twitter_2.11-1.4.1.jar:1.4.1]
	at org.sach.KibanaElasticSearch.TwitterTransmitter$.Fetchallinfo(TwitterTransmitter.scala:39) ~[classes/:na]
	at org.sach.KibanaElasticSearch.CallServlet.doPost(CallServlet.java:58) ~[classes/:na]
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:648) ~[servlet-api.jar:na]
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:729) ~[servlet-api.jar:na]
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:292) ~[catalina.jar:8.0.37]
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207) ~[catalina.jar:8.0.37]
	at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52) ~[tomcat-websocket.jar:8.0.37]
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:240) ~[catalina.jar:8.0.37]
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207) ~[catalina.jar:8.0.37]
	at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:212) ~[catalina.jar:8.0.37]
	at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:106) [catalina.jar:8.0.37]
	at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:502) [catalina.jar:8.0.37]
	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:141) [catalina.jar:8.0.37]
	at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:79) [catalina.jar:8.0.37]
	at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:616) [catalina.jar:8.0.37]
	at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:88) [catalina.jar:8.0.37]
	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:528) [catalina.jar:8.0.37]
	at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1100) [tomcat-coyote.jar:8.0.37]
	at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:687) [tomcat-coyote.jar:8.0.37]
	at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1520) [tomcat-coyote.jar:8.0.37]
	at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1476) [tomcat-coyote.jar:8.0.37]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_65]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_65]
	at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) [tomcat-util.jar:8.0.37]
	at java.lang.Thread.run(Thread.java:745) [na:1.8.0_65]
1 REPLY 1

avatar
Expert Contributor

You are currently unable to restart a streaming context after it has been stopped.  You can instead create a new streaming context or you can restart the entire application.  You can also enable checkpointing and start the context from the checkpoints to recover from any unclean stops: http://spark.apache.org/docs/latest/streaming-programming-guide.html#checkpointing