Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

How to restart spark streaming

Highlighted

How to restart spark streaming

Hello There,

 

Can anyone please let me know how can i restart spark streaming. 

 

Below code for stop the streaming 

 

ssc.stop(stopSparkContext = false, stopGracefully = true)

 

and again when i try to start it . it is giving me bellow error 

java.lang.IllegalStateException: Adding new inputs, transformations, and output operations after stopping a context is not supported
	at org.apache.spark.streaming.dstream.DStream.validateAtInit(DStream.scala:224) ~[spark-streaming_2.11-1.4.1.jar:1.4.1]
	at org.apache.spark.streaming.dstream.DStream.<init>(DStream.scala:64) ~[spark-streaming_2.11-1.4.1.jar:1.4.1]
	at org.apache.spark.streaming.dstream.InputDStream.<init>(InputDStream.scala:41) ~[spark-streaming_2.11-1.4.1.jar:1.4.1]
	at org.apache.spark.streaming.dstream.ReceiverInputDStream.<init>(ReceiverInputDStream.scala:41) ~[spark-streaming_2.11-1.4.1.jar:1.4.1]
	at org.apache.spark.streaming.twitter.TwitterInputDStream.<init>(TwitterInputDStream.scala:46) ~[spark-streaming-twitter_2.11-1.4.1.jar:1.4.1]
	at org.apache.spark.streaming.twitter.TwitterUtils$.createStream(TwitterUtils.scala:44) ~[spark-streaming-twitter_2.11-1.4.1.jar:1.4.1]
	at org.sach.KibanaElasticSearch.TwitterTransmitter$.Fetchallinfo(TwitterTransmitter.scala:39) ~[classes/:na]
	at org.sach.KibanaElasticSearch.CallServlet.doPost(CallServlet.java:58) ~[classes/:na]
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:648) ~[servlet-api.jar:na]
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:729) ~[servlet-api.jar:na]
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:292) ~[catalina.jar:8.0.37]
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207) ~[catalina.jar:8.0.37]
	at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52) ~[tomcat-websocket.jar:8.0.37]
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:240) ~[catalina.jar:8.0.37]
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207) ~[catalina.jar:8.0.37]
	at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:212) ~[catalina.jar:8.0.37]
	at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:106) [catalina.jar:8.0.37]
	at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:502) [catalina.jar:8.0.37]
	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:141) [catalina.jar:8.0.37]
	at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:79) [catalina.jar:8.0.37]
	at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:616) [catalina.jar:8.0.37]
	at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:88) [catalina.jar:8.0.37]
	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:528) [catalina.jar:8.0.37]
	at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1100) [tomcat-coyote.jar:8.0.37]
	at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:687) [tomcat-coyote.jar:8.0.37]
	at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1520) [tomcat-coyote.jar:8.0.37]
	at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1476) [tomcat-coyote.jar:8.0.37]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_65]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_65]
	at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) [tomcat-util.jar:8.0.37]
	at java.lang.Thread.run(Thread.java:745) [na:1.8.0_65]
1 REPLY 1

Re: How to restart spark streaming

Expert Contributor

You are currently unable to restart a streaming context after it has been stopped.  You can instead create a new streaming context or you can restart the entire application.  You can also enable checkpointing and start the context from the checkpoints to recover from any unclean stops: http://spark.apache.org/docs/latest/streaming-programming-guide.html#checkpointing