No idea why but livy stopped working today.
It spits the following out non stop. No idea why anyone got an idea of where to look.
org.apache.zeppelin.livy.LivyException: Session 349 is finished, appId: application_1522011783190_0196, log: [ at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1866), at org.apache.spark.deploy.SparkHadoopUtil.runAsSparkUser(SparkHadoopUtil.scala:66), at org.apache.spark.deploy.yarn.ApplicationMaster$.main(ApplicationMaster.scala:766), at org.apache.spark.deploy.yarn.ApplicationMaster.main(ApplicationMaster.scala), 18/03/29 17:06:54 INFO ApplicationMaster: Final app status: FAILED, exitCode: 10, (reason: Uncaught exception: java.lang.ClassNotFoundException: org.apache.livy.rsc.driver.RSCDriverBootstrapper), 18/03/29 17:06:54 INFO ApplicationMaster: Unregistering ApplicationMaster with FAILED (diag message: Uncaught exception: java.lang.ClassNotFoundException: org.apache.livy.rsc.driver.RSCDriverBootstrapper), 18/03/29 17:06:54 INFO ApplicationMaster: Deleting staging directory hdfs://lrdccdhm01.cloud.res.bngf.local:8020/user/andm013/.sparkStaging/application_1522011783190_0196, 18/03/29 17:06:55 INFO ShutdownHookManager: Shutdown hook called, , Failing this attempt. Failing the application.]
Livy logs:
18/03/29 17:07:35 INFO RSCClient: Failing pending job 94534a77-1855-45c9-bc18-8e53c8e42165 due to shutdown.
18/03/29 17:07:35 INFO InteractiveSession: Failed to ping RSC driver for session 348. Killing application.
18/03/29 17:07:35 INFO InteractiveSession: Stopping InteractiveSession 348...
18/03/29 17:07:35 WARN InteractiveSession: (Fail to get rsc uri,java.util.concurrent.ExecutionException: java.util.concurrent.TimeoutException: Timed out waiting for context to start.)
18/03/29 17:07:38 INFO InteractiveSession: Stopped InteractiveSession 348.
18/03/29 17:08:09 ERROR RSCCli
ent: Failed to connect to context.
java.util.concurrent.TimeoutException: Timed out waiting for context to start.
at org.apache.livy.rsc.ContextLauncher.connectTimeout(ContextLauncher.java:134)
at org.apache.livy.rsc.ContextLauncher.access$300(ContextLauncher.java:63)
at org.apache.livy.rsc.ContextLauncher$2.run(ContextLauncher.java:122)
at io.netty.util.concurrent.PromiseTask$RunnableAdapter.call(PromiseTask.java:38)
at io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:120)
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:357)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:357)
at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
at java.lang.Thread.run(Thread.java:745)
18/03/29 17:08:09 INFO RSCClient: Failing pending job 04982890-5fc8-4015-98a9-ad888b4e6b45 due to shutdown.
18/03/29 17:08:09 INFO InteractiveSession: Failed to ping RSC driver for session 349. Killing application.
18/03/29 17:08:09 INFO InteractiveSession: Stopping InteractiveSession 349...
18/03/29 17:08:09 WARN InteractiveSession: (Fail to get rsc uri,java.util.concurrent.ExecutionException: java.util.concurrent.TimeoutException: Timed out waiting for context to start.)
18/03/29 17:08:09 INFO InteractiveSession: Stopped InteractiveSession 349.
