question Re: Spark shows all jobs completed. iPython still wait in Support Questions

question Re: Spark shows all jobs completed. iPython still wait in Support Questions https://community.cloudera.com/t5/Support-Questions/Spark-shows-all-jobs-completed-iPython-still-wait/m-p/51935#M23515 <P>A bit more info.... (and this is cross-posted in project jupyter list)</P><P> </P><DIV>I think that messaging is getting screwed up between Pyspark and Livy. When the last cell is executed, I will see this on the client side. </DIV><DIV> </DIV><DIV><DIV> </DIV><DIV>2017-03-08 22:24:48,505 INFO    EventsHandler   InstanceId: 0e1c8fd2-047e-4337-b264-5b64ba74de5a,EventName: notebookStatementExecutionStart,Timestamp: 2017-03-08 22:24:48.504920,SessionGuid: 03d14478-6adc-4b</DIV><DIV>34-abef-b9b6fd400543,LivyKind: pyspark,SessionId: 8,StatementGuid: f1933b11-b767-4a18-b311-c48901ad8369</DIV><DIV>2017-03-08 22:24:48,788 DEBUG   Command Status of statement 8 is running.</DIV><DIV>2017-03-08 22:24:50,920 DEBUG   Command Status of statement 8 is running.</DIV></DIV><DIV> </DIV><DIV> ...and it never comes back.</DIV><DIV> </DIV><DIV>On the livy end, I see </DIV><DIV> </DIV><DIV><DIV>17/03/08 17:26:26 INFO ContextLauncher: 17/03/08 17:26:26 INFO scheduler.DAGScheduler: ResultStage 17 (collect at <stdin>:5) finished in 1.521 s</DIV><DIV>17/03/08 17:26:26 INFO ContextLauncher: 17/03/08 17:26:26 INFO scheduler.DAGScheduler: Job 8 finished: collect at <stdin>:5, took 3.729078 s</DIV><DIV>17/03/08 17:26:27 DEBUG RpcDispatcher: [ClientProtocol] Registered outstanding rpc 230 (com.cloudera.livy.rsc.BaseProtocol$GetReplJobResult).</DIV><DIV>17/03/08 17:26:27 DEBUG KryoMessageCodec: Encoded message of type com.cloudera.livy.rsc.rpc.Rpc$MessageHeader (6 bytes)</DIV><DIV>17/03/08 17:26:27 DEBUG KryoMessageCodec: Encoded message of type com.cloudera.livy.rsc.BaseProtocol$GetReplJobResult (91 bytes)</DIV><DIV>17/03/08 17:26:27 DEBUG KryoMessageCodec: Decoded message of type com.cloudera.livy.rsc.rpc.Rpc$MessageHeader (6 bytes)</DIV><DIV>17/03/08 17:26:27 DEBUG KryoMessageCodec: Decoded message of type com.cloudera.livy.rsc.rpc.Rpc$NullMessage (2 bytes)</DIV><DIV>17/03/08 17:26:27 DEBUG RpcDispatcher: [ClientProtocol] Received RPC message: type=REPLY id=230 payload=com.cloudera.livy.rsc.rpc.Rpc$NullMessage</DIV><DIV>17/03/08 17:26:28 DEBUG RpcDispatcher: [ClientProtocol] Registered outstanding rpc 231 (com.cloudera.livy.rsc.BaseProtocol$GetReplJobResult).</DIV></DIV><DIV> </DIV><DIV>ad infinitum</DIV><DIV> </DIV><DIV>So, with my limited knowledge, it looks to me that Livy thinks it has sent a result to a finished job, but pyspark hasn't received it.</DIV><DIV>Anyone seen this before? Any thoughts?</DIV> Wed, 08 Mar 2017 22:55:18 GMT wkupersa 2017-03-08T22:55:18Z