question Spark shows all jobs completed. iPython still wait in Support Questions

question Spark shows all jobs completed. iPython still wait in Support Questions https://community.cloudera.com/t5/Support-Questions/Spark-shows-all-jobs-completed-iPython-still-wait/m-p/51890#M23514 Hello, I am running Ipython -> livy to send jobs to my CDH 5.9.0 cluster running Spark. My job runs through a few operations reading files from HDFS into  dataframes and then doing some operations on those dataframes. The code then reaches a cell with a join and stops progressing. If I leave it along for long enough, the session is eventually killed.  I am not sure how to debug this. Yarn shows the job as still running. Spark shows all jobs completed and no active of pending jobs.  All the Spark jobs say that they succeeded though some were skipped. If I go to the details for the last stage, all statuses say "Success." The logs for the executors all say Finished task ###. #### bytes sent to driver. The thread dump for the driver shows a lot of waiting threads. If I run the job via pyspark, not through Ipython/Livy, it works fine.  But there are no errors in the livy log either. I'm not sure how to figure this out. Any thoughts?Thanks!     Fri, 16 Sep 2022 11:12:45 GMT wkupersa 2022-09-16T11:12:45Z