Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Spark Job not able to shutdown after successfully completing tasks

Highlighted

Spark Job not able to shutdown after successfully completing tasks

New Contributor

Hi All ,

I have a spark job thats isn't able to shutdown after successfully completing its tasks. it just hangs in there for ever with out shutting down. below are that last few lines from client log

17/01/27 15:45:58 INFO Hive: Renaming src: hdfs://apps/.hive-staging_hive_2017-01-27_15-18-18_942_4992404400603125547-1/-ext-10000/datadate=2016-02-24/part-00197, dest: hdfs://apps/datadate=2016-02-24/part-00197, Status:true

17/01/27 15:45:58 INFO Hive: Renaming src: hdfs://apps/.hive-staging_hive_2017-01-27_15-18-18_942_4992404400603125547-1/-ext-10000/datadate=2016-02-24/part-00198, dest: hdfs://apps/datadate=2016-02-24/part-00198, Status:true 17/01/27 15:45:58 INFO Hive: Renaming src: hdfs://apps/.hive-staging_hive_2017-01-27_15-18-18_942_4992404400603125547-1/-ext-10000/datadate=2016-02-24/part-00199, dest: hdfs://apps/datadate=2016-02-24/part-00199, Status:true

17/01/27 15:45:58 INFO Hive: New loading path = hdfs://apps/.hive-staging_hive_2017-01-27_15-18-18_942_4992404400603125547-1/-ext-10000/datadate=2016-02-24 with partSpec {datadate=2016-02-24} 17/01/27 15:45:58

INFO SparkContext: Invoking stop() from shutdown hook

17/01/27 15:45:58 INFO SparkUI: Stopped Spark web UI at http://10.196.195.20:5001

can you please help me why it isnt able to shutdown completely though it has initiated the process?

Regards

Karteek

Don't have an account?
Coming from Hortonworks? Activate your account here