question Re: Spark job keeps on running even after killing application using yarn kill in Support Questions

Spark job keeps on running even after killing application using yarn kill

ravindranath_or — Wed, 04 Oct 2017 18:34:35 GMT

Submitting a spark job by using shell script as follows:

nohup spark-submit --master yarn-client --py-files libs.zip parser.py --jobname dicomparser >> out.log &

Job started to perform required processing, in between killed using

yarn application -kill <application_id>

and job disappeared from pending list. But could see the process still going on

What am I doing wrong?

Re: Spark job keeps on running even after killing application using yarn kill

smayani — Wed, 04 Oct 2017 20:03:13 GMT

Hi @Ravindranath Oruganti you are running spark driver in yarn-client mode i.e. on machine where you initiated the spark-submit command. You must also kill this process where you initiated this command.

Re: Spark job keeps on running even after killing application using yarn kill

ravindranath_or — Wed, 04 Oct 2017 20:41:51 GMT

Hi @Saumil Mayani, thanks for the reply. Tried that way and killed the process with name as org.apache.spark.deploy.SparkSubmit and yarn application as well. Still process is going on

Re: Spark job keeps on running even after killing application using yarn kill

smayani — Wed, 04 Oct 2017 21:20:55 GMT

Could you share which process is running (driver, AM, executors) and where (local, yarn - nodemanagers server)?

Re: Spark job keeps on running even after killing application using yarn kill

ravindranath_or — Thu, 05 Oct 2017 01:31:11 GMT

I am not sure which process is running but my files(which is part of the job process) are getting processed after killing job

Re: Spark job keeps on running even after killing application using yarn kill

smayani — Thu, 05 Oct 2017 03:21:23 GMT

Hi @Ravindranath Oruganti could you run the the following on all servers and check.

 ps -ef | grep -i parser.py

Re: Spark job keeps on running even after killing application using yarn kill

ravindranath_or — Thu, 05 Oct 2017 12:32:09 GMT

Thanks a lot @Saumil Mayani. Process was running for parser.py. After killing those processes, it got stopped. Thanks again