Spark job keeps on running even after killing application using yarn kill

[#1]

Submitting a spark job by using shell script as follows:

nohup spark-submit --master yarn-client --py-files libs.zip parser.py --jobname dicomparser >> out.log &

Job started to perform required processing, in between killed using

yarn application -kill <application_id>

and job disappeared from pending list. But could see the process still going on

What am I doing wrong?

[#2 · Accepted solution]

Hi @Ravindranath Oruganti you are running spark driver in yarn-client mode i.e. on machine where you initiated the spark-submit command. You must also kill this process where you initiated this command.

[#3]

Hi @Saumil Mayani, thanks for the reply. Tried that way and killed the process with name as org.apache.spark.deploy.SparkSubmit and yarn application as well. Still process is going on

[#4]

Could you share which process is running (driver, AM, executors) and where (local, yarn - nodemanagers server)?

[#5]

I am not sure which process is running but my files(which is part of the job process) are getting processed after killing job

[#6]

Hi @Ravindranath Oruganti could you run the the following on all servers and check.

 ps -ef | grep -i parser.py

[#7]

Thanks a lot @Saumil Mayani. Process was running for parser.py. After killing those processes, it got stopped. Thanks again