Support Questions
Find answers, ask questions, and share your expertise

RM not killing Spark AM after the job finishes

RM not killing Spark AM after the job finishes

Hi,

Today we have seen few of the Spark jobs are running/hanging. The jobs are already finished, however the spark AMs were not killed by the RM. Could you please help me in checking the issue.

I have checked the AM logs. The last time the log was updated 10 hours back.

I have checked the Yarn RM logs. Here also the last time the log was updated 10 hours back. Since then, there is no progress in the logs. However, the containers/jobs are in RUNNING state in Yarn UI holding some containers impacting other jobs to run

3 REPLIES 3

Re: RM not killing Spark AM after the job finishes

Explorer

@rmr1989 , seems log process is hanging and not writing logs to files. Do you see any alerts in ambari ?

Regards,

Fahim

Re: RM not killing Spark AM after the job finishes

@Mohammedfahim Pathan

Hi,

I don't see any alerts in Ambari. The rest of the jobs are running good. I have killed these jobs and re ran. The jobs went successful now. Don't understand how the jobs were stuck.

Re: RM not killing Spark AM after the job finishes

Explorer

@rmr1989 , Seems logging is not working. Try to see if you can restart services under maintenance. And Check if you open track with Hortonworks support and get them to look into this.

Regards,

Fahim