Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Spark job stalling indefinitely

Highlighted

Spark job stalling indefinitely

New Contributor

Hi,

I am running a spark job to compress a file, most of the times its successful, but every once in a while this issue occurs and sometimes its stuck indefinitely. Since the job is still in running phase , we don't find any error in the yarn logs. It gets stuck right after the driver requests to kill the executor. In this particular job it got stuck in this phase for about 6 hours although it easily completed for data much smaller in size.

Also, dynamic allocation is enabled in our cluster.

19/01/28 09:19:40 INFO YarnAllocator: Driver requested a total number of 1 executor(s).
19/01/28 09:19:40 INFO ApplicationMaster$AMEndpoint: Driver requested to kill executor(s) 2, 1.
19/01/28 09:19:41 INFO YarnAllocator: Driver requested a total number of 0 executor(s).
19/01/28 09:19:41 INFO ApplicationMaster$AMEndpoint: Driver requested to kill executor(s) 3.
19/01/28 15:14:39 INFO ApplicationMaster$AMEndpoint: Driver terminated or disconnected! Shutting down. 
19/01/28 15:14:39 INFO ApplicationMaster$AMEndpoint: Driver terminated or disconnected! Shutting down.