I'm having the exact same issue. All of a sudden yesterday - on a cluster that has been up and running for weeks - started spawning six of these at a time for no apparent reason. I kill them and they come back. I've poured over every single log, checked every nook and cranny and cannot figure it out. I have no idea where they are coming from. It is most definitely not a resource issue - these jobs shouldn't even be running - and it's not cron. They are sucking up major CPU when it runs. If anyone has any thoughts I'd be grateful to hear them! The other odd thing is that in the past I would see one of these jobs - but only one - never like this. , I'm having the exact same problem. All of a sudden on a cluster that has not changed is spawning off these jobs that are in ACCEPTED status as user Dr.Who and called MYYARN. I've poured over every single log, bounced my cluster several times, there are no cron jobs and it is most definitely not a resource issue. Looking at old logs it looks like it happened periodically - but only once of twice and then it stops. Yesterday it started running wild and as quick as I kill them off it starts another 6 of the exact same job. If anyone has any insight at all I'd be grateful. And I'm not even using HDP - this is standard Apache Hadoop/Yarn/Spark 2.7.5
... View more
