question Re: Why are there dr.who "MYYARN" applications running and all failing in what seems to be a loop? in Support Questions

Why are there dr.who "MYYARN" applications running and all failing in what seems to be a loop?

david_lopez_pen — Wed, 02 May 2018 16:19:19 GMT

We're running a HDP 2.5 cluster and today we noticed a series of dr.who "MYYARN" applications running, failing, and then resubmitting to YARN again and again. In what seems to be an "infinite loop". We can't figure out what the applications are doing and why they are failing. Any thoughts? Many thanks in advance!

Re: Why are there dr.who "MYYARN" applications running and all failing in what seems to be a loop?

cdever — Wed, 02 May 2018 16:48:39 GMT

I'm hitting exactly the same issue here with HDP 2.6.

It looks like some kind of DOS attack but I have no clue on how to handle this?

Any help from Hortonworks would be appreciated.

Re: Why are there dr.who "MYYARN" applications running and all failing in what seems to be a loop?

harshaldj — Wed, 02 May 2018 17:05:03 GMT

I am also facing issue with dr.who user with MYYARN application submitting in loop. But those are staying in "ACCEPTED" status. Total 18 application are launched. no clue !!

Re: Why are there dr.who "MYYARN" applications running and all failing in what seems to be a loop?

cdever — Wed, 02 May 2018 17:51:05 GMT

Mine are also staying in "ACCEPTED" status. And they're launched every 3 seconds... It's becoming a problem on the ResourceManager. When I look at the logs, I can only see actions coming from within my cluster.

Re: Why are there dr.who "MYYARN" applications running and all failing in what seems to be a loop?

arald — Wed, 02 May 2018 18:06:14 GMT

did you check this:
https://community.hortonworks.com/questions/2349/tip-when-you-get-a-message-in-job-log-user-dr-who.html

The solution there was:
RESOLUTION Customer changed the following property in core-site.xml to resolve the issue. Other values such as hdfs or mapred also resolve the issue. If the cluster is managed by Ambari, this should be added in Ambari > HDFS > Configurations>Advanced core-site > Add Property

hadoop.http.staticuser.user=yarn

Re: Why are there dr.who "MYYARN" applications running and all failing in what seems to be a loop?

harshaldj — Wed, 02 May 2018 18:52:12 GMT

Yeah I was thinking to change static user but this was not there until today afternoon. Its suddenly started spawning applications using dr.user.

Re: Why are there dr.who "MYYARN" applications running and all failing in what seems to be a loop?

cdever — Wed, 02 May 2018 18:55:55 GMT

I did the change but it didn't change anything. Instead of being 'dr.who', it's now 'yarn' user that is feeding applications every 3 seconds that get stuck as "ACCEPTED". I still can't find how these applications are being triggered. Any other clue?

Re: Why are there dr.who "MYYARN" applications running and all failing in what seems to be a loop?

harshaldj — Wed, 02 May 2018 19:13:42 GMT

can you check if any cronjob triggering this ? crontab -l !

Re: Why are there dr.who "MYYARN" applications running and all failing in what seems to be a loop?

david_lopez_pen — Wed, 02 May 2018 19:16:13 GMT

Our jobs are indeed stuck in "ACCEPTED" status, and then eventually fail due to a time-out. I can't get any further useful log information. Having checked the RM UI logs for "FAILED" jobs, I noticed it started on April 30 for 3 hours straight then stoped. It started again on May 1 up until today.

Re: Why are there dr.who "MYYARN" applications running and all failing in what seems to be a loop?

cdever — Wed, 02 May 2018 19:26:06 GMT

Thanks but I did check beforehand and there was no crontab whatsoever running for any user on any machine in my cluster.

Re: Why are there dr.who "MYYARN" applications running and all failing in what seems to be a loop?

arald — Wed, 02 May 2018 20:21:52 GMT

This is typically the case when the resources are exceeded. This could be the memory of the node, but also the queue itself. Can you check if the jobs getting stuck are all submitted in the same queue?

https://community.hortonworks.com/questions/96750/yarn-application-stuck-in-accepted-state.html

Re: Why are there dr.who "MYYARN" applications running and all failing in what seems to be a loop?

arald — Wed, 02 May 2018 20:23:14 GMT

and no user connected to the system to start jobs in your cluster?

Re: Why are there dr.who "MYYARN" applications running and all failing in what seems to be a loop?

peter_memon — Wed, 02 May 2018 20:34:02 GMT

I'm having the exact same issue. All of a sudden yesterday - on a cluster that has been up and running for weeks - started spawning six of these at a time for no apparent reason. I kill them and they come back. I've poured over every single log, checked every nook and cranny and cannot figure it out. I have no idea where they are coming from. It is most definitely not a resource issue - these jobs shouldn't even be running - and it's not cron. They are sucking up major CPU when it runs.

If anyone has any thoughts I'd be grateful to hear them!

The other odd thing is that in the past I would see one of these jobs - but only one - never like this.

I'm having the exact same problem. All of a sudden on a cluster that has not changed is spawning off these jobs that are in ACCEPTED status as user Dr.Who and called MYYARN. I've poured over every single log, bounced my cluster several times, there are no cron jobs and it is most definitely not a resource issue. Looking at old logs it looks like it happened periodically - but only once of twice and then it stops. Yesterday it started running wild and as quick as I kill them off it starts another 6 of the exact same job. If anyone has any insight at all I'd be grateful.

And I'm not even using HDP - this is standard Apache Hadoop/Yarn/Spark 2.7.5

Re: Why are there dr.who "MYYARN" applications running and all failing in what seems to be a loop?

harshaldj — Wed, 02 May 2018 20:36:19 GMT

I am wondering if this a security loophole ,since my cluster is not yet kerberized !

Re: Why are there dr.who "MYYARN" applications running and all failing in what seems to be a loop?

peter_memon — Wed, 02 May 2018 20:43:35 GMT

If anyone has any thoughts I'd be grateful to hear them!

The other odd thing is that in the past I would see one of these jobs - but only one - never like this.

Re: Why are there dr.who "MYYARN" applications running and all failing in what seems to be a loop?

harshaldj — Wed, 02 May 2018 20:48:07 GMT

Temporary workaround could be set hadoop.http.staticuser.user=testuser

assign testuser to queue testqueue with 1% resources ?

Re: Why are there dr.who "MYYARN" applications running and all failing in what seems to be a loop?

cdever — Wed, 02 May 2018 20:51:17 GMT

I have the same question (for the same reason, ie. not being kerberized yet).

Re: Why are there dr.who "MYYARN" applications running and all failing in what seems to be a loop?

cdever — Wed, 02 May 2018 22:04:33 GMT

No. I'm the only user connected. And while my cluster is not kerberized, my Ambari connection is made through HTTPS.

Re: Why are there dr.who "MYYARN" applications running and all failing in what seems to be a loop?

xoffey — Thu, 03 May 2018 05:21:12 GMT

(this might be the real answer)

It looks like some kind of an attack. I have seen it on 2 clusters, 1 running HDP and 1 running Hadoop 2.7.4..

Re: Why are there dr.who "MYYARN" applications running and all failing in what seems to be a loop?

xoffey — Thu, 03 May 2018 06:01:28 GMT

Using iptables firewall, I blocked port 8088 and the situiation improved. Too soon to tell if this is a real fix.