Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Why are there dr.who "MYYARN" applications running and all failing in what seems to be a loop?

Why are there dr.who "MYYARN" applications running and all failing in what seems to be a loop?

New Contributor

We're running a HDP 2.5 cluster and today we noticed a series of dr.who "MYYARN" applications running, failing, and then resubmitting to YARN again and again. In what seems to be an "infinite loop". We can't figure out what the applications are doing and why they are failing. Any thoughts? Many thanks in advance!

34 REPLIES 34

Re: Why are there dr.who "MYYARN" applications running and all failing in what seems to be a loop?

New Contributor

I'm hitting exactly the same issue here with HDP 2.6.

It looks like some kind of DOS attack but I have no clue on how to handle this?

Any help from Hortonworks would be appreciated.

Re: Why are there dr.who "MYYARN" applications running and all failing in what seems to be a loop?

New Contributor

I am also facing issue with dr.who user with MYYARN application submitting in loop. But those are staying in "ACCEPTED" status. Total 18 application are launched. no clue !!

Re: Why are there dr.who "MYYARN" applications running and all failing in what seems to be a loop?

New Contributor

Mine are also staying in "ACCEPTED" status. And they're launched every 3 seconds... It's becoming a problem on the ResourceManager. When I look at the logs, I can only see actions coming from within my cluster.

Re: Why are there dr.who "MYYARN" applications running and all failing in what seems to be a loop?

Super Collaborator

did you check this:
https://community.hortonworks.com/questions/2349/tip-when-you-get-a-message-in-job-log-user-dr-who.h...

The solution there was:
RESOLUTION Customer changed the following property in core-site.xml to resolve the issue. Other values such as hdfs or mapred also resolve the issue. If the cluster is managed by Ambari, this should be added in Ambari > HDFS > Configurations>Advanced core-site > Add Property

hadoop.http.staticuser.user=yarn

Re: Why are there dr.who "MYYARN" applications running and all failing in what seems to be a loop?

New Contributor

Yeah I was thinking to change static user but this was not there until today afternoon. Its suddenly started spawning applications using dr.user.

Re: Why are there dr.who "MYYARN" applications running and all failing in what seems to be a loop?

New Contributor

I did the change but it didn't change anything. Instead of being 'dr.who', it's now 'yarn' user that is feeding applications every 3 seconds that get stuck as "ACCEPTED". I still can't find how these applications are being triggered. Any other clue?

Re: Why are there dr.who "MYYARN" applications running and all failing in what seems to be a loop?

New Contributor

can you check if any cronjob triggering this ? crontab -l !

Re: Why are there dr.who "MYYARN" applications running and all failing in what seems to be a loop?

New Contributor

Thanks but I did check beforehand and there was no crontab whatsoever running for any user on any machine in my cluster.

Highlighted

Re: Why are there dr.who "MYYARN" applications running and all failing in what seems to be a loop?

Super Collaborator

and no user connected to the system to start jobs in your cluster?