Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Why are there dr.who "MYYARN" applications running and all failing in what seems to be a loop?

Re: Why are there dr.who "MYYARN" applications running and all failing in what seems to be a loop?

New Contributor

Definitely an attack.
I finally managed to spot a malicious IP address on my ResourceManager node that I could block.

Re: Why are there dr.who "MYYARN" applications running and all failing in what seems to be a loop?

New Contributor

We have the same issue, it started at the same moment (30 April and restarts 1 May) and we are using HDP 2.6, do you think is a Hortonworks error or a time bomb?

Please, if anyone finds a solution, please tell us in this post

Highlighted

Re: Why are there dr.who "MYYARN" applications running and all failing in what seems to be a loop?

New Contributor

@Michael Coffey was right. It seems to be a DOS attack on port 8088. Blocking this port should (temporarily) alleviate the problem. I'm also not sure if this is the final solution...

Re: Why are there dr.who "MYYARN" applications running and all failing in what seems to be a loop?

New Contributor

This solution works, but it is strange that 5 different clusters of 5 different companies have the same attack at the same time.

Re: Why are there dr.who "MYYARN" applications running and all failing in what seems to be a loop?

New Contributor

I totally agree with this. It's weird. It sounds like some kind of time bomb. I tried blocking the 8088 port but it didn't work for me (at least not for long). MYYARN jobs kept on piling up. It's over 20,000 jobs for the last couple of days. Any help from Hortonworks team would be appreciated.

Re: Why are there dr.who "MYYARN" applications running and all failing in what seems to be a loop?

Super Collaborator

Is your cluster directly connected to the internet, so that any internet user can connect to your port 8088? And also your cluster is not kerberized?

There are regulary running kind of campaigns to search for unprotected or vulnerable services via Internet, so it shouldn't surprise that the attack is almost simultaneously hitting several clusters. There are even search engines available that will list you all services reachable from the internet, so that one can search for 'give me all unprotected hadoop machines'.

If your cluster is unprotected, the only solution will be to protect it, via firewall, via kerberos etc...

Re: Why are there dr.who "MYYARN" applications running and all failing in what seems to be a loop?

Can you guys check if you see the below process in your nodemanager machines?

/tmp/java -c /tmp/h.conf

Re: Why are there dr.who "MYYARN" applications running and all failing in what seems to be a loop?

New Contributor

according to ps, there is no process with "conf.h"; according to ls, there is no /tmp/java; checked on 2 nodes on 2 clusters

Re: Why are there dr.who "MYYARN" applications running and all failing in what seems to be a loop?

New Contributor

Temporary solution of blocking port 8088 is working for me as of now.

Re: Why are there dr.who "MYYARN" applications running and all failing in what seems to be a loop?

New Contributor

The problem is that the user dr.who who is launching the applications through the Ambari's API

We find another solution, we do not give access with ranger to the user dr.who and the port 8088 closed