Why are there dr.who "MYYARN" applications running and all failing in what seems to be a loop?

[#1]

We're running a HDP 2.5 cluster and today we noticed a series of dr.who "MYYARN" applications running, failing, and then resubmitting to YARN again and again. In what seems to be an "infinite loop". We can't figure out what the applications are doing and why they are failing. Any thoughts? Many thanks in advance!

[#2]

Definitely an attack.
I finally managed to spot a malicious IP address on my ResourceManager node that I could block.

[#3]

We have the same issue, it started at the same moment (30 April and restarts 1 May) and we are using HDP 2.6, do you think is a Hortonworks error or a time bomb?

Please, if anyone finds a solution, please tell us in this post

[#4]

@Michael Coffey was right. It seems to be a DOS attack on port 8088. Blocking this port should (temporarily) alleviate the problem. I'm also not sure if this is the final solution...

[#5]

This solution works, but it is strange that 5 different clusters of 5 different companies have the same attack at the same time.

[#6]

I totally agree with this. It's weird. It sounds like some kind of time bomb. I tried blocking the 8088 port but it didn't work for me (at least not for long). MYYARN jobs kept on piling up. It's over 20,000 jobs for the last couple of days. Any help from Hortonworks team would be appreciated.

[#7]

Is your cluster directly connected to the internet, so that any internet user can connect to your port 8088? And also your cluster is not kerberized?

There are regulary running kind of campaigns to search for unprotected or vulnerable services via Internet, so it shouldn't surprise that the attack is almost simultaneously hitting several clusters. There are even search engines available that will list you all services reachable from the internet, so that one can search for 'give me all unprotected hadoop machines'.

If your cluster is unprotected, the only solution will be to protect it, via firewall, via kerberos etc...

[#8]

Can you guys check if you see the below process in your nodemanager machines?

/tmp/java -c /tmp/h.conf

[#9]

according to ps, there is no process with "conf.h"; according to ls, there is no /tmp/java; checked on 2 nodes on 2 clusters

[#10]

Temporary solution of blocking port 8088 is working for me as of now.

[#11]

The problem is that the user dr.who who is launching the applications through the Ambari's API

We find another solution, we do not give access with ranger to the user dr.who and the port 8088 closed