Completely uninstall HDP and Ambari

[#1]

After an unsuccessful upgrade, I was forced to completely remove HDP 2.4, Ambari 2.5 and install HDP 2.6. I wanted to avoid reinstalling the OS, so I took advantage of this instruction. Unfortunately, it is not complete. For the problem-free installation of HDP 2.6, you also need to do things like removing service users, cleaning the cron.

So here is my plan of action:

1. Stop all services in Ambari or kill them. In my case, Ambari damaged his database during downgrade and could not start. So I manually killed all the processes on all nodes:

ps –u hdfs (see list of all services below)
kill PID

2. Run python script on all cluster nodes

python /usr/lib/python2.6/site-packages/ambari_agent/HostCleanup.py --silent --skip=users

3. Remove Hadoop packages on all nodes

yum remove hive\*
yum remove oozie\*
yum remove pig\*
yum remove zookeeper\*
yum remove tez\*
yum remove hbase\*
yum remove ranger\*
yum remove knox\*
yum remove storm\*
yum remove accumulo\*
yum remove falcon\*
yum remove ambari-metrics-hadoop-sink 
yum remove smartsense-hst
yum remove slider_2_4_2_0_258
yum remove ambari-metrics-monitor
yum remove spark2_2_5_3_0_37-yarn-shuffle
yum remove spark_2_5_3_0_37-yarn-shuffle
yum remove ambari-infra-solr-client

4. Remove ambari-server (on ambari host) and ambari-agent (on all nodes)

ambari-server stop
ambari-agent stop
yum erase ambari-server
yum erase ambari-agent

5. Remove repositories on all nodes

rm -rf /etc/yum.repos.d/ambari.repo /etc/yum.repos.d/HDP*
yum clean all

6. Remove log folders on all nodes

rm -rf /var/log/ambari-agent
rm -rf /var/log/ambari-metrics-grafana
rm -rf /var/log/ambari-metrics-monitor
rm -rf /var/log/ambari-server/
rm -rf /var/log/falcon
rm -rf /var/log/flume
rm -rf /var/log/hadoop
rm -rf /var/log/hadoop-mapreduce
rm -rf /var/log/hadoop-yarn
rm -rf /var/log/hive
rm -rf /var/log/hive-hcatalog
rm -rf /var/log/hive2
rm -rf /var/log/hst
rm -rf /var/log/knox
rm -rf /var/log/oozie
rm -rf /var/log/solr
rm -rf /var/log/zookeeper

7. Remove Hadoop folders including HDFS data on all nodes

rm -rf /hadoop/*
rm -rf /hdfs/hadoop
rm -rf /hdfs/lost+found
rm -rf /hdfs/var
rm -rf /local/opt/hadoop
rm -rf /tmp/hadoop
rm -rf /usr/bin/hadoop
rm -rf /usr/hdp
rm -rf /var/hadoop

8. Remove config folders on all nodes

rm -rf /etc/ambari-agent
rm -rf /etc/ambari-metrics-grafana
rm -rf /etc/ambari-server
rm -rf /etc/ams-hbase
rm -rf /etc/falcon
rm -rf /etc/flume
rm -rf /etc/hadoop
rm -rf /etc/hadoop-httpfs
rm -rf /etc/hbase
rm -rf /etc/hive 
rm -rf /etc/hive-hcatalog
rm -rf /etc/hive-webhcat
rm -rf /etc/hive2
rm -rf /etc/hst
rm -rf /etc/knox 
rm -rf /etc/livy
rm -rf /etc/mahout 
rm -rf /etc/oozie
rm -rf /etc/phoenix
rm -rf /etc/pig 
rm -rf /etc/ranger-admin
rm -rf /etc/ranger-usersync
rm -rf /etc/spark2
rm -rf /etc/tez
rm -rf /etc/tez_hive2
rm -rf /etc/zookeeper

9. Remove PIDs on all nodes

rm -rf /var/run/ambari-agent
rm -rf /var/run/ambari-metrics-grafana
rm -rf /var/run/ambari-server
rm -rf /var/run/falcon
rm -rf /var/run/flume
rm -rf /var/run/hadoop 
rm -rf /var/run/hadoop-mapreduce
rm -rf /var/run/hadoop-yarn
rm -rf /var/run/hbase
rm -rf /var/run/hive
rm -rf /var/run/hive-hcatalog
rm -rf /var/run/hive2
rm -rf /var/run/hst
rm -rf /var/run/knox
rm -rf /var/run/oozie 
rm -rf /var/run/webhcat
rm -rf /var/run/zookeeper

10. Remove library folders on all nodes

rm -rf /usr/lib/ambari-agent
rm -rf /usr/lib/ambari-infra-solr-client
rm -rf /usr/lib/ambari-metrics-hadoop-sink
rm -rf /usr/lib/ambari-metrics-kafka-sink
rm -rf /usr/lib/ambari-server-backups
rm -rf /usr/lib/ams-hbase
rm -rf /usr/lib/mysql
rm -rf /var/lib/ambari-agent
rm -rf /var/lib/ambari-metrics-grafana
rm -rf /var/lib/ambari-server
rm -rf /var/lib/flume
rm -rf /var/lib/hadoop-hdfs
rm -rf /var/lib/hadoop-mapreduce
rm -rf /var/lib/hadoop-yarn 
rm -rf /var/lib/hive2
rm -rf /var/lib/knox
rm -rf /var/lib/smartsense
rm -rf /var/lib/storm

11. Clean folder /var/tmp/* on all nodes

rm -rf /var/tmp/*

12. Delete HST from cron on all nodes

0 * * * * /usr/hdp/share/hst/bin/hst-scheduled-capture.sh sync
0 2 * * 0 /usr/hdp/share/hst/bin/hst-scheduled-capture.sh

13. Remove databases. I remove the instances of MySQL and Postgres so that Ambari installed and configured fresh databases.

yum remove mysql mysql-server
yum erase postgresql
rm -rf /var/lib/pgsql
rm -rf /var/lib/mysql

14. Remove symlinks on all nodes. Especially check folders /usr/sbin and /usr/lib/python2.6/site-packages

cd /usr/bin
rm -rf accumulo
rm -rf atlas-start
rm -rf atlas-stop
rm -rf beeline
rm -rf falcon
rm -rf flume-ng
rm -rf hbase
rm -rf hcat
rm -rf hdfs
rm -rf hive
rm -rf hiveserver2
rm -rf kafka
rm -rf mahout
rm -rf mapred
rm -rf oozie
rm -rf oozied.sh
rm -rf phoenix-psql
rm -rf phoenix-queryserver
rm -rf phoenix-sqlline
rm -rf phoenix-sqlline-thin
rm -rf pig
rm -rf python-wrap
rm -rf ranger-admin
rm -rf ranger-admin-start
rm -rf ranger-admin-stop
rm -rf ranger-kms
rm -rf ranger-usersync
rm -rf ranger-usersync-start
rm -rf ranger-usersync-stop
rm -rf slider
rm -rf sqoop
rm -rf sqoop-codegen
rm -rf sqoop-create-hive-table
rm -rf sqoop-eval
rm -rf sqoop-export
rm -rf sqoop-help
rm -rf sqoop-import
rm -rf sqoop-import-all-tables
rm -rf sqoop-job
rm -rf sqoop-list-databases
rm -rf sqoop-list-tables
rm -rf sqoop-merge
rm -rf sqoop-metastore
rm -rf sqoop-version
rm -rf storm
rm -rf storm-slider
rm -rf worker-lanucher
rm -rf yarn
rm -rf zookeeper-client
rm -rf zookeeper-server
rm -rf zookeeper-server-cleanup

15. Remove service users on all nodes

userdel -r accumulo
userdel -r ambari-qa
userdel -r ams
userdel -r falcon
userdel -r flume
userdel -r hbase
userdel -r hcat
userdel -r hdfs
userdel -r hive
userdel -r kafka
userdel -r knox
userdel -r mapred
userdel -r oozie
userdel -r ranger
userdel -r spark
userdel -r sqoop
userdel -r storm
userdel -r tez
userdel -r yarn
userdel -r zeppelin
userdel -r zookeeper

16. Run find / -name ** on all nodes. You will definitely find several more files/folders. Remove them.

find / -name *ambari*
find / -name *accumulo*
find / -name *atlas*
find / -name *beeline*
find / -name *falcon*
find / -name *flume*
find / -name *hadoop*
find / -name *hbase*
find / -name *hcat*
find / -name *hdfs*
find / -name *hdp*
find / -name *hive*
find / -name *hiveserver2*
find / -name *kafka*
find / -name *mahout*
find / -name *mapred*
find / -name *oozie*
find / -name *phoenix*
find / -name *pig*
find / -name *ranger*
find / -name *slider*
find / -name *sqoop*
find / -name *storm*
find / -name *yarn*
find / -name *zookeeper*

17. Reboot all nodes

reboot

[#2]

Thanks for sharing this dude!

HDP 2.6 upgrade is very bumpy. I have upgraded from 2.3 to 2.4, but failed at 2.4 ~2.6

[#3]

Hi, i also came across following directories

sudo rm -rf /var/spool/mail/hive

sudo rm -rf /var/spool/mail/zookeeper

sudo rm -rf /var/spool/mail/oozie

sudo rm -rf /var/spool/mail/ams

sudo rm -rf /var/spool/mail/tez

sudo rm -rf /var/spool/mail/livy

sudo rm -rf /var/spool/mail/ambari-qa

sudo rm -rf /var/spool/mail/flume

sudo rm -rf /var/spool/mail/kafka

sudo rm -rf /var/spool/mail/yarn

sudo rm -rf /var/spool/mail/mapred

sudo rm -rf /var/spool/mail/hcat

sudo rm -rf /var/spool/mail/slider

sudo rm -rf /var/spool/mail/sqoop

sudo rm -rf /var/spool/mail/zeppelin

[#4]

Thanks a lot for this post! It helped me a lot removing HDP-2.6.1.0 from my cluster. I used terminator and the following sequence of commands (hope it helps someone else). [See the atached file]completely-uninstall-hw.txt

[#5]

Awesome. Thanks for sharing.

[#6]

Thanks for providing a complete list.

[#7]

Nice Post... Useful for admins...

[#8]

Please add "yum remove hdp-select* -y" and "rm -rf /usr/bin/oozie-client".. I found this was missing when i was uninstalling it using the above script

[#9]

yum remove postgresql*