Securing Your Cluster in the Public Cloud

[#1]

Protect Your Cloud Big Data Assets

Step 1: Do not put anything into the cloud unless you have a CISO, Chieft Security Architect, Certified Cloud Administrator, full understanding of your PII and private data, a Lawyer to defend you against the coming lawsuits, full understanding of Hadoop, Hadoop Certified Administrators, a Hadoop premier support contract, a security plan, full understanding of your Hadoop architecture and layout.

Step 2: Study all running services in Ambari.

Step 3: Confirm and check all of your TCP/IP ports. Hadoop has a lot of them!

Step 4: if you are not using a service, do not run it.

Step 5: By default, disable all access to everything, always. Only open ports and access when something and someone critical cannot access them.

Step 6: SSL, SSH, VPN and Encryption Everywhere.

Step 7: Run Knox! Set it up correctly.

Step 8: Run Kali and audit all your IPs and ports.

Step 9: Use Kali hacking tools to attempt to access all your web ports, shells and other access points.

Step 10: Run in a VPC

Step 11: Setup security groups. Never open to 0.0.0.0 or all ports or all IPs!?!??!?!!!

Step 12: If this seems too hard, don't run in the cloud.

Step 14: Step 13 is unlucky, skip that one.

Step 15: Read all the recommended security documentation and use it.

Step 16: Kerberize everything.

Step 17: Run Metron

My recommendation is get a professional services contract with an experience Hadoop organization or use something like Microsoft HDInsight or HDC that is managed.

TCP/IP Ports

• 50070 : Name Node Web UI
• 50470 : Name Node HTTPS Web UI
• 8020, 8022, 9000 : Name Node via HDFS
• 50075 : Data Node(s) WebUI
• 50475 : Data Node(s) HTTPS Web UI
• 50090 : Secondary Name Node
• 60000 : HBase Master
• 8080 : HBase REST
• 9090 : Thrift Server
• 50111 : WebHCat
• 8005 : Sqoop2
• 2181: Zookeeper
• 9010: Zookeeper JMX
• 50020
• 50010
• 50030
• 8021
• 50060
• 51111
• 9083
• 10000, 60010, 60020, 60030, 2888, 3888, 8660, 8661, 8662, 8663, 8660, 8651, 3306, 80, 8085, 1004, 1006, 8485, 8480, 2049, 4242,14000, 14001, 8021, 9290, 50060, 8032, 8030, 8031, 8033, 8088, 8040, 8042, 8041, 10020, 13562, 19888, 9090, 9095, 9083, 16000, 12000, 12001, 3181, 4181, 8019, 9010, 8888, 11000, 11001, 7077, 7078, 18080, 18081, 50100

There's more of these if you are also running your own visualization tools, other data websites, other tools, Oracle, SQL Server, mail, NiFi, Druid, etc...

Reference

http://www.slideshare.net/bunkertor/hadoop-security-54483815

https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.4.0/bk_installing_manually_book/content/set_up_...

https://aws.amazon.com/articles/1233/

http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-network-security.html

https://www.quora.com/What-are-the-best-practices-in-hardening-Amazon-EC2-instance

https://stratumsecurity.com/2012/12/03/practical-tactical-cloud-security-ec2/

http://hortonworks.com/solutions/security-and-governance/

http://metron.incubator.apache.org/