2209
Posts
230
Kudos Received
82
Solutions
About
My expertise is not in hadoop but rather online communities, support and social media. Interests include: photography, travel, movies and watching sports.
My Accepted Solutions
| Title | Views | Posted |
|---|---|---|
| 507 | 05-07-2025 11:41 AM | |
| 1014 | 02-27-2025 12:49 PM | |
| 2889 | 06-29-2023 05:42 AM | |
| 2430 | 05-22-2023 07:03 AM | |
| 1799 | 05-22-2023 05:42 AM |
01-20-2017
12:42 PM
We have just published a new Engineering blog post How to secure ‘Internet exposed’ Apache Hadoop that may be of interest as well.
... View more
01-18-2017
10:46 AM
2 Kudos
Cloudera takes cluster security very seriously, and provides guidelines for securing CDH environments:
http://www.cloudera.com/documentation/enterprise/latest/topics/sg_edh_overview.html
Security measures become especially important when clusters are exposed to the internet. For example, hackers using network scanning tools are actively looking for WebHDFS ports on clusters, and when they find an open port, they can wreak havoc on the cluster (delete data, steal data, corrupt data).
There are many other services and access considerations that should be protected as well.
It’s imperative that you design clusters in a secure fashion which will not leave the services interfaces exposed to the Internet this way. It’s our strong recommendation that you secure your cluster with kerberos, TLS, proper firewall or proxy access, and use the guidance from our security guide to protect your deployment.
Users affected:
All unsecured clusters exposed to the internet
Impact:
Cluster data may be copied, downloaded and deleted. Cluster altered or permanently disabled
Action required:
For perimeter security consider a quick test to be a check of: "Can I access this cluster from a public network with no vpn or other security in place?" If so, check with your network administration team or in the Cloudera community discussion forums as a resource to this evaluation and setup of proper security.
Securing a cluster requires the following
Perimeter security configured to protect access to your deployment
https://www.cloudera.com/documentation/enterprise/latest/topics/sg_edh_overview.html
If using Amazon or Azure, review the Director discussion on cloud security group setup and allow only inbound SSH for authentication and encryption of access for VPC security for cloud here at:
http://www.cloudera.com/documentation/director/latest/topics/director_get_started.html
A KDC to be provided to enable kerberos authentication
https://www.cloudera.com/documentation/enterprise/latest/topics/sg_auth_overview.html
Enabling kerberos authentication through Cloudera Manager's web UI by using the wizard (consider enabling Cloudera Manager TLS first)
https://www.cloudera.com/documentation/enterprise/latest/topics/cm_sg_intro_kerb.html
Enabling encryption of data in transit for both RPC and data, and Cloudera Manager as well as cluster web UI's
Encryption Overview:
http://www.cloudera.com/documentation/enterprise/latest/topics/sg_encryption.html
TLS:
http://www.cloudera.com/documentation/enterprise/latest/topics/cm_sg_guide_ssl_certs.html
http://www.cloudera.com/documentation/enterprise/latest/topics/cm_sg_config_tls_security.html
Kerberos RPC (HDFS Encrypted Transport):
http://www.cloudera.com/documentation/enterprise/latest/topics/cm_sg_hdfs_encrypt_transport.html
Best practice considerations like changing default passwords, creating users and groups deploying navigator auditing and reviewing access attempts on an ongoing basis.
Cloudera provides an overview on securing a cluster properly for the Cloudera 5.x platform in a Vision blog post. It is provided here for reference:
https://vision.cloudera.com/production-ready-hadoop-an-overview-of-security-in-cloudera-5/
To check if your existing cluster has authentication security enabled: Navigate within Cloudera Manager from the home page to the Administration menu. Click the "Security" sub menu. A table of the clusters being managed is presented, and the statement "Successfully enabled Kerberos" will be next to the cluster name. The following link discusses the concepts and steps to completing this setup properly:
http://www.cloudera.com/documentation/enterprise/latest/topics/cm_sg_authentication.html
If you are using CDH without Cloudera Manager, both the hadoop.security.authentication parameter needs to not be set to “kerberos”, and the hadoop.security.authorization parameter needs to be set to “true” in core-site.xml to indicate that security is enabled:
http://www.cloudera.com/documentation/enterprise/latest/topics/cdh_sg_hadoop_security_enable.html
To verify if TLS is enabled for Cloudera Manager and Navigator, navigate from the Cloudera Manager home page to the Administration Menu -> Settings, and search for TLS in the configuration settings search field:
http://www.cloudera.com/documentation/enterprise/latest/topics/how_to_configure_cm_tls.html
To verify if TLS is enabled for CDH components managed by Cloudera Manager, search for “tls enabled” in each of the services:
http://www.cloudera.com/documentation/enterprise/latest/topics/cm_sg_hadoop_ssl_cm.html
To verify if TLS is enabled for CDH components not managed by Cloudera Manager, look for the setting “hadoop.ssl.enabled” within the configuration files.
For CDH and hadoop community users the following Apache reference documentation can be consulted for considerations on securing webHDFS.
https://hadoop.apache.org/docs/r2.7.3/hadoop-project-dist/hadoop-hdfs/WebHDFS.html#Authentication
Here is a copy of the Apache release documentation in our mirror for current platform:
https://archive.cloudera.com/cdh5/cdh/5/hadoop/hadoop-project-dist/hadoop-hdfs/WebHDFS.html?_ga=1.208877266.1151128972.1420475180#Authentication
If your cluster has been compromised, data has been deleted, or you would like to engage with a Cloudera security professional services team member, please reach out to your account manager or contact us at sales@cloudera.com.
... View more
01-10-2017
05:22 AM
Thank you for marking your issue as solved @DanielWhite. Can you advise what the solution was?
... View more
11-30-2016
08:09 AM
1 Kudo
I checked with the certification team and received this response.
Please see the exams of interest mapped to their corresponding course. Note that while our courses will assist with preparing for the certification exams, additional study and/or hands on experience may also be required to ensure a passing score. This is especially true for the CCP Data Engineer exam.
CCAH --> Cloudera Administrator Training CCA Spark and Hadoop Developer --> Developer Training for Spark and Hadoop CCP Data Engineer --> Big Data Applications (OnDemand Only)
Each of the elements above are linked to their specific information page on our website. The courses are offered as Instructor-led as well as OnDemand (self-paced) for flexible scheduling. Dates, locations and prices are available to you on the website as well.
... View more
11-23-2016
12:21 AM
1 Kudo
I stand corrected. I double checked with the certification team and they are working on a new cluster now based on 5.8/5.9 to be added near the start of next year. Thanks
... View more
11-21-2016
11:23 PM
Since the exam is based on the major release of CDH 5 I wouldn't expect any changes until CDH 6 at this point. I can ask the certification team to be sure though if you would like.
... View more
11-10-2016
05:14 AM
I am happy to see that the upgrade resolved your issue. Best of luck as you continue with the project. 🙂
... View more
11-02-2016
12:35 PM
Thanks for asking @sim6. Since the community is a peer to peer network it can take some time to receive a response. Clouderans do participate as time allows but for the most part it is not their primary task. Depending on what your situation is, the Cloudera Developer Program may match up to your needs.
... View more
10-14-2016
05:31 AM
@kerjo and @manpreet2 I would like to clarify something. Are you both facing the same issue or something similar?
If the issues are only similar it may be better for @manpreet2 to post the second issue into a new thread to avoid confusion. If they are the same issue, please disregard my question and carry on. 🙂
... View more
10-13-2016
05:41 AM
Due to the changes involving support access the correct route would be to go up your chain and through your Cloudera representative. If you run into issues with that route send me a private message with additional details and I can get with the Cloudera representative to contact your company.
... View more