Key Trustee Ships as a self contained product. While the front end of Key Trustee can and does support TLS 1.2 the postgreSQL version that presently ships with Key Trustee has no support for cipher enforcement. Even though it is capable of supporting TLS 1.2 other cipher specs cannot be forcefully disabled.   If your goal is to ensure that all of the components on your system use only TLS 1.2 you should be advised that the security scanners will continue to report other TLS versions for the release of postgreSQL that currently ships with Key Trustee even with available work arounds. We also do not provide any UI based methods for altering the TLS configuration for the backing Key Trustee Database. You will need to reach out through normal support channels to obtain the work around we can provide.   The KT service exports the following parameters along with others during startup to the environment. You should not attempt to update any part of the parcel content. Note where the python home is located.   export KEYTRUSTEE_PYTHONHOME=$KEYTRUSTEE_SERVER_DIRNAME/lib/python2.6 export KEYTRUSTEE_PYTHONPATH=$KEYTRUSTEE_PYTHONHOME/lib:$KEYTRUSTEE_PYTHONHOME/site-packages export PYTHONPATH=$KEYTRUSTEE_PYTHONPATH:$KEYTRUSTEE_SERVER_DIRNAME export BIN=${KEYTRUSTEE_SERVER_DIRNAME}/bin export KEYTRUSTEE_PYTHON=${BIN}/python export CRYPTOGRAPHY_ALLOW_OPENSSL_100=True   The parcel presently ships with pyOpenSSL-0.14.   KEYTRUSTEE_PYTHONHOME/site-packages/pyOpenSSL-0.14-py2.6.egg-info   Please show us the documentation you are referencing and also please be advised that there are various other components which also use python through out the platform. ... View more

Hello,   Please review the Horton works community documentation. It covers rack awareness better than out documentation currently does and it is accurate. The behaviour you are describing is exactly how rack awareness works.   When HDFS is made rack aware it will place 2 replicas within the same rack and a 3rd in a remote rack. That is because local nodes within the same rack are preferable both for the HDFS framework as well as most job schedulres. With a replication factor of 3 HDFS will not place a block on every rack. ... View more

Hello,   Can you please tell us what documentation you have reviewed? Setting the rack locations of host is normally what is used to determine block placement. If HDFS for example is aware of your topology it should ensure that at least one replica is on another rack.   https://www.cloudera.com/documentation/enterprise/5-15-x/topics/cm_mc_specify_rack.html https://community.hortonworks.com/articles/43057/rack-awareness-1.html ... View more

Hello,   I've reviewed the errors you have provided and this error appears to be coming from the underlying crypto library. According to RFC-5280 standards the CN and Description Fields each must not exceed 64 characters in length. This character limit is hard coded in the openSSL framework and cannot be altered without changing the source code within openSSL. The Subject Alt Name field has a much longer character limit.   Unfortunately the log data you provided is truncated and it is difficult to tell what precisely was being performed and with what options when the failure occured.   It would appear as though we use the following information to obtain the hostname for the CN field during the init process.   hostname = socket.gethostname()   It's fairly trival to create a short online python command to see what this returns. Can you please use something like this in order to perform your word count on the output?   python -c "import socket; hostname = socket.gethostname(); print hostname;"     ... View more

Hello,   The KMS service within the hadoop frame work is responsible for handling key material. The KMS is not responsible for encrypting or decrypting data. The KTS is not connected to access control over data. All encrypted data handling occurs within the DFS client framework.   1.) You will need to review and understand the concepts laid out in our documentation and upstream related to securing the KMS. Cloudera ships a secure by default ACL configuration. New keys are not automatically alotted any access controls. No users are authorized to access new keys which have undefined Acess Controls. The KMS ACL engine is designed to control key release and it is not in any way connected to the underlying HDFS Posix controls. The ACL engine indicately controls access to Encrypted data by controlling access to key material.   https://www.cloudera.com/documentation/enterprise/5-16-x/topics/cdh_sg_kms_acl_config.html   2.) Your question here is moderately confusing. HDFS Encryption is Transparent to the DFS client. If a user is authorized to perform decrypt EEK operations they may view the encrypted data. Raw encrypted data is not normally visible to clients in the capacity I believe you are attempting to describe outside the context of the raw end point exposed to the supergroup users.   3.) You can access the raw data end point as a super user if you would like to verify that the data is encrypted. This information is documented publicly in both upstream and in our documentation.   hdfs dfs -ls /.reserved/raw/   4.) The Generate EEK operation is handled internally by the HDFS service user and is not normally exposed to operators.   If you are a cloudera customer you should reach out to your account team for additional training and details. ... View more

We have started the process of certifying certain OpenJDK versions for some releases of CDH. However full JDK feature convergance between the Oracle JDK and OpenJDK is not planned to occur until OpenJDK 11. ... View more

Issues with the links in /etc/alternatives are normally caused by corruption of the alternatives subsystem in the Operating system. We do not own this component and we've filed request with Red Hat in the past in an effort to address it. The corruption can occur for a variety of reasons but the solution is usually fairly simple.   The troublshooting steps you have taken however may make the situation worse or more specifically steps 3 and 4. The information in /etc typically impacts clients and not service daemons. If the upgrade completed without error the downgrade may have sweeping impacts that can break other services unless you have also rolled back all of the metadata services manually using backups. I'd recommend that you move back to the newer release of CDH.   On each host where you are having the problem the typicaly steps to resolve this issue are as follows, assuming nothing else is broken.   1. Stop all roles on the host. 2. Stop the agent. 3. Ensure that the agent and supervisord are fully stopped. 4. Verify the output of the follow command and ensure that it only captures information related to cloudera.   \ls -l /etc/alternatives/ | grep "\/opt\/cloudera"   5. Run the following shell script to cleanup and remove all alternatives related to cloudera parcels.   $ \ls -l /etc/alternatives/ | grep "\/opt\/cloudera" | awk {'print $9'} | \ while read m; do if [[ -e /var/lib/alternatives/${m} ]] ;then echo "Removing ${m}"; \ rm -fv /var/lib/alternatives/${m} ; fi; rm -fv /etc/alternatives/${m}; done   6. Start the agent and review the log data in /var/log/cloudera-scm-agent/cloudera-scm-agent.log. You should see the agent attempt to create new alternatives for your parcels.   7. Verify where the alternative point and restart the roles on the host. ... View more

Hello,   Can you try commenting out this line in your nginx configuration?   >  proxy_set_header X-Forwarded-Proto https;   The error that you are reporting now is being returned directly by Nginx. It means that something is trying to use plain text instead of TLS. You may also have to alter the way the server is configured. At the moment you have set Nginx to accept on TLS request only which may impact your ability to proxy to the backend since it does not use TLS.   You may need to alter the server block, y ou may need to comment out:   > ssl on   Then alter the listen paramter on Niginx like so:   > listen 8001 ssl;   https://docs.nginx.com/nginx/admin-guide/security-controls/terminating-ssl-tcp/ ... View more

Hi,   This is quite unusual, the configuration normally doesn't change like this. Can you please login to Cloudera Manager then goto the follow location and check to see if it's been set to something unexepcted.   CM -> Administration -> Settings ->  Cloudera Manager Hostname Override   If the value is blank, which is the default, it may indicate that the result of InetAddress.getLocalhost() is incorrect which can be caused by a number of things including entries in /etc/hosts. If you are certain that DNS works properly and that there are no erronous entries in /etc/hosts you can try setting the HostName Override. Then restart both Cloudera Manager and the Management services. ... View more

If your production environment will lack internet access I highly recommend that you work with members of your account team and/or infrastructure teams to setup an internal mirror/repository which you can point your installation to. Setting this up now will allow you to be more prepared for later deployments. You should review this set of documentation. More specifically the documentation related to setting up and using an internal parcel or package repository. Please be advised that the links below are for 6x and they may or may not match the release you intend to deploy. Custom Installations: https://www.cloudera.com/documentation/enterprise/latest/topics/cm_ig_custom_installation.html Internal Parcel: https://www.cloudera.com/documentation/enterprise/latest/topics/cm_ig_create_local_parcel_repo.html Internal Package: https://www.cloudera.com/documentation/enterprise/latest/topics/cm_ig_create_local_package_repo.html ... View more

Outside of the last time items below I am not seeing anything else that might be wrong with your configuration. Are you certain that TLS is not already enabled on Hue?    You seem to have proxy_set_header Host twice in the first location path. Can you please remove this one shown below?    >    proxy_set_header Host $host;   Also please uncomment the follow line under the static location. This alias is required if you deployed using parcels if you deployed using packages the path will be slightly different.   > #alias /opt/cloudera/parcels/CDH/lib/hue/build/static/;   If you used packages.   # If Hue was installed with packaging install: ## alias /usr/lib/hue/build/static/; ... View more

Hello,   While we do not provide support directly for Nginx reviewing the log data you have posted it would appear as though the Hue backend you are attempting to proxy to on Node 1 is not accepting incoming request. Are you sure that there are no firewalls between the proxy and the node yo uare connecting to? Are you sure that Hue is available at the address you have configured?   2018/12/11 20:04:18 [error] 19352#19352: *14 connect() failed (111: Connection refused) while connecting to upstream, client: <client_ip>, server: gravalytics.com, request: "GET /favicon.ico HTTP/1.1", upstream: "http://<node1_ip>:8888/favicon.ico", host: "gravalytics.com:8001", referrer: " https://gravalytics.com:8001/ " ... View more

Hello,   In order to try to help you with this issue we will need to see more log data or you will need to review the log data. A 404 error is a generic error code which may originate from one of two places in your scenario. The 404 may originate directly from Nginx or it may originate from Hue.    You will need to review the nginx logs and hue logs to determine what is returning the 404 error and for what resource. One way to make this easier is to remove one of your upstream servers from the server group so that it only proxies to one Hue instance while you investigate the 404 error condition. ... View more

Hello,   Changes to the Yarn configuration should not have direct impacts on The Managment services or CM. The error you have reported is routing through the JDBC driver. Communication link failures where packets have not been recieved over a period of time are generally caused by a problem on the backing database system.   com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@20bf4111 -- Acquisition Attempt Failed!!! Clearing pending acquires. While trying to acquire a needed new resource, we failed to succeed more than the maximum number of allowed acquisition attempts (5). Last acquisition attempt exception: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure   The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server. We would suggest reviewing the database logging data to ensure that the connection is not being rejected for any particular reason. Certain version of Mysql track so called "bad" clients and store them in a table after a certain number of failed communication attempts. ... View more

Hello,   CDH 6 is classified as a Major Upgrade. CDH 5 is presently based on Hadoop 2.x where as CDH 6 has moved forward to Hadoop 3.x. There are many feature enhancements and changes across the platform related to enhanced capabilities, performance, and security.   If you desire features only available in newer releases of Hadoop and it's components then CDH 6 may be the version for you though we do intend to maintain CDH 5 in accordance with our EOL. The overall life of CDH 5 is subject to change based on current external activities.   If you are planning a new cluster deployment that does not have any data it may be a good idea to make this decision early. While there is an upgrade path from CDH 5 -> CDH 6 we have restricted those paths to certain releases though we expect to address additional releases as time moves forward. ... View more

Hello AKB,   Unfortunately the answer to your question is, no. It will not be easier or better to rely soley on TLS termination on a reverse proxy. For most balancing/proxying algorithims, hardware, and software we recommend TCP Passthrough which means that all Hadoop services must still have TLS properly deployed as well as enabled.   If you cluster is accessible by any external network we would advise that you properly deploy both Kerberos and TLS on your cluster. ... View more

Hello,   Is this something you are still actively seeing? The 5.16 release has not yet been made available. If you are still having trouble obtaining the proper repo information please update this post so that we can work with our teams internally. ... View more

@balusu     If you are a licensed customer, using Key Trustee, please open a case immediately. While we would like to help you with this on the community, parts of the diagnostic process on Key Trustee Server and it's clients may expose sensitive information from your environment.   DO NOT arbutrarily attempt to sync the KMS client data again without diagnostics performed by Cloudera!   Syncing the client data again without working with us may result in unexpected data loss. There may be less risk if this is a POC, DEV, or a new environment but at this point in time that is not visible to us.   When we are working with the Key Trustee KMS component and not the Key Server there are no Active or Passive delegations. All configured Key Management Server Proxies are used within the cluster. ... View more

Hi,   From the post it is not clear what type of KMS is deployed on this cluster. As this is a community post I will assume that you are using the JCEKS backed KMS. Please note that the ASF and Cloudera both suggest that this KMS implementation not be used in production deployments.   https://hadoop.apache.org/docs/current/hadoop-kms/index.html   > Caused by: java.lang.NullPointerException: No KeyVersion exists for key 'testTLS1'   The error above occurs when a KMS instance makes a call to getKeyVersion. This call only occurs when the KMS is actually attempting to retrieve a key from the backend key datastore. When you see this error it quite literally means that the requested key cannot be found in the backing key datastore.   If you have more than one KMS for example it likely means that the backing keystore on each KMS is not in sync. The KMS core does not in any way replicate the key material between individual instances at this time. n+1 capabilities are offered through other means in other providers that use external methods to synchronize this data.   If however you believe you have performed a synchronization of the keystores on your own in a safe manner it is possible that one or more backing keystors are corrupt. The KMS will attempt to automatically create a new JCE Key Store when this occurs and export the keys it can. Unfortunately the KMS stores information in these keystores in a format that cannot be manipulate with Keytool. If the automated recovery fails then all data in the JCE keystore is lost and by proxy all keys as well as data are lost.   For the core JCE backed KMS the logging information will appear in /var/log/hadoop-kms. In most cases in order to have a meaningful idea of what is wrong you will need to review the kms.log, kms-audit.log. and the catalina logs. A this time if you are on 6.x tomcat has been replaced by Jetty so the catalina logs will not exists.   The behavior you are seeing suggest that you have more than one KMS instance and the request fails over to a working instance which allows the write to occur. Actual data encryption and decryption occurs on the DFS client. This means that in order for a read or write to occur the client must have a key and the information required to open the read or write pipe. ... View more

Hi,   Auto TLS in CDH 6 at present simplifies and automates a number of task across supported services in relation to the deployment of TLS. There are still however some manual steps depending on what precisely you are trying to do. W e do plan to continue working on Auto TLS to add features and capabilities in the future.   https://www.cloudera.com/documentation/enterprise/latest/topics/auto_tls.html#auto_tls   If you are a licenesed customer you should reach out to your account team. We do have other non-public and unoffical tools that can help automate certificate handling for all releases.  ... View more

Hi,   Even though this documentation targets Kafka you can use a similar process to create your own CA using openssl. You will want to review steps 1-3 but please do not use the bash script that is in the documentation. The bash script is an example to automate things but it will not do what you need it to do. I know that this answer is a little terse but you should be able to use what you need from this documentation and the normal TLS deployment documentation to get you to a place you want to be. You may have done step 1 on all of your host for example or already created CSRs.   https://www.cloudera.com/documentation/kafka/latest/topics/kafka_security.html#deploying_ssl_for_kafka   Once you have certificates and truststores on all of your host you can configure services to use the keystores/truststores, and pem files where required. If you follow the steps in the guide you should in theory have all of the formats you need.   - Step 1 creates all of the java keys stores on each host. - Step 2 creates the certificate signing authority and the java truststore you should distribute. Be aware that you will also need to distribute your root certificate in x509 pem format for the agent. - Step 3 creates the CSR on each host, signs that CSR resulting in a usable client certificate, and imports that certificate into the Java Keystore. Be aware that you will also want to distribute the x509 pem encoded version of each host certificate to the host it belongs to so that it may be used by the agent.   I'd recommend creating a structure similar to this on each node.   /opt/cloudera/security/ /opt/cloudera/security/ca-certs /opt/cloudera/security/jks /opt/cloudera/security/x509   On the host you decide to make your CA you should add an additional sub-folder labeled setup with a structure similar to the one below. Keep in mind that this is for example purposes only and you can ultimately use any structure you decide.     /opt/cloudera/security/setup/keys /opt/cloudera/security/setup/client_certs /opt/cloudera/security/setup/csr /opt/cloudera/security/setup/rootca ... View more

Hello AKB,   @AKB wrote: I have created a slef-signed cert using this:  Where do I find the other files mentioned in step 7 - https://www.cloudera.com/documentation/enterprise/5-15-x/topics/how_to_configure_cm_tls.html#concept_wk4_jlx_qw ? While it is possible to use Self-Signed certificates we strongly recommend against their use for a variety of reasons including but not limited to Security, Usability, and Maintenance. We are debating the removal of this documentation internally. However with that said if you have followed this guide successfully the files you need to work with in step 7 should already exists on each host.   @AKB wrote: Can one SSL certificate be used on all nodes of the cluster?   While it is possible to use SAN and Wildcard certificates for a cluster we also strongly discourage the use of a single certificate for all cluster services. Using a single certificate exposes you to unnecessary risk and actually reduces the overall security of your TLS implementation because all of your certificates and services will rely on a single private key. There are certainly uses for SAN and Wildcards but you should avoid using these to cover all services in an environment. You will find similar information published by OWASP. https://www.owasp.org/index.php/Transport_Layer_Protection_Cheat_Sheet#Rule_-_Do_Not_Use_Wildcard_Certificates ... View more