Member since
04-22-2014
1218
Posts
341
Kudos Received
157
Solutions
My Accepted Solutions
| Title | Views | Posted |
|---|---|---|
| 26238 | 03-03-2020 08:12 AM | |
| 16377 | 02-28-2020 10:43 AM | |
| 4709 | 12-16-2019 12:59 PM | |
| 4470 | 11-12-2019 03:28 PM | |
| 6652 | 11-01-2019 09:01 AM |
07-09-2019
06:09 PM
@TCloud, First, some background on what the settings do: (1) Use TLS Encryption for Agents - This means that Cloudera Manager's heartbeat port (7182) will require TLS. All agents need in order to use tls is the following: [Security] use_tls=1 (2) If you then want the agents to verify the Cloudera Manager certificate, you can then add the verify_cert_file configuration: [Security] use_tls=1 verify_cert_file=/path/to/truststore The "verify_cert_file" value is a path to a file that contains one or more certfificates trusted by the agent. If you are using self-signed certificates, you can simply use the PEM format of the CM server's certificate. In your case, the file needs to contain the following: -----BEGIN CERTIFICATE----- MIID9TCCAt2gAwIBAgIEVUJkFTANBgkqhkiG9w0BAQsFADB1MQswCQYDVQQGEwJV UzETMBEGA1UECBMKQ2FsaWZvcm5pYTEUMBIGA1UEBxMLTG9zIEFuZ2VsZXMxDDAK BgNVBAoTA01EUzEMMAoGA1UECxMDTURTMR8wHQYDVQQDExZjbS1yMDFubjAxLm13 cy5tZHMueHl6MB4XDTE5MDQwMTAyMzc1MFoXDTI5MDMyOTAyMzc1MFowdTELMAkG A1UEBhMCVVMxEzARBgNVBAgTCkNhbGlmb3JuaWExFDASBgNVBAcTC0xvcyBBbmdl bGVzMQwwCgYDVQQKEwNNRFMxDDAKBgNVBAsTA01EUzEfMB0GA1UEAxMWY20tcjAx bm4wMS5td3MubWRzLnh5ejCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEB AIVFcUGgk3fKC1zuKuSTmc6HQLIxvI9fZXjWMOwYagoJjAgKEFUKtjUTgY15IqT9 VXeBuOICTzDvXQbHEh87KAVLoyQa+DkBR9yoyj5hhRUz+fG7erVgvf1AyB5V289Q ZZir70nzrnRfGdGh4gFCPH2CKvTBndvLYIbDlSh8P5gjLyck7KMW7GrbtZfxcTGD qBJYB8okK6/Fs6fYD5UggfYIVhEgAorciHCYlnsJagSda/8Mn5a9WtYLjMJY8hEW RjZxLLI6aunrd/J24ulUEURaWiqjtfna7Q++mDL4GLtf/sUe4xYjt5zy7qMWPh6Q aGNVcR+bMVgs+e2qATPbiLkCAwEAAaOBjDCBiTBFBgNVHSUEPjA8BggrBgEFBQcD AQYIKwYBBQUHAwIGCCsGAQUFBwMDBggrBgEFBQcDBAYIKwYBBQUHAwgGCCsGAQUF BwMJMCEGA1UdEQQaMBiCFmNtLXIwMW5uMDEubXdzLm1kcy54eXowHQYDVR0OBBYE FCE0pxrJ9cnfzdjidWWTm5YQFoZ/MA0GCSqGSIb3DQEBCwUAA4IBAQBe9m5Tg76b Uux3yZudIAtISKwSGP2hz4kYpGEM5ykjiY4UAtScFHLcPEJxFN6K6KtZDy02nStI aLE9dNB3SayLspC6nC1gAB05D8viFKniOVSG8TQgTytOHG/A7UWz5yQF4uJXiETe de82xiMt75O1jaePFrsMG2twSZxRTLpd52WGit4A5B8LM2ADTR/wsFvQMhvZA+y5 N+1/pw8bQK4SGGWZ4DWXP+Q5bC/6xP5gN1H9bmvvZILNVma7w+ko7Wr8qIfRB8RU 2j1EiHAp99CjA1on1XVH//3TAhRhvXojh+YnZAwe4JCkCI6vVEYSkbgWBK1txYEa uZw5Rx5pl+gE -----END CERTIFICATE----- (3) Use TLS Authentication of Agents to Server - This means that Cloudera Manager requires that any agent heartbeating will supply its public certificate as means of authenticating to Cloudera Manager. Agent requirements: Along with the previous configuration, you need the following: client_key_file=/path/to/private/key/file client_keypw_file=/path/to/file/containing/password/for/private/key/file client_cert_file=/path/to/server/certificate/paired/with/agent/private/key For example: [Security] use_tls=1 max_cert_depth=9 verify_cert_file=/opt/cloudera/security/cacerts/ClouderaSEC_combined.pem client_key_file=/opt/cloudera/security/x509/host.key client_keypw_file=/opt/cloudera/security/agent_key_pw client_cert_file=/opt/cloudera/security/x509/host.pem Cloudera Manager requirement: CM truststore must contain the signer of each agent's certificate CM must be configured with a trust store in "Cloudera Manager TLS/SSL Client Trust Store File" It appears this file may contain all the hosts' agent certificates: /etc/pki/ca-trust/extracted/java/jssecacerts NEXT STEPS: It appears that your agents are not configured to perform agent authentication so Cloudera Manager is returning the Bad Certificate TLS Alert and failing the TLS Handshake. This is a good sign as it means the agent seems to now be able to trust the signer of your CM certificate. To correct the situation, you need to make sure that you configure client_key_file, client_keypw_file, and client_cert_file Make sure to restart the agent after adding them.
... View more
07-09-2019
05:26 PM
1 Kudo
NOTE: You can grep your /var/log/cloudera-scm-server/cloudera-scm-server.log file for the following which will be printed if Auto-TLS failed due to license restriction: Did not generate Auto-TLS certificates because of missing enterprise license This will appear at the time you attempted the install.
... View more
07-09-2019
05:15 PM
@kernel8liang, Actually, yes you do need an enterprise license to use Auto-TLS as mentioned here: https://www.cloudera.com/documentation/enterprise/latest/topics/auto_tls.html#auto_tls This applies to Cloudera Manager 6.0.1 and up.
... View more
07-09-2019
01:26 PM
1 Kudo
@kal, Yup, you can stop Service Monitor and Host Monitor and then remove the contents of cloudera-host-monitor and cloudera-service-monitor directories. When you restart Servcie and Host Monitor, new indexes will be start from scratch. You will only lose the historical information.
... View more
07-09-2019
12:55 PM
1 Kudo
@VijayM, I was out on PTO for a couple weeks so I just saw your comment. This looks like a pretty unusual issue. Based on the fact that the agent never seems to know that it should start the role, this is a problem that is occurring in Cloudera Manager only (before the command is to be started by the agent). The stack trace and error messages show us something interesting: (1) CM sees that you want to start Navigator Metadata Server: 2019-06-25 08:33:07,341 INFO 439059796@scm-web-6908:com.cloudera.cmf.command.flow.CmdStep: Executing command work: Execute command Start this Navigator Metadata Server on role Navigator Metadata Server (a301-8744-7259) (2) An exception occurs right away: 2019-06-25 08:33:07,884 WARN 439059796@scm-web-6908:com.cloudera.cmf.service.GenericBringUpRoleCommand: Unexpected exception in command execution java.util.NoSuchElementException at com.google.common.collect.Iterators$1.next(Iterators.java:77) at com.google.common.collect.Iterators.getOnlyElement(Iterators.java:325) at com.google.common.collect.Iterables.getOnlyElement(Iterables.java:271) at com.cloudera.cmf.service.mgmt.RoleSSLUtils.getTrustStoreServerOpts(RoleSSLUtils.java:43) at com.cloudera.cmf.service.mgmt.NavMetaServerRoleHandler.getEnvironmentForConcreteRole(NavMetaServerRoleHandler.java:99) (3) Cloudera Manager fails the command start with: 2019-06-25 08:33:07,885 ERROR 439059796@scm-web-6908:com.cloudera.cmf.command.flow.WorkOutputs: CMD id: 7269 Role failed to start due to error null. 2019-06-25 08:33:08,032 WARN 439059796@scm-web-6908:com.cloudera.cmf.service.GenericBringUpRoleCommand: Role 324/mgmt-NAVIGATORMETASERVER-220e59e0745b944070fe360a2cc717c7 Process cloudera-mgmt-NAVIGATORMETASERVER does not exist - unable to update status CONCLUSION: Based on my review of the stack trace and Cloudera Manager code, the cause of this issue is that TLS/SSL Client Truststore File Location is configured with a path, but Cloudera Manager Server TLS/SSL Client Trust Store Password is empty in your Cloudera Management Service Configuration. The Navigator code will try to pull the values for each and the line where it fails is where it attempts to read the Trust Store password. I removed the password for my truststore and was able to reproduce. FIX: Configure Cloudera Manager Server TLS/SSL Client Trust Store Password with the password for your truststore file. Let us know if that helps!
... View more
07-09-2019
12:18 PM
@BiggieSmalls, The "you are a Hue admin..." error is pertty generic and means that something went wrong when trying to access HDFS. I suggest checking your /var/log/runcpserver.log file for error messages and stack traces that occur when the UI error message occurs. That will help us start investigating the cause. We don't know what will solve your problem until we know what is causing it. Cheers, Ben
... View more
07-09-2019
12:11 PM
Hello @kal , First, we need to be sure about the directory to which you are referring. You mentioned /var/lib/cloudera-service/host-monitor , but that seems unconventional. If we are talking about the Cloudera Host Monitor Host Monitor Storage Directory, that is usually: /var/lib/cloudera-host-monitor If you are, indeed, referring to the directory that is defined in the Cloudera Management Service "Host Monitor Storage Directory" configuration parameter, then that is where the Host Monitor stores information collected about hosts. Host Monitor pulls from indexes in that directory in order to provide information for Host Charts and historical data. CDH does not depend on it so it will not impact your services. If you remove the data in /var/lib/cloudera-host-monitor, then you will not have access to historical host information and your charts will start from the point the Host Monitor is restarted. If you are OK with that, then you can do the following: Stop the Host Monitor (via Cloudera Manager) move all the contents (recursively) out of /var/lib/cloudera-host-monitor Start the Host Monitor (via Cloudera Manager) If everything looks OK to you at that point, you could go ahead and delete the files you moved out. In short, you can delete the host monitor storage files without impacting your cluster. Host Monitor is used for charts and historical reference, but it will start from scratch if you delete all the files. Ben
... View more
07-09-2019
09:51 AM
NOTE: If you are trying to do agent authentication to CM, then you will need to configure the key/cert for the agent to present to CM. This is a separate, additional configuration than we have been discussing if you have configured Agent Authentication to CM in Cloudera Manager.
... View more
07-09-2019
09:49 AM
@TCloud, If you are using self-signed certificates, then it sounds as if you may have things right. For the heartbeat issue, though, we can use openssl to see if we have things configured correctly and also see what public certificate is returned by Cloudera Manager for the heartbeat Run: # openssl s_client -connect cm-r01nn01.mws.mds.xyz:7182 -showcerts -CAfile /opt/cloudera/security/pki/cm-r01en01.mws.mds.xyz.pem I believe you will still get an error now that you have explained some of the backstory. When the agent is making a connection to CM to perform the heartbeat, it needs to do a TLS handshake per your configuration. Part of this handshake is the client (the agent) deciding if it trusts the certificate returned from CM. In your config.ini, you are connecting to the CM host: cm-r01nn01.mws.mds.xyz However, your pem file name indicates that you are pointing verify_cert_file to the agent's host's certificate, not the certificate of the Cloudera Manager host. In essence, the agent needs to trust the certificate returned by CM in the handshake. So, the verify_cert_file file will need to contain the certificate for cm-r01nn01.mws.mds.xyz If that doesn't make sense, provide the output from the openssl command above and then also the output from: # openssl x509 -in /opt/cloudera/security/pki/cm-r01en01.mws.mds.xyz.pem -text -noout From that we can provide further advice.
... View more
07-08-2019
02:18 PM
@BiggieSmalls, Just to be sure, it would be good to share the stack trace from your incident to be sure it is exactly the same. In the original post, the stack trace that Hue is attempting to issue a GET REST API call to Oozie to get its status. When attempting a connection, the client (Hue) decides it cannot verify the server certificate that is presented by the Oozie Server. By default, Hue will attempt to verify the certificate returned by the server to which it is trying to connect. It looks to see if it trusts the signer, the hostname to which the connection was made is included in the certificate among other things. In order to know what to suggest exactly in terms of determining the cause of the problem, we need to see the stack trace you are seeing that led you to determine you are seeing the same issue in the original post. Thanks, Ben
... View more