@Lingesh This is not specific to one region. This happens diffferent ones at different time. Region is getting offlined midstream or marks it as not serving exception.   There are no inconsistencies reported in hbck - Even hbase master doesn't show any exceptions or errors related to it.   What would be the best approach in our case ? Please advise.     ... View more

@bgooley Thanks for the explanation and help. Intially I have tried with self signed and now I have received signed certificates.   Fixed the issue after creating a trustore.  ... View more

@bgooley   Thank you for the response. I have created keystore file using public cert and the private cert and was able to pass from this issue, but Cloudera management services are not starting.   After the creation of keystore, I have copied cacerts from java directory and named it as truststore.jks. This truststore contains root CA , Intermediate CA from the issuing authority. I have added trust store path and password in web console.   I see authentication failures in scm server log and cloudera management services are not coming-    INFO 285679310@scm-web-11:com.cloudera.server.web.cmf.AuthenticationFailureEventListener: Authentication failure for user: '__cloudera_internal_user__mgmt-ACTIVITYMONITOR-15d443db68f73fcfa654fd83bf04540e' from   Host monitor log   INFO com.cloudera.cmf.BasicScmProxy: Authentication to SCM required. INFO com.cloudera.cmf.BasicScmProxy: Using encrypted credentials for SCM   Please let me know if I'm missing anything.         ... View more

Hi @bgooley, Thank you for the response.   Here are the contents - On CM host below CA cert is generated [root@hostname pki]# ls -ltr total 20 -rw-r--r-- 1 root root 1121 Sep 2 12:27 hostname-server.csr -rw-r--r-- 1 root root 1834 Sep 2 12:27 ca-key -rw-r--r-- 1 root root 1314 Sep 2 12:27 ca-cert -rw-r--r-- 1 root root 4198 Sep 2 12:28 hostname.jks lrwxrwxrwx 1 root root 7 Sep 6 10:21 4ba83cc1.0 -> ca-cert [root@hostname pki]# pwd /opt/cloudera/security/pki -> ca cert is imported to keystore along with jsse certs on CM host and same copied to one of the agent hosts. -> copied the contents for ca-cert generate through - openssl req -new -x509 -keyout ca-key -out ca-cert -days 3650 -> created the sym link for the ca-cert, updated the parameters in /etc/cloudera-scm-agent/config.ini along verify_cert_dir -> Here is the output from verification - [root@hostname1.domain cloudera-scm-agent]# openssl s_client -connect hostname:7183 -CAfile /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem < /dev/null CONNECTED(00000003) depth=0 C = Unknown, ST = Unknown, L = Unknown, O = Unknown, OU = Unknown, CN = hostname verify return:1 --- Certificate chain 0 s:/C=Unknown/ST=Unknown/L=Unknown/O=Unknown/OU=Unknown/CN=hostname i:/C=Unknown/ST=Unknown/L=Unknown/O=Unknown/OU=Unknown/CN=hostname --- Server certificate -----BEGIN CERTIFICATE----- MIIDmTCCAoGgAwIBAgIEFdAE2jANBgkqhkiG9w0BAQsFADB9MRAwDgYDVQQGEwdV bmtub3duMRAwDgYDVQQIEwdVbmtub3duMRAwDgYDVQQHEwdVbmtub3duMRAwDgYD VQQKEwdVbmtub3duMRAwDgYDVQQLEwdVbmtub3duMSEwHwYDVQQDExhjc2VraGFy MDIucWU5LmppZ3Nhdy5jb20wHhcNMTgwOTAyMTkyNzAyWhcNMjgwODMwMTkyNzAy WjB9MRAwDgYDVQQGEwdVbmtub3duMRAwDgYDVQQIEwdVbmtub3duMRAwDgYDVQQH EwdVbmtub3duMRAwDgYDVQQKEwdVbmtub3duMRAwDgYDVQQLEwdVbmtub3duMSEw HwYDVQQDExhjc2VraGFyMDIucWU5LmppZ3Nhdy5jb20wggEiMA0GCSqGSIb3DQEB AQUAA4IBDwAwggEKAoIBAQCg4ailB41WoGo6h3fepc6wILw37WaRZ3IApxj0ZTO0 n8po151baw0IEk1X+lYXAkuXloycCKsFUhjiOU2AF2iBbVOYXyA9IB5a9a+b507r EG1NqdJrdN6k4kZrsv+91CizYunjNNheDIgUv/gA7U/sSbvvchXUlN+j4y9LpwKH wcbY/kKLZV9EoNeyrJANccHYiHRQgPLfEcQoHJtDRuWt0TIDqZOR+Xe6fICLfDiC 8WRXqpYKRfFoLVwzFEJuilZXT/J9obzhRgxt9sF7eVV86kFa2EIEZ7aAzZGdIeVT s3RpUO/iSq2X5DxQ7BH8Duc6kgfuaAIOCuANeEGO7trxAgMBAAGjITAfMB0GA1Ud DgQWBBTPmi/hBkF9xJxiN4bvB2QehVuM5TANBgkqhkiG9w0BAQsFAAOCAQEAJEgp 747BfFBIKVPoBCOnxopLM0iJ0jhOTMPo2gld8XoHdH52UOKPlnTsYxqRv9qcU2m8 zjlNxpzT2LMEMGWwk8cP+4ikGrW1vI/bUZOVwFg2pDeho/AE9IOgdn6kd9OQXnL+ D2AB8woJwLWEQ755CEETjvk84kC4BVMMCAB4tDhwAEmzhsehg5B1FzVGfNE5k7fS Ey+ZkNMr/9omWYd4lzHja8pCBCISUCVU7c4ybnEdvLfhyGk/uoNzE2Tf11OsmNtj Fqc4Wbfr9BmcEzDWP6pO5ID6hOu93JaKPmMiW6ofzoJfUdF9Mj/q8pEBc88dqpbf aMvL2RAdfPcFFqSEWA== -----END CERTIFICATE----- subject=/C=Unknown/ST=Unknown/L=Unknown/O=Unknown/OU=Unknown/CN=hostname issuer=/C=Unknown/ST=Unknown/L=Unknown/O=Unknown/OU=Unknown/CN=hostname --- No client certificate CA names sent Peer signing digest: SHA512 Server Temp Key: ECDH, P-256, 256 bits --- SSL handshake has read 1409 bytes and written 415 bytes --- New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-GCM-SHA384 Server public key is 2048 bit Secure Renegotiation IS supported Compression: NONE Expansion: NONE No ALPN negotiated SSL-Session: Protocol : TLSv1.2 Cipher : ECDHE-RSA-AES256-GCM-SHA384 Session-ID: 5B91D501E15D42E800371EBAFE7BF3FD673EEEDA1A10E30CBEC808C2431F2325 Session-ID-ctx: Master-Key: A0DD11CA122343D962AFE236893EC2F00371D37DF3BDC340AB3F1CCDC25C8E48F4DC28255A6CC1926654D1708FE23B9A Key-Arg : None Krb5 Principal: None PSK identity: None PSK identity hint: None Start Time: 1536283905 Timeout : 300 (sec) Verify return code: 0 (ok) --- DONE restarted CM server and agent Please advise if I'm missing anything. ... View more

I did perform rehash after installing open-ssl-perl package. Still we're see SSLError: certificate verify failed on agent logs.    [root@hostname pki]# ls -tlr total 16 -rw-r--r-- 1 root root 1834 Sep 5 20:54 ca-key -rw-r--r-- 1 root root 1314 Sep 5 20:59 ca-cert lrwxrwxrwx 1 root root 7 Sep 6 11:58 4ba83bb9.0 -> ca-cert   Looks like no where to go further :(. Ideas please ...     ... View more

On CM host - Keystore file has been generated and root ca is imported to the same keystore file. In the newly added host, i have copied the cert files under /opt/cloudera/security/pki/ , jsse file and updated the same value in config.ini as indicated in the documentation. I'm not if I missed anything. Any ideas would be really helpful.   Thanks. ... View more

Can you check the unknown host exception whether it is a valid hostname.   Caused by: java.net.UnknownHostException: masterhost1 ... View more

Please check if everything in KDC has been configured and all the processes of KDC master/slaves daemons are running. ... View more

Thanks @bgooley for quick response. Let me check on this and will update the post. ... View more

Thanks Gekas for the suggestions. I am able to do kinit with the keytab that is generated without any issue. While starting zookeeper is taking JAVA_HOME as /usr/java/jdk1.8.0 and I have installed the JCE files at the  /usr/java/jdk1.8.0/jre/lib/security. There is only one version of java that is installed. I did a netcat from source to KDC hosts and those are working fine as well.   Not sure If I'm missing anything.  ... View more

@bgooley   Thank you for the response. Activity monitor logs shows there are a token issue with hue user, which is not the correct thing as Generate credentials was workign fine. This issue is coming after spark being added to the existing , even though situation is completed unreal ..    if we remove activity monitor and install would help ? ... View more