Normally Region splits in HBase is lightweight (the major delay could be attributed by a reference link file creation RPC call to Namenode) and hence should be pretty fast unless NN is undergoing performance issue. If client access this region during this timeframe, it may experience the said exception but that should be transient, transparent and non-fatal to the client application. Do you see any fatal errors at your client application? Do you have customized retry attempts in your client? ... View more

@bgooley Thanks for the explanation and help. Intially I have tried with self signed and now I have received signed certificates.   Fixed the issue after creating a trustore.  ... View more

Hi @bgooley, Thank you for the response.   Here are the contents - On CM host below CA cert is generated [root@hostname pki]# ls -ltr total 20 -rw-r--r-- 1 root root 1121 Sep 2 12:27 hostname-server.csr -rw-r--r-- 1 root root 1834 Sep 2 12:27 ca-key -rw-r--r-- 1 root root 1314 Sep 2 12:27 ca-cert -rw-r--r-- 1 root root 4198 Sep 2 12:28 hostname.jks lrwxrwxrwx 1 root root 7 Sep 6 10:21 4ba83cc1.0 -> ca-cert [root@hostname pki]# pwd /opt/cloudera/security/pki -> ca cert is imported to keystore along with jsse certs on CM host and same copied to one of the agent hosts. -> copied the contents for ca-cert generate through - openssl req -new -x509 -keyout ca-key -out ca-cert -days 3650 -> created the sym link for the ca-cert, updated the parameters in /etc/cloudera-scm-agent/config.ini along verify_cert_dir -> Here is the output from verification - [root@hostname1.domain cloudera-scm-agent]# openssl s_client -connect hostname:7183 -CAfile /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem < /dev/null CONNECTED(00000003) depth=0 C = Unknown, ST = Unknown, L = Unknown, O = Unknown, OU = Unknown, CN = hostname verify return:1 --- Certificate chain 0 s:/C=Unknown/ST=Unknown/L=Unknown/O=Unknown/OU=Unknown/CN=hostname i:/C=Unknown/ST=Unknown/L=Unknown/O=Unknown/OU=Unknown/CN=hostname --- Server certificate -----BEGIN CERTIFICATE----- MIIDmTCCAoGgAwIBAgIEFdAE2jANBgkqhkiG9w0BAQsFADB9MRAwDgYDVQQGEwdV bmtub3duMRAwDgYDVQQIEwdVbmtub3duMRAwDgYDVQQHEwdVbmtub3duMRAwDgYD VQQKEwdVbmtub3duMRAwDgYDVQQLEwdVbmtub3duMSEwHwYDVQQDExhjc2VraGFy MDIucWU5LmppZ3Nhdy5jb20wHhcNMTgwOTAyMTkyNzAyWhcNMjgwODMwMTkyNzAy WjB9MRAwDgYDVQQGEwdVbmtub3duMRAwDgYDVQQIEwdVbmtub3duMRAwDgYDVQQH EwdVbmtub3duMRAwDgYDVQQKEwdVbmtub3duMRAwDgYDVQQLEwdVbmtub3duMSEw HwYDVQQDExhjc2VraGFyMDIucWU5LmppZ3Nhdy5jb20wggEiMA0GCSqGSIb3DQEB AQUAA4IBDwAwggEKAoIBAQCg4ailB41WoGo6h3fepc6wILw37WaRZ3IApxj0ZTO0 n8po151baw0IEk1X+lYXAkuXloycCKsFUhjiOU2AF2iBbVOYXyA9IB5a9a+b507r EG1NqdJrdN6k4kZrsv+91CizYunjNNheDIgUv/gA7U/sSbvvchXUlN+j4y9LpwKH wcbY/kKLZV9EoNeyrJANccHYiHRQgPLfEcQoHJtDRuWt0TIDqZOR+Xe6fICLfDiC 8WRXqpYKRfFoLVwzFEJuilZXT/J9obzhRgxt9sF7eVV86kFa2EIEZ7aAzZGdIeVT s3RpUO/iSq2X5DxQ7BH8Duc6kgfuaAIOCuANeEGO7trxAgMBAAGjITAfMB0GA1Ud DgQWBBTPmi/hBkF9xJxiN4bvB2QehVuM5TANBgkqhkiG9w0BAQsFAAOCAQEAJEgp 747BfFBIKVPoBCOnxopLM0iJ0jhOTMPo2gld8XoHdH52UOKPlnTsYxqRv9qcU2m8 zjlNxpzT2LMEMGWwk8cP+4ikGrW1vI/bUZOVwFg2pDeho/AE9IOgdn6kd9OQXnL+ D2AB8woJwLWEQ755CEETjvk84kC4BVMMCAB4tDhwAEmzhsehg5B1FzVGfNE5k7fS Ey+ZkNMr/9omWYd4lzHja8pCBCISUCVU7c4ybnEdvLfhyGk/uoNzE2Tf11OsmNtj Fqc4Wbfr9BmcEzDWP6pO5ID6hOu93JaKPmMiW6ofzoJfUdF9Mj/q8pEBc88dqpbf aMvL2RAdfPcFFqSEWA== -----END CERTIFICATE----- subject=/C=Unknown/ST=Unknown/L=Unknown/O=Unknown/OU=Unknown/CN=hostname issuer=/C=Unknown/ST=Unknown/L=Unknown/O=Unknown/OU=Unknown/CN=hostname --- No client certificate CA names sent Peer signing digest: SHA512 Server Temp Key: ECDH, P-256, 256 bits --- SSL handshake has read 1409 bytes and written 415 bytes --- New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-GCM-SHA384 Server public key is 2048 bit Secure Renegotiation IS supported Compression: NONE Expansion: NONE No ALPN negotiated SSL-Session: Protocol : TLSv1.2 Cipher : ECDHE-RSA-AES256-GCM-SHA384 Session-ID: 5B91D501E15D42E800371EBAFE7BF3FD673EEEDA1A10E30CBEC808C2431F2325 Session-ID-ctx: Master-Key: A0DD11CA122343D962AFE236893EC2F00371D37DF3BDC340AB3F1CCDC25C8E48F4DC28255A6CC1926654D1708FE23B9A Key-Arg : None Krb5 Principal: None PSK identity: None PSK identity hint: None Start Time: 1536283905 Timeout : 300 (sec) Verify return code: 0 (ok) --- DONE restarted CM server and agent Please advise if I'm missing anything. ... View more

@vijithv,   First, firewalls can easily block UDP and allow TCP.  I mentioned that was a possible cause. Also, depending on how you have your /etc/krb5.conf configured, a different KDC could have been contacted.   You can see distinctly in the failure via UDP that there is a socket timeout for each attempt to connect to the KDC.  This is a failure at the networking side where a client cannot connect to a server.  Since no connection was ever made via UDP, there was no change for it to know to try TCP.  That "switching" is done based on a response of KRB5KRB_ERR_RESPONSE_TOO_BIG I believe so if no response is made, no "switching" to TCP will occur.   If you really want to get to the bottom of this, recreate the problem while capturing packets via tcpdump like this:   # tcpdump -i any -w ~/kerberos_broken.pcap port 88   Then, with the problem fixed reproduce again while capturing packets:   # tcpdump -i any -w ~/kerberos_fixed.pcap port 88   Use Wireshark (it does a great job of decoding Kerberos packets) and you will be able to see the entire interaction. This will show us information to help determine the cause. Wireshark is here:  https://www.wireshark.org/     ... View more

Could you please mark it as answered so the community will be benifited ?  ... View more