HBase Java client connection timeout

Question

I have a Spring boot application using HBase Java client, version 2.1.0-cdh6.1.1, connect to a Hbase with kerberos. When running a tomcat instance in our server, a simple query could take 5 minutes to complete and timeout found in log file. I can't find out what's wrong.

Connection configuration:

config.set("hbase.zookeeper.quorum",  quorum);
 config.set("hbase.zookeeper.clientPort", "2181");
config.set("hadoop.security.authentication", "kerberos");
config.set("hbase.security.authentication", "kerberos");
config.set("hbase.cluster.distributed", "true");
config.set("hbase.rpc.protection", "privacy");
config.set("hbase.regionserver.kerberos.principal", "hbase/_HOST@mycompany.com");
config.set("hbase.master.kerberos.principal", "hbase/_HOST@mycompany.com");
			
config.set("hbase.client.retries.number", "3");
config.set("hbase.client.pause", "500");
config.set("zookeeper.recovery.retry",  "1");    System.setProperty("java.security.krb5.conf",ResourceUtils.getFile("classpath:krb5.conf").getPath());
	        
File file = ResourceUtils.getFile(keytabFileLocation);

UserGroupInformation.setConfiguration(config);
UserGroupInformation.loginUserFromKeytab(userIdEmail, file.getPath());
Connection conn = ConnectionFactory.createConnection(config);

here is a one minute gap from 17:23:42,637 to 17:24:43,126 in log file, then the time out at the end.

How to fix this issue?

Thank you

smdas · Answer

Hello @bigdatanewbie

Thanks for using Cloudera Community. Based on the Post, a Spring Boot Application fails to connect to HBase in a Kerberized Cluster.

Looking at the Logs, We observe the RegionServer "fepp-cdhdn-d2.mycompany.com/172.29.233.141" isn't able to complete the RPC Request within 60 Seconds Timeout. With 3 retries, the failure being persisted causes the Overall App failure. The fact that the Application identifies the RegionServer hosting the Regions of Table "hbasepoc:alan_test" indicates the Client is able to fetch the Metadata (hbase:meta) Table's Region from the ZooKeeper & connect with RegionServer hosting "hbase:met" Region to pull the required Metadata information.

Let's verify the Table "hbasepoc:alan_test" is Healthy by running an HBCK on the Table & using HBase Shell to perform the same Operation as being performed by the Spring Boot Application. If the HBCK Report on the table (Obtained via "hbase hbck -details hbasepoc:alan_test") shows no Inconsistency & HBase Shell access to the Table with the same Operation completes successfully, Reviewing the concerned Host (Wherein Spring Boot Application is running) connectivity with the HBase Setup along with RegionServer Logs would be helpful.

Additionally, We can try increasing the Timeout or Retries to confirm the Issue lies with Delayed Response or any other Underlying issues.

- Smarak

Sign up

Please use your Cloudera account.

Login to the community

Please use your Cloudera account.