Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Hbase Canary monitor is running too long

Hbase Canary monitor is running too long

Explorer

Hi All,

In one of our CDH clusters running with CDH 5.1, Hbase component canary is logging a lot of error messages even when the overall Hbase health is good and the jobs are running fine. The only notifications we have from CM are regionservers going in and out of concerning health (one at a time).

 

Hbase canary logs are full of :

 

"ERROR org.apache.hadoop.hbase.tool.Canary: The monitor is running too long (15002) after timeout limit:15000 will be killed itself !!"

 

I understand that Canary keeps checking for overall region and regionserver health in the cluster but appreciate if someone can share their experience to pin point on the reason.

 

 

Thanks & Regards

Pravdeep

1 REPLY 1

Re: Hbase Canary monitor is running too long

Explorer

For the viewers of the post :

 

Few of the potential reasons which were figured out as the reasons for canary monitor failures :

 

1. Manual restarts of RegionServers.

2. Fetch fail by canary services in case of a remote read(usually related to network hitches).

3. Could also arise due to Hbase-Bug affecting the lower CDH releases (CHD 5.1 or below).

 

These are few of the items that we figured, please feel free to add more causative scenarios.

 

Thanks & Regards
Pravdeep