In one of our CDH clusters running with CDH 5.1, Hbase component canary is logging a lot of error messages even when the overall Hbase health is good and the jobs are running fine. The only notifications we have from CM are regionservers going in and out of concerning health (one at a time).
Hbase canary logs are full of :
"ERROR org.apache.hadoop.hbase.tool.Canary: The monitor is running too long (15002) after timeout limit:15000 will be killed itself !!"
I understand that Canary keeps checking for overall region and regionserver health in the cluster but appreciate if someone can share their experience to pin point on the reason.