Hbase master problem : Exception in doWork for task: hbase_HBASE_TABLE_AND_REGION_INFO_TASK

[#1]

hi, all:

I have build a cluster using 4 nodes with cm5.12.0  and cdh5.12.1

In Cloudera Manager web screen of HBASE, it's shown that "service monitor did not find activate master"

The service monitor log is as following:

 WARN com.cloudera.cmon.firehose.polling.CdhTask: (14 skipped) Exception in doWork for task: hbase_HBASE_TABLE_AND_REGION_INFO_TASK
java.io.IOException: java.lang.reflect.InvocationTargetException
        at org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:240)
        at org.apache.hadoop.hbase.client.ConnectionManager.createConnection(ConnectionManager.java:412)
        at org.apache.hadoop.hbase.client.ConnectionManager.createConnection(ConnectionManager.java:405)
        at org.apache.hadoop.hbase.client.ConnectionManager.getConnectionInternal(ConnectionManager.java:283)
        at org.apache.hadoop.hbase.client.HConnectionManager.getConnection(HConnectionManager.java:116)
        at com.cloudera.cmf.cdh5client.CDH5HbaseObjectFactoryImpl.getConnection(CDH5HbaseObjectFactoryImpl.java:41)
        at com.cloudera.cmon.firehose.polling.AbstractHConnectionClientTask.doWorkWithClientConfig(AbstractHConnectionClientTask.java:94)
        at com.cloudera.cmon.firehose.polling.AbstractHConnectionClientTask.doWorkWithClientConfig(AbstractHConnectionClientTask.java:26)
        at com.cloudera.cmon.firehose.polling.AbstractCdhWorkUsingClientConfigs.doWork(AbstractCdhWorkUsingClientConfigs.java:45)
        at com.cloudera.cmon.firehose.polling.CdhTask$InstrumentedWork.doWork(CdhTask.java:230)
        at com.cloudera.cmf.cdhclient.util.ImpersonatingTaskWrapper.runTask(ImpersonatingTaskWrapper.java:72)
        at com.cloudera.cmf.cdhclient.util.ImpersonatingTaskWrapper.access$000(ImpersonatingTaskWrapper.java:21)
        at com.cloudera.cmf.cdhclient.util.ImpersonatingTaskWrapper$1.run(ImpersonatingTaskWrapper.java:107)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:415)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1693)
        at com.cloudera.cmf.cdh5client.security.UserGroupInformationImpl.doAs(UserGroupInformationImpl.java:44)
        at com.cloudera.cmf.cdhclient.util.ImpersonatingTaskWrapper.doWork(ImpersonatingTaskWrapper.java:103)
        at com.cloudera.cmf.cdhclient.CdhExecutor$1.call(CdhExecutor.java:125)
        at java.util.concurrent.FutureTask.run(FutureTask.java:262)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.reflect.InvocationTargetException
        at sun.reflect.GeneratedConstructorAccessor94.newInstance(Unknown Source)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
        at org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:238)
        ... 22 more
Caused by: java.lang.NoClassDefFoundError: Could not initialize class org.apache.hadoop.hbase.protobuf.ProtobufUtil
        at org.apache.hadoop.hbase.ClusterId.parseFrom(ClusterId.java:64)
        at org.apache.hadoop.hbase.zookeeper.ZKClusterId.readClusterIdZNode(ZKClusterId.java:75)
        at org.apache.hadoop.hbase.client.ZooKeeperRegistry.getClusterId(ZooKeeperRegistry.java:86)
        at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.retrieveClusterId(ConnectionManager.java:849)
        at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.<init>(ConnectionManager.java:670)
        ... 26 more

This is the ps info of hbase master:

# ps -ef |grep -i master  | grep -i hbase
hbase     75612  41075  0 09:56 ?        00:00:15 /usr/java/jdk1.7.0_67-cloudera/bin/java -Dproc_master -XX:OnOutOfMemoryError=kill -9 %p -Djava.net.preferIPv4Stack=true -Xms855638016 -Xmx855638016 -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=70 -XX:+CMSParallelRemarkEnabled -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp/hbase_hbase-MASTER-7c59a1ea0e2d3f1ab05d0ec168fd90ff_pid75612.hprof -XX:OnOutOfMemoryError=/opt/cloudera-manager/cm-5.12.0/lib64/cmf/service/common/killparent.sh -Dhbase.log.dir=/var/log/hbase -Dhbase.log.file=hbase-cmf-hbase-MASTER-inspire-dev-3.log.out -Dhbase.home.dir=/opt/cloudera/parcels/CDH-5.12.1-1.cdh5.12.1.p0.3/lib/hbase -Dhbase.id.str= -Dhbase.root.logger=INFO,RFA -Djava.library.path=/opt/cloudera/parcels/CDH-5.12.1-1.cdh5.12.1.p0.3/lib/hadoop/lib/native:/opt/cloudera/parcels/CDH-5.12.1-1.cdh5.12.1.p0.3/lib/hbase/lib/native/Linux-amd64-64 -Dhbase.security.logger=INFO,RFAS org.apache.hadoop.hbase.master.HMaster start
hbase     75673  75612  0 09:56 ?        00:00:00 python2.6 /opt/cloudera-manager/cm-5.12.0/lib64/cmf/agent/build/env/bin/cmf-redactor /opt/cloudera-manager/cm-5.12.0/lib64/cmf/service/hbase/hbase.sh master start
hbase     75729  75612  0 09:56 ?        00:00:00 /bin/bash /opt/cloudera-manager/cm-5.12.0/lib64/cmf/service/hbase/hbase.sh master start

This is the ps info of service monitor:

479       73057  41075  6 09:51 ?        00:02:31 /usr/java/jdk1.7.0_67-cloudera/bin/java -server -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -Dmgmt.log.file=mgmt-cmf-mgmt-SERVICEMONITOR-inspire-dev-3.log.out -Djava.awt.headless=true -Djava.net.preferIPv4Stack=true -Dfirehose.schema.dir=/opt/cloudera-manager/cm-5.12.0/share/cmf/schema -XX:PermSize=128m -Dsun.rmi.transport.tcp.handshakeTimeout=10000 -Dsun.rmi.transport.tcp.responseTimeout=10000 -Dlibrary.leveldbjni.path=/opt/cloudera-manager/cm-5.12.0/run/cloudera-scm-agent/process/61-cloudera-mgmt-SERVICEMONITOR -Xms855638016 -Xmx855638016 -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp/mgmt_mgmt-SERVICEMONITOR-7c59a1ea0e2d3f1ab05d0ec168fd90ff_pid73057.hprof -XX:OnOutOfMemoryError=/opt/cloudera-manager/cm-5.12.0/lib64/cmf/service/common/killparent.sh -cp /opt/cloudera-manager/cm-5.12.0/run/cloudera-scm-agent/process/61-cloudera-mgmt-SERVICEMONITOR:/usr/share/java/mysql-connector-java.jar:/opt/cloudera-manager/cm-5.12.0/share/cmf/lib/postgresql-9.0-801.jdbc4.jar:/usr/share/java/oracle-connector-java.jar:/opt/cloudera-manager/cm-5.12.0/share/cmf/lib/*: com.cloudera.cmon.firehose.Main --pipeline-type SERVICE_MONITORING --mgmt-home /opt/cloudera-manager/cm-5.12.0/share/cmf

How to solve it?