Member since
09-12-2017
16
Posts
0
Kudos Received
0
Solutions
09-27-2018
06:55 AM
@ramarov Thank you for the suggestion! We'll keep it in mind for our future sprint updates.
... View more
06-26-2018
04:56 AM
Check your /tmp directory on your Cloudera Manager server. It's possibly at 100%. Looks like the service monitor writes an hbase directory to /tmp and it it can't, you get some strange errors around incorrectly loaded protocol buffers.
... View more
09-26-2017
08:36 PM
@mbigelow Any ideas?
... View more
09-20-2017
05:16 AM
I would suggest starting a new thread with all the details involved in your specific case.
... View more
09-19-2017
02:18 AM
Hello, do you solved this?
... View more
09-12-2017
03:33 AM
hi, all: I have build a cluster using 4 nodes with cm5.12.0 and cdh5.12.1 In Cloudera Manager web screen of HBASE, it's shown that "service monitor did not find activate master" The service monitor log is as following: WARN com.cloudera.cmon.firehose.polling.CdhTask: (14 skipped) Exception in doWork for task: hbase_HBASE_TABLE_AND_REGION_INFO_TASK
java.io.IOException: java.lang.reflect.InvocationTargetException
at org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:240)
at org.apache.hadoop.hbase.client.ConnectionManager.createConnection(ConnectionManager.java:412)
at org.apache.hadoop.hbase.client.ConnectionManager.createConnection(ConnectionManager.java:405)
at org.apache.hadoop.hbase.client.ConnectionManager.getConnectionInternal(ConnectionManager.java:283)
at org.apache.hadoop.hbase.client.HConnectionManager.getConnection(HConnectionManager.java:116)
at com.cloudera.cmf.cdh5client.CDH5HbaseObjectFactoryImpl.getConnection(CDH5HbaseObjectFactoryImpl.java:41)
at com.cloudera.cmon.firehose.polling.AbstractHConnectionClientTask.doWorkWithClientConfig(AbstractHConnectionClientTask.java:94)
at com.cloudera.cmon.firehose.polling.AbstractHConnectionClientTask.doWorkWithClientConfig(AbstractHConnectionClientTask.java:26)
at com.cloudera.cmon.firehose.polling.AbstractCdhWorkUsingClientConfigs.doWork(AbstractCdhWorkUsingClientConfigs.java:45)
at com.cloudera.cmon.firehose.polling.CdhTask$InstrumentedWork.doWork(CdhTask.java:230)
at com.cloudera.cmf.cdhclient.util.ImpersonatingTaskWrapper.runTask(ImpersonatingTaskWrapper.java:72)
at com.cloudera.cmf.cdhclient.util.ImpersonatingTaskWrapper.access$000(ImpersonatingTaskWrapper.java:21)
at com.cloudera.cmf.cdhclient.util.ImpersonatingTaskWrapper$1.run(ImpersonatingTaskWrapper.java:107)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1693)
at com.cloudera.cmf.cdh5client.security.UserGroupInformationImpl.doAs(UserGroupInformationImpl.java:44)
at com.cloudera.cmf.cdhclient.util.ImpersonatingTaskWrapper.doWork(ImpersonatingTaskWrapper.java:103)
at com.cloudera.cmf.cdhclient.CdhExecutor$1.call(CdhExecutor.java:125)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.GeneratedConstructorAccessor94.newInstance(Unknown Source)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:238)
... 22 more
Caused by: java.lang.NoClassDefFoundError: Could not initialize class org.apache.hadoop.hbase.protobuf.ProtobufUtil
at org.apache.hadoop.hbase.ClusterId.parseFrom(ClusterId.java:64)
at org.apache.hadoop.hbase.zookeeper.ZKClusterId.readClusterIdZNode(ZKClusterId.java:75)
at org.apache.hadoop.hbase.client.ZooKeeperRegistry.getClusterId(ZooKeeperRegistry.java:86)
at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.retrieveClusterId(ConnectionManager.java:849)
at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.<init>(ConnectionManager.java:670)
... 26 more This is the ps info of hbase master: # ps -ef |grep -i master | grep -i hbase
hbase 75612 41075 0 09:56 ? 00:00:15 /usr/java/jdk1.7.0_67-cloudera/bin/java -Dproc_master -XX:OnOutOfMemoryError=kill -9 %p -Djava.net.preferIPv4Stack=true -Xms855638016 -Xmx855638016 -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=70 -XX:+CMSParallelRemarkEnabled -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp/hbase_hbase-MASTER-7c59a1ea0e2d3f1ab05d0ec168fd90ff_pid75612.hprof -XX:OnOutOfMemoryError=/opt/cloudera-manager/cm-5.12.0/lib64/cmf/service/common/killparent.sh -Dhbase.log.dir=/var/log/hbase -Dhbase.log.file=hbase-cmf-hbase-MASTER-inspire-dev-3.log.out -Dhbase.home.dir=/opt/cloudera/parcels/CDH-5.12.1-1.cdh5.12.1.p0.3/lib/hbase -Dhbase.id.str= -Dhbase.root.logger=INFO,RFA -Djava.library.path=/opt/cloudera/parcels/CDH-5.12.1-1.cdh5.12.1.p0.3/lib/hadoop/lib/native:/opt/cloudera/parcels/CDH-5.12.1-1.cdh5.12.1.p0.3/lib/hbase/lib/native/Linux-amd64-64 -Dhbase.security.logger=INFO,RFAS org.apache.hadoop.hbase.master.HMaster start
hbase 75673 75612 0 09:56 ? 00:00:00 python2.6 /opt/cloudera-manager/cm-5.12.0/lib64/cmf/agent/build/env/bin/cmf-redactor /opt/cloudera-manager/cm-5.12.0/lib64/cmf/service/hbase/hbase.sh master start
hbase 75729 75612 0 09:56 ? 00:00:00 /bin/bash /opt/cloudera-manager/cm-5.12.0/lib64/cmf/service/hbase/hbase.sh master start This is the ps info of service monitor: 479 73057 41075 6 09:51 ? 00:02:31 /usr/java/jdk1.7.0_67-cloudera/bin/java -server -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -Dmgmt.log.file=mgmt-cmf-mgmt-SERVICEMONITOR-inspire-dev-3.log.out -Djava.awt.headless=true -Djava.net.preferIPv4Stack=true -Dfirehose.schema.dir=/opt/cloudera-manager/cm-5.12.0/share/cmf/schema -XX:PermSize=128m -Dsun.rmi.transport.tcp.handshakeTimeout=10000 -Dsun.rmi.transport.tcp.responseTimeout=10000 -Dlibrary.leveldbjni.path=/opt/cloudera-manager/cm-5.12.0/run/cloudera-scm-agent/process/61-cloudera-mgmt-SERVICEMONITOR -Xms855638016 -Xmx855638016 -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp/mgmt_mgmt-SERVICEMONITOR-7c59a1ea0e2d3f1ab05d0ec168fd90ff_pid73057.hprof -XX:OnOutOfMemoryError=/opt/cloudera-manager/cm-5.12.0/lib64/cmf/service/common/killparent.sh -cp /opt/cloudera-manager/cm-5.12.0/run/cloudera-scm-agent/process/61-cloudera-mgmt-SERVICEMONITOR:/usr/share/java/mysql-connector-java.jar:/opt/cloudera-manager/cm-5.12.0/share/cmf/lib/postgresql-9.0-801.jdbc4.jar:/usr/share/java/oracle-connector-java.jar:/opt/cloudera-manager/cm-5.12.0/share/cmf/lib/*: com.cloudera.cmon.firehose.Main --pipeline-type SERVICE_MONITORING --mgmt-home /opt/cloudera-manager/cm-5.12.0/share/cmf How to solve it?
... View more
Labels: