ambari-metrics : The table SYSTEM.CATALOG does not exist in meta but has a znode

[#1]

Hi all,

I've change ambari-metrics to "DISTRIBUTED" so with HDFS.

when I've restarted ambari-metrics-collector, I got this messages :

hbase-ams-regionserver.log

2016-10-10 10:58:56,981 INFO  [B.defaultRpcServer.handler=26,queue=2,port=61300] master.HMaster: Client=ams/null create 'SYSTEM.CATALOG', {TABLE_ATTRIBUTES => {coprocessor$1 => '|org.apache.phoenix.coprocessor.ScanRegionObserver|805306366|', coprocessor$2 => '|org.apache.phoenix.coprocessor.UngroupedAggregateRegionObserver|805306366|', coprocessor$3 => '|org.apache.phoenix.coprocessor.GroupedAggregateRegionObserver|805306366|', coprocessor$4 => '|org.apache.phoenix.coprocessor.ServerCachingEndpointImpl|805306366|', coprocessor$5 => '|org.apache.phoenix.coprocessor.MetaDataEndpointImpl|805306366|', coprocessor$6 => '|org.apache.phoenix.coprocessor.MetaDataRegionObserver|805306367|'}, {NAME => '0', BLOOMFILTER => 'ROW', VERSIONS => '1000', IN_MEMORY => 'false', KEEP_DELETED_CELLS => 'true', DATA_BLOCK_ENCODING => 'FAST_DIFF', TTL => 'FOREVER', COMPRESSION => 'NONE', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}
2016-10-10 10:58:57,095 WARN  [ProcedureExecutorThread-8] procedure.CreateTableProcedure: The table SYSTEM.CATALOG does not exist in meta but has a znode. run hbck to fix inconsistencies.
2016-10-10 10:58:57,308 INFO  [ProcedureExecutorThread-8] procedure2.ProcedureExecutor: Rolledback procedure CreateTableProcedure (table=SYSTEM.CATALOG) id=26 owner=ams state=ROLLEDBACK exec-time=221msec exception=org.apache.hadoop.hbase.TableExistsException: SYSTEM.CATALOG
2016-10-10 10:59:45,777 INFO  [timeline] timeline.HadoopTimelineMetricsSink: Unable to connect to collector, http://datanode001:6188/ws/v1/timeline/metrics

2016-10-10 10:59:45,778 WARN  [timeline] timeline.HadoopTimelineMetricsSink: Unable to send metrics to collector by address:http://datanode001:6188/ws/v1/timeline/metrics
2016-10-10 10:59:45,778 INFO  [timeline] timeline.HadoopTimelineMetricsSink: Unable to connect to collector, http://datanode001:6188/ws/v1/timeline/metrics

2016-10-10 10:59:45,778 WARN  [timeline] timeline.HadoopTimelineMetricsSink: Unable to send metrics to collector by address:http://datanode001:6188/ws/v1/timeline/metrics
2016-10-10 10:59:45,779 INFO  [timeline] timeline.HadoopTimelineMetricsSink: Unable to connect to collector, http://datanode001:6188/ws/v1/timeline/metrics

[#2 · Accepted solution]

@mayki wogno

There is a chance that the master might be went down while creating the znode or system catalog table data might be deleted from hdfs You can connect to zookeeper and remove the znode from zookeeper and create connection. The znode path is

{zookeeper.znode.parent}/table/SYSTEM.CATALOG.

Connect to zookeeper with this command

./hbase zkcli

Remove znode from zk client:

rmr /hbase-unsecure/table/SYSTEM.CATLOG