Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

ambari-metrics : The table SYSTEM.CATALOG does not exist in meta but has a znode

avatar
Rising Star

Hi all,

I've change ambari-metrics to "DISTRIBUTED" so with HDFS.

when I've restarted ambari-metrics-collector, I got this messages :

hbase-ams-regionserver.log

2016-10-10 10:58:56,981 INFO  [B.defaultRpcServer.handler=26,queue=2,port=61300] master.HMaster: Client=ams/null create 'SYSTEM.CATALOG', {TABLE_ATTRIBUTES => {coprocessor$1 => '|org.apache.phoenix.coprocessor.ScanRegionObserver|805306366|', coprocessor$2 => '|org.apache.phoenix.coprocessor.UngroupedAggregateRegionObserver|805306366|', coprocessor$3 => '|org.apache.phoenix.coprocessor.GroupedAggregateRegionObserver|805306366|', coprocessor$4 => '|org.apache.phoenix.coprocessor.ServerCachingEndpointImpl|805306366|', coprocessor$5 => '|org.apache.phoenix.coprocessor.MetaDataEndpointImpl|805306366|', coprocessor$6 => '|org.apache.phoenix.coprocessor.MetaDataRegionObserver|805306367|'}, {NAME => '0', BLOOMFILTER => 'ROW', VERSIONS => '1000', IN_MEMORY => 'false', KEEP_DELETED_CELLS => 'true', DATA_BLOCK_ENCODING => 'FAST_DIFF', TTL => 'FOREVER', COMPRESSION => 'NONE', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}
2016-10-10 10:58:57,095 WARN  [ProcedureExecutorThread-8] procedure.CreateTableProcedure: The table SYSTEM.CATALOG does not exist in meta but has a znode. run hbck to fix inconsistencies.
2016-10-10 10:58:57,308 INFO  [ProcedureExecutorThread-8] procedure2.ProcedureExecutor: Rolledback procedure CreateTableProcedure (table=SYSTEM.CATALOG) id=26 owner=ams state=ROLLEDBACK exec-time=221msec exception=org.apache.hadoop.hbase.TableExistsException: SYSTEM.CATALOG
2016-10-10 10:59:45,777 INFO  [timeline] timeline.HadoopTimelineMetricsSink: Unable to connect to collector, http://datanode001:6188/ws/v1/timeline/metrics

2016-10-10 10:59:45,778 WARN  [timeline] timeline.HadoopTimelineMetricsSink: Unable to send metrics to collector by address:http://datanode001:6188/ws/v1/timeline/metrics
2016-10-10 10:59:45,778 INFO  [timeline] timeline.HadoopTimelineMetricsSink: Unable to connect to collector, http://datanode001:6188/ws/v1/timeline/metrics

2016-10-10 10:59:45,778 WARN  [timeline] timeline.HadoopTimelineMetricsSink: Unable to send metrics to collector by address:http://datanode001:6188/ws/v1/timeline/metrics
2016-10-10 10:59:45,779 INFO  [timeline] timeline.HadoopTimelineMetricsSink: Unable to connect to collector, http://datanode001:6188/ws/v1/timeline/metrics

1 ACCEPTED SOLUTION

avatar
@mayki wogno

There is a chance that the master might be went down while creating the znode or system catalog table data might be deleted from hdfs You can connect to zookeeper and remove the znode from zookeeper and create connection. The znode path is

{zookeeper.znode.parent}/table/SYSTEM.CATALOG.

Connect to zookeeper with this command

./hbase zkcli

Remove znode from zk client:

rmr /hbase-unsecure/table/SYSTEM.CATLOG

View solution in original post

1 REPLY 1

avatar
@mayki wogno

There is a chance that the master might be went down while creating the znode or system catalog table data might be deleted from hdfs You can connect to zookeeper and remove the znode from zookeeper and create connection. The znode path is

{zookeeper.znode.parent}/table/SYSTEM.CATALOG.

Connect to zookeeper with this command

./hbase zkcli

Remove znode from zk client:

rmr /hbase-unsecure/table/SYSTEM.CATLOG