FATAL org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryServer: Error starting ApplicationHistoryServer

[#1]

Hi Gurus.

Our Ambari Metrics collector goes down. Our is Kerberos env. Ambari Version is :Version2.4.2.0

i see below errors

in ambari-metrics-collector.log i see belwo error

FATAL org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryServer: Error starting ApplicationHistoryServer org.apache.hadoop.yarn.server.applicationhistoryservice.metrics.timeline.MetricsSystemInitializationException: Error creating Metrics Schema in HBase using Phoenix. Caused by: org.apache.phoenix.exception.PhoenixIOException: SYSTEM.CATALOG Caused by: org.apache.hadoop.hbase.TableNotFoundException: SYSTEM.CATALOG

in hbase-ams-master-xxxxxxxxxx.log

2017-12-23 07:40:57,717 ERROR [xxxxxx,61300,1513971592600_ChoreService_1] master.BackupLogCleaner: Failed to get hbase:backup table, therefore will keep all files org.apache.hadoop.hbase.TableNotFoundException: hbase:backup

I implemented below procedure but no luck

https://community.hortonworks.com/questions/73577/problem-with-ambari-metrics.html

1) Turn on Maintenance mode

2) Stop Ambari Metrics

3) hadoop fs -rmr /ams/hbase/*

4) rm -rf /var/lib/ambari-metrics-collector/hbase-tmp/*

5) [zk: localhost:2181(CONNECTED) 0] ls /

[zk: localhost:2181(CONNECTED) 1] rmr /ams-hbase-unsecure

6) Start Ambari Metrics

7) Turn off Maintenance mode

My setting are :

hbase.rootdir = hdfs://hpdprod:8020/user/ams/hbase

hbase.cluster.distributed=true

timeline.metrics.service.operation.mode=distributed

Any help is really appreciate

Thanks

JJ

[#2]

@Jacqualin jasmin See if you are also facing the warning defined in link [1]. If yes, try the same resolution.

[1] https://community.hortonworks.com/articles/11805/how-to-solve-ambari-metrics-corrupted-data.html

ref : https://cwiki.apache.org/confluence/display/AMBARI/Cleaning+up+Ambari+Metrics+System+Data

[#3]

Hi Pankaj,

I implemented your suggested steps. Still no luck. Still, Ambari Metric Collector is going down with below errors

2017-12-27 22:30:21,221 ERROR org.apache.hadoop.yarn.server.applicationhistoryservice.metrics.timeline.PhoenixHBaseAccessor: HBaseAccessor getConnection failed after 10 attempts 2017-12-27 22:30:21,221 ERROR org.apache.hadoop.yarn.server.applicationhistoryservice.metrics.timeline.PhoenixHBaseAccessor: Error creating Metrics Schema in HBase using Phoenix. org.apache.phoenix.exception.PhoenixIOException: SYSTEM.CATALOG _____________________________

... 31 more 2017-12-27 22:30:21,222 INFO org.apache.hadoop.service.AbstractService: Service org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryServer failed in state INITED; cause: org.apache.hadoop.yarn.server.applicationhistoryservice.metrics.timeline.MetricsSystemInitializationException: Error creating Metrics Schema in HBase using Phoenix. org.apache.hadoop.yarn.server.applicationhistoryservice.metrics.timeline.MetricsSystemInitializationException: Error creating Metrics Schema in HBase using Phoenix.

______________________________

2017-12-27 22:30:21,220 WARN org.apache.hadoop.yarn.server.applicationhistoryservice.metrics.timeline.query.DefaultPhoenixDataSource: Unable to connect to HBase store using Phoenix. org.apache.phoenix.exception.PhoenixIOException: SYSTEM.CATALOG

Regards

JJ

[#4]

Hi Guys... same problem here.

After all steps to resolve, no luck.

Can you provide any further guidance ?

Thank you in advance

/J

,

Hi same error too, after all suggested steps. What to do ?
Greetings /J

[#5]

Hi,

solved this in Ambari 2.4.2 by Cleaning Up the Data after backing up. After

su ams -c '/usr/lib/ams-hbase/bin/hbase --config /etc/ams-hbase/conf clean --cleanAll

ambari-metrics-collector restart - it works . It started Metrics Collector from scratch

[#6]

For some reason

https://cwiki.apache.org/confluence/display/AMBARI/Cleaning+up+Ambari+Metrics+System+Data didn't work in my case.

Only this method was help:

/usr/lib/ams-hbase/hbase --config /etc/ams-hbase/conf clean --cleanAll