Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

FATAL org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryServer: Error starting ApplicationHistoryServer

FATAL org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryServer: Error starting ApplicationHistoryServer

New Contributor

Hi Gurus.

Our Ambari Metrics collector goes down. Our is Kerberos env. Ambari Version is :Version2.4.2.0

i see below errors

in ambari-metrics-collector.log i see belwo error

FATAL org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryServer: Error starting ApplicationHistoryServer org.apache.hadoop.yarn.server.applicationhistoryservice.metrics.timeline.MetricsSystemInitializationException: Error creating Metrics Schema in HBase using Phoenix. Caused by: org.apache.phoenix.exception.PhoenixIOException: SYSTEM.CATALOG Caused by: org.apache.hadoop.hbase.TableNotFoundException: SYSTEM.CATALOG

in hbase-ams-master-xxxxxxxxxx.log

2017-12-23 07:40:57,717 ERROR [xxxxxx,61300,1513971592600_ChoreService_1] master.BackupLogCleaner: Failed to get hbase:backup table, therefore will keep all files org.apache.hadoop.hbase.TableNotFoundException: hbase:backup

I implemented below procedure but no luck

https://community.hortonworks.com/questions/73577/problem-with-ambari-metrics.html

1) Turn on Maintenance mode

2) Stop Ambari Metrics

3) hadoop fs -rmr /ams/hbase/*

4) rm -rf /var/lib/ambari-metrics-collector/hbase-tmp/*

5) [zk: localhost:2181(CONNECTED) 0] ls /

[zk: localhost:2181(CONNECTED) 1] rmr /ams-hbase-unsecure

6) Start Ambari Metrics

7) Turn off Maintenance mode

My setting are :

hbase.rootdir = hdfs://hpdprod:8020/user/ams/hbase

hbase.cluster.distributed=true

timeline.metrics.service.operation.mode=distributed

Any help is really appreciate

Thanks

JJ

5 REPLIES 5
Highlighted

Re: FATAL org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryServer: Error starting ApplicationHistoryServer

New Contributor

Re: FATAL org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryServer: Error starting ApplicationHistoryServer

New Contributor

Hi Pankaj,

I implemented your suggested steps. Still no luck. Still, Ambari Metric Collector is going down with below errors

2017-12-27 22:30:21,221 ERROR org.apache.hadoop.yarn.server.applicationhistoryservice.metrics.timeline.PhoenixHBaseAccessor: HBaseAccessor getConnection failed after 10 attempts 2017-12-27 22:30:21,221 ERROR org.apache.hadoop.yarn.server.applicationhistoryservice.metrics.timeline.PhoenixHBaseAccessor: Error creating Metrics Schema in HBase using Phoenix. org.apache.phoenix.exception.PhoenixIOException: SYSTEM.CATALOG _____________________________

... 31 more 2017-12-27 22:30:21,222 INFO org.apache.hadoop.service.AbstractService: Service org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryServer failed in state INITED; cause: org.apache.hadoop.yarn.server.applicationhistoryservice.metrics.timeline.MetricsSystemInitializationException: Error creating Metrics Schema in HBase using Phoenix. org.apache.hadoop.yarn.server.applicationhistoryservice.metrics.timeline.MetricsSystemInitializationException: Error creating Metrics Schema in HBase using Phoenix.

______________________________

2017-12-27 22:30:21,220 WARN org.apache.hadoop.yarn.server.applicationhistoryservice.metrics.timeline.query.DefaultPhoenixDataSource: Unable to connect to HBase store using Phoenix. org.apache.phoenix.exception.PhoenixIOException: SYSTEM.CATALOG

Regards

JJ

Re: FATAL org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryServer: Error starting ApplicationHistoryServer

New Contributor

Hi Guys... same problem here.

After all steps to resolve, no luck.

Can you provide any further guidance ?

Thank you in advance

/J

,

Hi same error too, after all suggested steps. What to do ?
Greetings /J

Re: FATAL org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryServer: Error starting ApplicationHistoryServer

New Contributor

Hi,

solved this in Ambari 2.4.2 by Cleaning Up the Data after backing up. After

su ams -c '/usr/lib/ams-hbase/bin/hbase --config /etc/ams-hbase/conf clean --cleanAll

ambari-metrics-collector restart - it works . It started Metrics Collector from scratch

Re: FATAL org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryServer: Error starting ApplicationHistoryServer

Expert Contributor

For some reason

https://cwiki.apache.org/confluence/display/AMBARI/Cleaning+up+Ambari+Metrics+System+Data didn't work in my case.

Only this method was help:

/usr/lib/ams-hbase/hbase --config /etc/ams-hbase/conf clean --cleanAll
Don't have an account?
Coming from Hortonworks? Activate your account here