AMBARI METRICS restart randomly

[#1]

Hi,

I built a new standalone hdp 2.3. This standalone are not yet sollicited by application but the ambari metrics service shutdown and restart for no reason. It can happen twice/three or four times a week.

This is the logs of my ambari-metrics-collector.log:

2016-03-14 01:29:07,166 ERROR org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryServer: RECEIVED SIGNAL 15: SIGTERM 2016-03-14 01:29:07,169 INFO org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation: Closing master protocol: MasterService 2016-03-14 01:29:07,182 INFO org.mortbay.log: Stopped [email protected]:6188 2016-03-14 01:29:07,207 INFO org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation: Closing zookeeper sessionid=0x1536be5998a0001 2016-03-14 01:29:07,208 INFO org.apache.zookeeper.ZooKeeper: Session: 0x1536be5998a0001 closed 2016-03-14 01:29:07,208 INFO org.apache.zookeeper.ClientCnxn: EventThread shut down 2016-03-14 01:29:07,286 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping phoenix metrics system... 2016-03-14 01:29:07,289 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: phoenix metrics system stopped. 2016-03-14 01:29:07,289 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: phoenix metrics system shutdown complete. 2016-03-14 01:29:07,290 INFO org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryManagerImpl: Stopping ApplicationHistory 2016-03-14 01:29:07,290 INFO org.apache.hadoop.ipc.Server: Stopping server on 60200 2016-03-14 01:29:07,294 INFO org.apache.hadoop.ipc.Server: Stopping IPC Server Responder 2016-03-14 01:29:07,294 INFO org.apache.hadoop.ipc.Server: Stopping IPC Server listener on 60200 2016-03-14 01:29:07,295 INFO org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryServer: SHUTDOWN_MSG:

Do you know the reason of this restarts?

Thanks,

Gauthier

[#2]

@GAUTHIER CHRETIEN

There is a patch out check this jira

[#3]

Thanks for your reply.

But I run ambari-metrics 2.2 version:

rpm -qa | grep ambari-metrics

ambari-metrics-monitor-2.2.0.0-1310.x86_64

ambari-metrics-collector-2.2.0.0-1310.x86_64

ambari-metrics-hadoop-sink-2.2.0.0-1310.x86_64

Gauthier

[#4]

Could be memory issue.

Can you check /var/log/messages for Kernel OOM killer.

Also please give details of your environment. (OS, RAM, nodes etc)

[#5]

I have not memory issue ( No OOM Killer...).

I run a centos6 standalone server with 125G RAM.

My ams parameters values:

metrics_collector_heapsize: 1024MB

hbase_master_heapsize: 2048MB

hbase_regionserver_heapsize: 2048MB

[#6 · Accepted solution]

Could be because of https://issues.apache.org/jira/browse/AMBARI-15492. This has been fixed in the next Ambari release (2.2.2).

For a workaround please try commenting these 2 properties in /etc/ambari-server/conf/ambari.properties

#recovery.enabled_components=METRICS_COLLECTOR

#recovery.type=AUTO_START

Restart Ambari Server.

[#7]

It does not work if i comment those lines.

My ambari server still stop and now it does not start.

[#8]

Ok i restart ambari-agent too it seems good now. No alerts for 2 days.keep in touch ;).

Thanks

[#9]

Hi Gauthier,

Do you have any news about that issue? Does it work properly now?

Thanks

[#10]

Hi Vincent,

It works 😉 we have no more alerts from our server.