question Re: cant start ambari collector in Support Questions

cant start ambari collector

mike_bronson7 — Sun, 11 Mar 2018 22:04:42 GMT

we cant start the ambari collector

from the logs we got the following:

what chould be the problem ?


2018-03-11 15:50:20,249 ERROR org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher: hconnection-0x4bdeaabb-0x16215bf27850000, quorum=master02.sys673.com:61181, baseZNode=/ams-hbase-unsecure Received unexpected KeeperException, re-throwing exception
org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /ams-hbase-unsecure/meta-region-server
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
        at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1212)
        at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getData(RecoverableZooKeeper.java:354)
        at org.apache.hadoop.hbase.zookeeper.ZKUtil.getData(ZKUtil.java:622)
        at org.apache.hadoop.hbase.zookeeper.MetaTableLocator.getMetaRegionState(MetaTableLocator.java:491)
        at org.apache.hadoop.hbase.zookeeper.MetaTableLocator.getMetaRegionLocation(MetaTableLocator.java:172)
        at org.apache.hadoop.hbase.zookeeper.MetaTableLocator.blockUntilAvailable(MetaTableLocator.java:611)
        at org.apache.hadoop.hbase.zookeeper.MetaTableLocator.blockUntilAvailable(MetaTableLocator.java:592)
        at org.apache.hadoop.hbase.zookeeper.MetaTableLocator.blockUntilAvailable(MetaTableLocator.java:565)
        at org.apache.hadoop.hbase.client.ZooKeeperRegistry.getMetaRegionLocation(ZooKeeperRegistry.java:61)
        at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateMeta(ConnectionManager.java:1195)
        at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1162)
        at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.relocateRegion(ConnectionManager.java:1136)
        at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:298)
        at org.apache.hadoop.hbase.client.ScannerCallable.prepare(ScannerCallable.java:151)
        at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.prepare(ScannerCallableWithReplicas.java:376)
        at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:124)
        at org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture.run(ResultBoundedCompletionService.java:65)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)

grep error  ambari-metrics-collector.log
2018-03-11 14:59:04,182 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server master02.sys673.com/130.14.52.8:61181. Will not attempt to authenticate using SASL (unknown error)
2018-03-11 14:59:04,185 WARN org.apache.zookeeper.ClientCnxn: Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
2018-03-11 14:59:04,458 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server master02.sys673.com/130.14.52.8:61181. Will not attempt to authenticate using SASL (unknown error)
2018-03-11 14:59:04,459 WARN org.apache.zookeeper.ClientCnxn: Session 0x162158dcd9a0001 for server null, unexpected error, closing socket connection and attempting reconnect
2018-03-11 14:59:05,286 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server master02.sys673.com/130.14.52.8:61181. Will not attempt to authenticate using SASL (unknown error)
2018-03-11 14:59:05,286 WARN org.apache.zookeeper.ClientCnxn: Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
2018-03-11 14:59:05,812 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server master02.sys673.com/130.14.52.8:61181. Will not attempt to authenticate using SASL (unknown error)
2018-03-11 14:59:05,813 WARN org.apache.zookeeper.ClientCnxn: Session 0x162158dcd9a0001 for server null, unexpected error, closing socket connection and attempting reconnect
2018-03-11 14:59:06,388 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server master02.sys673.com/130.14.52.8:61181. Will not attempt to authenticate using SASL (unknown error)
2018-03-11 14:59:06,388 WARN org.apache.zookeeper.ClientCnxn: Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
2018-03-11 14:59:07,197 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server master02.sys673.com/130.14.52.8:61181. Will not attempt to authenticate using SASL (unknown error)
2018-03-11 14:59:07,197 WARN org.apache.zookeeper.ClientCnxn: Session 0x162158dcd9a0001 for server null, unexpected error, closing socket connection and attempting reconnect
2018-03-11 14:59:07,489 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server master02.sys673.com/130.14.52.8:61181. Will not attempt to authenticate using SASL (unknown error)
2018-03-11 14:59:07,490 WARN org.apache.zookeeper.ClientCnxn: Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
2018-03-11 14:59:08,591 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server master02.sys673.com/130.14.52.8:61181. Will not attempt to authenticate using SASL (unknown error)
2018-03-11 14:59:08,592 WARN org.apache.zookeeper.ClientCnxn: Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect

Re: cant start ambari collector

jsensharma — Mon, 12 Mar 2018 03:25:37 GMT

@Michael Bronson

Looks like AMS collector is not able to start properly and hence the zookeeper is showing connection Loss while finding the znode.

Sometimes it happens if the AMS collector is not tuned properly. Specially when the Heap Settings are not done properly according to the number of nodes present in the cluster. So can you please refer to the following doc to check if the Heap settings are fine and according to the cluster:

- https://cwiki.apache.org/confluence/display/AMBARI/Configurations+-+Tuning

- Sometimes cleaning up the "hbase.zookeeper.property.dataDir" and '${hbase.tmp.dir}/phoenix-spool temp directory content helps in making the zookeeper and spool dir data (which are temporary dirs).

Re: cant start ambari collector

mike_bronson7 — Mon, 12 Mar 2018 03:52:46 GMT

@Jay the link have a lot of info , can you please gives the important notes from the link?

Re: cant start ambari collector

mike_bronson7 — Mon, 12 Mar 2018 03:53:43 GMT

regarding to ${hbase.tmp.dir}/phoenix-spool temp directory , what is the path on ambari server?

Re: cant start ambari collector

jsensharma — Mon, 12 Mar 2018 05:00:17 GMT

@Michael Bronson

In the mentioned tuning link we can start from the "Suggested Memory settings" section (as most of the time incorrect memory tuning causes AMS collector to go down) to see if the memory is allocated properly to the AMS system accordingly.

The Data Dirs are as following :

hbase.tmp.dir=/var/lib/ambari-metrics-collector/hbase-tmp
hbase.zookeeper.property.dataDir=${hbase.tmp.dir}/zookeeper
phoenix.spool.directory=${hbase.tmp.dir}/phoenix-spool

Re: cant start ambari collector

mike_bronson7 — Mon, 12 Mar 2018 12:18:40 GMT

@Jay cluster size ( as appears in the link ) is the number of workers machines?

Re: cant start ambari collector

mike_bronson7 — Mon, 12 Mar 2018 13:07:49 GMT

@jay we clean both folder and start again the metrics but we the same isshue

Re: cant start ambari collector

jsensharma — Mon, 12 Mar 2018 13:17:36 GMT

@Michael Bronson

The Cluster Size means the total number of hosts present in the cluster. So if the number of cluster host increases then the ambari metrics data will grow accordingly and hence we will need more data aggregation and data so in that case we will need to have the heap size set accordingly. A rough calculation of heap settings is mentioned in the above tuning doc.

- In this case as the AMS is going down very frequently then we should check the following things:

1. Usually it happens when the Heap is not set properly. Increasing the memory works in many cases. So please try to Increased heap memory for AMS collector and hbase.

2. We need to know how many nodes are there in the cluster to see if the current tuning is OK or not?

3. We can check the current AMS tunning by looking at the following logs: (So if you can attach them then it will be good to review them)

# tar czhfv ./amsc_etc_$(hostname)_$(date +"%Y%m%d%H%M%S").tar.gz /etc/ambari-metrics-collector/ 
# tar czhfv ./amsh_etc_$(hostname)_$(date +"%Y%m%d%H%M%S").tar.gz /etc/ams-hbase/

4. The complete collector logs along with the following from location "/var/log/ambari-metrics-collector" on AMS host.

ambari-metrics-collector.log      (latest one)
hbase-ams-master-*.log    (latest one)
collector-gc.log (latest)
gc.log   (latest)

Re: cant start ambari collector

mike_bronson7 — Mon, 12 Mar 2018 13:27:01 GMT

@Jay we have in the cluster 3 master machines and 4 workers machine so total hosts are - 7 , so from the table we need to select the first table - (1 - 10) , an I correct ?

Re: cant start ambari collector

jsensharma — Sun, 18 Aug 2019 08:31:26 GMT

@Michael Bronson

Yes, correct, this one should be good.

So please check if the AMS collector host has enough free memory or not?

# free -m

Re: cant start ambari collector

mike_bronson7 — Mon, 12 Mar 2018 16:20:03 GMT

@Jay you mention to "Increased heap memory for AMS collector and hbase." , please let me know what are the variable that we need to increase ? ( they from ambari GUI ) ?

Re: cant start ambari collector

mike_bronson7 — Mon, 12 Mar 2018 16:34:19 GMT

@Jay , thank you so much the "Suggested Memory settings" new values solved the problem , and now metrics collector is up , and thank you for the time that you put on this case , we are very appreciate this

Re: cant start ambari collector

mike_bronson7 — Mon, 12 Mar 2018 19:28:40 GMT


ambari-metrics-collector.log
java.net.ConnectException: Connection refused
        at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
        at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
        at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1141)
2018-03-12 08:54:14,696 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server master02.sys3423.com/43.25.76.98:61181. Will not attempt to authenticate using SASL (unknown error)
2018-03-12 08:54:14,697 WARN org.apache.zookeeper.ClientCnxn: Session 0x1621966a8020001 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
        at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
        at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
        at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1141)
2018-03-12 08:54:14,797 WARN org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Possibly transient ZooKeeper, quorum=master02.sys3423.com:61181, exception=org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /ams-hbase-unsecure/meta-region-server
2018-03-12 08:54:15,859 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server master02.sys3423.com/43.25.76.98:61181. Will not attempt to authenticate using SASL (unknown error)
2018-03-12 08:54:15,860 WARN org.apache.zookeeper.ClientCnxn: Session 0x1621966a8020001 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
        at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
        at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
        at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1141)
2018-03-12 08:54:17,363 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server master02.sys3423.com/43.25.76.98:61181. Will not attempt to authenticate using SASL (unknown error)
2018-03-12 08:54:17,364 WARN org.apache.zookeeper.ClientCnxn: Session 0x1621966a8020001 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
        at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
        at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
        at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1141)
2018-03-12 08:54:19,037 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server master02.sys3423.com/43.25.76.98:61181. Will not attempt to authenticate using SASL (unknown error)
2018-03-12 08:54:19,037 WARN org.apache.zookeeper.ClientCnxn: Session 0x1621966a8020001 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
        at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
        at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
        at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1141)

Re: cant start ambari collector

jsensharma — Tue, 13 Mar 2018 06:02:13 GMT

@Michael Bronson

The latest error indicates that AMS collector is still going down. But after running for some time. Means it requires further tuning.

2018-03-12 08:54:19,037 WARN org.apache.zookeeper.ClientCnxn: Session 0x1621966a8020001 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
        at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)

So can you please restart the AMS collector and then check the heap usage after some time to see if it is reaching the maximum limit?

# $JAVA_HOME/bin/jmap --heap $PID_AMS

Re: cant start ambari collector

mike_bronson7 — Tue, 13 Mar 2018 12:55:24 GMT

@Jay after we fine-tune the parameters (yesterday), now every thing is ok and metrics collector is up , in spite all this do you want to restart the AMS collector? anyway?

Re: cant start ambari collector

jsensharma — Tue, 13 Mar 2018 13:49:25 GMT

@Michael Bronson

No if the AMS is running fine for now then we do not need to restart it. We can keep monitoring it for some time to see if everything is going fine.