Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Kerberized HDP 2.4 - Ambari Metric (v 2.2.2.0) shutting down

Kerberized HDP 2.4 - Ambari Metric (v 2.2.2.0) shutting down

Expert Contributor

Hi All - i've a Kerberized HDP 2.4 and have Ambari v2.2.2.0 ..

Ambari-metrics keeps shutting down .. the error (in file -

/var/log/ambari-metrics-collector/ambari-metrics-collector.log) is as shown below ..

Any ideas on what needs to be done for this ?

------------------------------------------------------------------------------------------

2016-12-09 22:46:25,428 INFO org.apache.zookeeper.ZooKeeper: Initiating client connection, connectString=localhost:61181 sessionTimeout=120000 watcher=hconnection-0x61bbbcfe0x0, quorum=localhost:61181, baseZNode=/ams-hbase-secure 2016-12-09 22:46:25,435 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server localhost.localdomain/127.0.0.1:61181. Will not attempt to authenticate using SASL (unknown error) 2016-12-09 22:46:25,436 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to localhost.localdomain/127.0.0.1:61181, initiating session 2016-12-09 22:46:25,448 INFO org.apache.zookeeper.ClientCnxn: Session establishment complete on server localhost.localdomain/127.0.0.1:61181, sessionid = 0x158e5c405310003, negotiated timeout = 120000 2016-12-09 22:46:25,450 INFO org.apache.hadoop.hbase.client.ZooKeeperRegistry: ClusterId read in ZooKeeper is null 2016-12-09 22:46:25,642 WARN org.apache.hadoop.hbase.io.util.HeapMemorySizeUtil: hbase.regionserver.global.memstore.upperLimit is deprecated by hbase.regionserver.global.memstore.size 2016-12-09 22:46:29,200 INFO org.apache.zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x158e5c405310003, likely server has closed socket, closing socket connection and attempting reconnect 2016-12-09 22:46:29,202 INFO org.apache.zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x158e5c405310000, likely server has closed socket, closing socket connection and attempting reconnect 2016-12-09 22:46:30,645 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server localhost.localdomain/127.0.0.1:61181. Will not attempt to authenticate using SASL (unknown error) 2016-12-09 22:46:30,646 WARN org.apache.zookeeper.ClientCnxn: Session 0x158e5c405310003 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:744) at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1125) 2016-12-09 22:46:30,756 WARN org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Possibly transient ZooKeeper, quorum=localhost:61181, exception=org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /ams-hbase-secure 2016-12-09 22:46:31,012 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server localhost.localdomain/127.0.0.1:61181. Will not attempt to authenticate using SASL (unknown error) 2016-12-09 22:46:31,013 WARN org.apache.zookeeper.ClientCnxn: Session 0x158e5c405310000 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:744) at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1125)

------------

regds,

Karan Alang

17 REPLIES 17
Highlighted

Re: Kerberized HDP 2.4 - Ambari Metric (v 2.2.2.0) shutting down

The connection to ZK was refused.

Please make sure that AMS has the correct user and configs for ZK, especially for the quorum hostnames and port. In ams-site config, check cluster.zookeeper.quorum and cluster.zookeeper.property.clientPort

Highlighted

Re: Kerberized HDP 2.4 - Ambari Metric (v 2.2.2.0) shutting down

Expert Contributor

@Alejandro Fernandez - i just checked the ams-site.xml (location - /var/lib/ambari-server/resources/common-services/AMBARI_METRICS/0.1.0/configuration) , and don't see properties -

cluster.zookeeper.quorum, cluster.zookeeper.property.clientPort in the file .. attaching the file.

Any ideas on what needs to be done ?

Thanks !

ams-site.xml

Pls note - i've just upgraded Amabari from 2.2. to 2.2.2.0 & installed Grafana (which comed bundled with Amabari 2.2.2.0)

Highlighted

Re: Kerberized HDP 2.4 - Ambari Metric (v 2.2.2.0) shutting down

Expert Contributor

@Alejandro Fernandez - i added the properties you mentioned in ams-site.xml

cluster.zookeeper.quorum = sandbox.hortonworks.com:2181

cluster.zookeeper.property.clientPort = 2181

The Ambari metrics starts up, but then it stops after a few mins.

Any ideas ?

Highlighted

Re: Kerberized HDP 2.4 - Ambari Metric (v 2.2.2.0) shutting down

Expert Contributor

@Aravindan Vijayan, @Kuldeep Kulkarni, @Neeraj Sabharwal, - looping you into this..

I seem to be having issue wrt Ambari Metric - it is starting up, but then stopping..

Appreciate if you could provide inputs on this ?

thanks !

Highlighted

Re: Kerberized HDP 2.4 - Ambari Metric (v 2.2.2.0) shutting down

Expert Contributor

@Karan Alang In 2.2.2, AMS in embedded mode uses its own local zookeeper (port 61181). Please have the zookeeper quorum and clientPort config values as default. Try Stopping the metrics collector, clearing out the /var/lib/ambari-metrics-collector/hbase-tmp/zookeeper ({ams-hbase-site:hbase.tmp.dir}/zookeeper), and then Starting it again.

If it does not work, please attach the entire metrics collector and ams-hbase master log. They should be present in /var/log/ambari-metrics-collector/.

Highlighted

Re: Kerberized HDP 2.4 - Ambari Metric (v 2.2.2.0) shutting down

Explorer

Worked like a charm,Thanks Vijayan :)

Highlighted

Re: Kerberized HDP 2.4 - Ambari Metric (v 2.2.2.0) shutting down

Explorer

Vijayan-do you know why we need to cleanup manually?

Highlighted

Re: Kerberized HDP 2.4 - Ambari Metric (v 2.2.2.0) shutting down

Expert Contributor

@Aravindan Vijayan,

thanks for response .. i've the zookeeper quorom & client port set to default .. also, cleared out /var/lib/ambari-metrics-collector/hbase-tmp/zookeeper ({ams-hbase-site:hbase.tmp.dir}/zookeeper), and started again, however is still the same .. attaching the log fies, also /etc/ambari-metrics-collector/conf/ams-site.xml.

Pls take a look & let me know changes to fix the issue.

ams-site.xml

hdp24-hbase-ams-master-sandboxhortonworkscom.txt

hdp24-ambari-metrics-collector.txt

Thanks !

Highlighted

Re: Kerberized HDP 2.4 - Ambari Metric (v 2.2.2.0) shutting down

Expert Contributor

@Aravindan Vijayan -

also attaching - hbase-site.xml/etc/ams-hbase/conf/hbase-site.xml

Don't have an account?
Coming from Hortonworks? Activate your account here