I am currently running HDP version 2.4 which contains hbase version 1.1.2. I'm hitting the following bug which is causing empty results to be returned from hbase: https://issues.apache.org/jira/browse/HBASE-15378 Ambari currently only supports upgrading the entire stack and not individual components, but even so it looks like HDP 2.5 is still shipping with hbase version 1.1.2: https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.5.0/bk_release-notes/content/ch_relnotes_v250.html I'd like to upgrade hbase to 1.1.4 to fix the issue above. Has anyone done this? Will it break something else in the cluster/ambari? Is anyone aware of a procedure to do this? ... View more

On start, we get spam in the logs for the following Warning: 2017-01-04 21:01:11,912 WARN org.apache.zookeeper.ClientCnxn: Session 0x1596b43b51c0005 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1125) 2017-01-04 21:01:12,780 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server localhost/127.0.0.1:61181. Will not attempt to authenticate using SASL (unknown error) 2017-01-04 21:01:12,781 WARN org.apache.zookeeper.ClientCnxn: Session 0x1596b43b51c0001 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1125) 2017-01-04 21:01:13,237 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server localhost/127.0.0.1:61181. Will not attempt to authenticate using SASL (unknown error) and eventually the following error: 2017-01-04 21:01:18,552 ERROR org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher: hconnection-0x77a98a6a-0x1596b43b51c0005, quorum=localhost:61181, baseZNode=/falcon-ams-hbase Received u nexpected KeeperException, re-throwing exception org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /falcon-ams-hbase ZK health seems ok from an ambari perspective. Logging in via the ZK CLI, I don't see /facon-ams-hbase: [zk: localhost:2181(CONNECTED) 0] ls / [zookeeper, falcon-hbase] Is this expected? Any idea on how to recover this? ... View more

After one of our region servers went down (down for about 30 minutes), we are getting quite a bit of this sort of spam in the hbase master log: 2016-12-05 21:20:30,036 WARN [PriorityRpcServer.handler=14,queue=0,port=16000] master.ServerManager: RegionServer <hostname>,16020,1472677155412 indicates a last flushed sequence id (8454457) that is less than the previous last flushed sequence id (8454459) for region metrics,AAAAAfUA54Mq0nkzloefXqeNG0Skrwr6^@^@^@^@XB�5,1480811500859.51b6b955b6638ffe2e60b562385a909b. Ignoring. Is there a way to correct this? Should I be concerned? ... View more