Support Questions
Find answers, ask questions, and share your expertise

Hortonworks Sandbox v2.4 - Ambari Metrics Collector stopping randomly

New Contributor

Currently I'm experiencing an issue with the AMC in the Hortonworks Sandbox v2.4. I've only just managed to get the machine I'm running the VM on working and I've noticed that the Ambari Metrics periodically stop anywhere from 10 minutes to an hour after opening Ambari. If I restart the Metrics Collector it starts back up again but I want to figure out what the root cause is and resolve it permanently if possible.

See below for the log file output from the latest ambari-metrics-collector.log;

2016-07-13 15:44:09,141 ERROR TimelineClusterAggregatorSecond: Exception during aggregating metrics.
org.apache.phoenix.exception.PhoenixIOException: hconnection-0x2b2b7313 closed
        at org.apache.phoenix.util.ServerUtil.parseServerException(ServerUtil.java:108)
        at org.apache.phoenix.query.ConnectionQueryServicesImpl.metaDataCoprocessorExec(ConnectionQueryServicesImpl.java:1039)
        at org.apache.phoenix.query.ConnectionQueryServicesImpl.metaDataCoprocessorExec(ConnectionQueryServicesImpl.java:1002)
        at org.apache.phoenix.query.ConnectionQueryServicesImpl.getTable(ConnectionQueryServicesImpl.java:1235)
        at org.apache.phoenix.schema.MetaDataClient.updateCache(MetaDataClient.java:436)
        at org.apache.phoenix.schema.MetaDataClient.updateCache(MetaDataClient.java:379)
        at org.apache.phoenix.schema.MetaDataClient.updateCache(MetaDataClient.java:375)
        at org.apache.phoenix.compile.FromCompiler$BaseColumnResolver.createTableRef(FromCompiler.java:413)
        at org.apache.phoenix.compile.FromCompiler$SingleTableColumnResolver.<init>(FromCompiler.java:288)
        at org.apache.phoenix.compile.FromCompiler.getResolverForQuery(FromCompiler.java:189)
        at org.apache.phoenix.jdbc.PhoenixStatement$ExecutableSelectStatement.compilePlan(PhoenixStatement.java:359)
        at org.apache.phoenix.jdbc.PhoenixStatement$ExecutableSelectStatement.compilePlan(PhoenixStatement.java:340)
        at org.apache.phoenix.jdbc.PhoenixStatement$1.call(PhoenixStatement.java:248)
        at org.apache.phoenix.jdbc.PhoenixStatement$1.call(PhoenixStatement.java:243)
        at org.apache.phoenix.call.CallRunner.run(CallRunner.java:53)
        at org.apache.phoenix.jdbc.PhoenixStatement.executeQuery(PhoenixStatement.java:242)
        at org.apache.phoenix.jdbc.PhoenixPreparedStatement.executeQuery(PhoenixPreparedStatement.java:186)
        at org.apache.hadoop.yarn.server.applicationhistoryservice.metrics.timeline.aggregators.AbstractTimelineAggregator.doWork(AbstractTimelineAggregator.java:253)
        at org.apache.hadoop.yarn.server.applicationhistoryservice.metrics.timeline.aggregators.AbstractTimelineAggregator.runOnce(AbstractTimelineAggregator.java:129)
        at org.apache.hadoop.yarn.server.applicationhistoryservice.metrics.timeline.aggregators.AbstractTimelineAggregator.run(AbstractTimelineAggregator.java:105)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: hconnection-0x2b2b7313 closed
        at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1153)
        at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.relocateRegion(ConnectionManager.java:1133)
        at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegionInMeta(ConnectionManager.java:1338)
        at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1162)
        at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1146)
        at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1103)
        at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.getRegionLocation(ConnectionManager.java:938)
        at org.apache.hadoop.hbase.client.HRegionLocator.getRegionLocation(HRegionLocator.java:83)
        at org.apache.hadoop.hbase.client.HTable.getRegionLocation(HTable.java:504)
        at org.apache.hadoop.hbase.client.HTable.getKeysAndRegionsInRange(HTable.java:747)
        at org.apache.hadoop.hbase.client.HTable.getKeysAndRegionsInRange(HTable.java:717)
        at org.apache.hadoop.hbase.client.HTable.getStartKeysInRange(HTable.java:1784)
        at org.apache.hadoop.hbase.client.HTable.coprocessorService(HTable.java:1739)
        at org.apache.hadoop.hbase.client.HTable.coprocessorService(HTable.java:1719)
        at org.apache.phoenix.query.ConnectionQueryServicesImpl.metaDataCoprocessorExec(ConnectionQueryServicesImpl.java:1022)
        ... 19 more
2016-07-13 15:44:09,141 INFO org.apache.zookeeper.ZooKeeper: Session: 0x155e47b94aa0001 closed
2016-07-13 15:44:09,141 INFO org.apache.zookeeper.ClientCnxn: EventThread shut down
2016-07-13 15:44:09,142 INFO TimelineClusterAggregatorSecond: End aggregation cycle @ Wed Jul 13 15:44:09 UTC 2016

It looks like a ZooKeeper-related issue but as I'm only just getting the hang of Ambari I'm not knowledgeable enough to resolve this on my own. Can anyone assist me please? πŸ™‚

8 REPLIES 8

Super Guru

@Fraser Mills

Can you please check if you have below properties set in /etc/ambari-server/conf/ambari.properites

recovery.enabled_components=METRICS_COLLECTOR 
recovery.type=AUTO_START

If yes, comment/remove them and restart ambari-server followed by restart ambari-metrics collector.

New Contributor

Hi @Kuldeep Kulkarni, I just did this and restarted the Ambari Server and AMS processes. I noticed from the CLI that I was getting this error when attempting to start AMS;

2016-07-14 08:50:00,584 ERROR [main] zookeeper.ZooKeeperWatcher: hconnection-0x2971dd6f0x0, quorum=localhost:61181, baseZNode=/ams-hbase-unsecure Received unexpected KeeperException, re-throwing exception
org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /ams-hbase-unsecure
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
        at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1045)
        at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.exists(RecoverableZooKeeper.java:221)
        at org.apache.hadoop.hbase.zookeeper.ZKUtil.checkExists(ZKUtil.java:417)
        at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.checkIfBaseNodeAvailable(ConnectionManager.java:902)
        at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.access$400(ConnectionManager.java:552)
        at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$StubMaker.makeStubNoRetries(ConnectionManager.java:1490)
        at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$StubMaker.makeStub(ConnectionManager.java:1531)
        at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$MasterServiceStubMaker.makeStub(ConnectionManager.java:1560)
        at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.getKeepAliveMasterService(ConnectionManager.java:1711)
        at org.apache.hadoop.hbase.client.MasterCallable.prepare(MasterCallable.java:38)
        at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:124)
        at org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:4082)
        at org.apache.hadoop.hbase.client.HBaseAdmin.listTables(HBaseAdmin.java:420)
        at org.apache.hadoop.hbase.client.HBaseAdmin.listTables(HBaseAdmin.java:414)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at org.jruby.javasupport.JavaMethod.invokeDirectWithExceptionHandling(JavaMethod.java:450)
        at org.jruby.javasupport.JavaMethod.invokeDirect(JavaMethod.java:311)
        at org.jruby.java.invokers.InstanceMethodInvoker.call(InstanceMethodInvoker.java:59)
        at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:312)
        at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:169)
        at org.jruby.ast.CallOneArgNode.interpret(CallOneArgNode.java:57)
        at org.jruby.ast.CallNoArgBlockNode.interpret(CallNoArgBlockNode.java:60)
        at org.jruby.ast.NewlineNode.interpret(NewlineNode.java:104)
        at org.jruby.evaluator.ASTInterpreter.INTERPRET_METHOD(ASTInterpreter.java:74)
        at org.jruby.internal.runtime.methods.InterpretedMethod.call(InterpretedMethod.java:190)
        at org.jruby.internal.runtime.methods.DefaultMethod.call(DefaultMethod.java:199)
        at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:312)
        at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:169)
        at org.jruby.ast.CallOneArgNode.interpret(CallOneArgNode.java:57)
        at org.jruby.ast.LocalAsgnNode.interpret(LocalAsgnNode.java:123)
        at org.jruby.ast.NewlineNode.interpret(NewlineNode.java:104)
        at org.jruby.ast.BlockNode.interpret(BlockNode.java:71)
        at org.jruby.evaluator.ASTInterpreter.INTERPRET_METHOD(ASTInterpreter.java:74)
        at org.jruby.internal.runtime.methods.InterpretedMethod.call(InterpretedMethod.java:169)
        at org.jruby.internal.runtime.methods.DefaultMethod.call(DefaultMethod.java:191)
        at org.jruby.RubyClass.finvoke(RubyClass.java:552)
        at org.jruby.RubyBasicObject.send(RubyBasicObject.java:2777)
        at org.jruby.RubyKernel.send(RubyKernel.java:2105)
        at org.jruby.RubyKernel$s$send.call(RubyKernel$s$send.gen:65535)
        at org.jruby.internal.runtime.methods.JavaMethod$JavaMethodZeroOrOneOrNBlock.call(JavaMethod.java:266)
        at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:312)
        at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:169)
        at org.jruby.ast.FCallSpecialArgNode.interpret(FCallSpecialArgNode.java:39)
        at org.jruby.ast.NewlineNode.interpret(NewlineNode.java:104)
        at org.jruby.evaluator.ASTInterpreter.INTERPRET_BLOCK(ASTInterpreter.java:111)
        at org.jruby.runtime.InterpretedBlock.evalBlockBody(InterpretedBlock.java:374)
        at org.jruby.runtime.InterpretedBlock.yield(InterpretedBlock.java:295)
        at org.jruby.runtime.InterpretedBlock.yieldSpecific(InterpretedBlock.java:229)
        at org.jruby.runtime.Block.yieldSpecific(Block.java:99)
        at org.jruby.ast.ZYieldNode.interpret(ZYieldNode.java:25)
        at org.jruby.ast.NewlineNode.interpret(NewlineNode.java:104)
        at org.jruby.ast.RescueNode.executeBody(RescueNode.java:216)
        at org.jruby.ast.RescueNode.interpretWithJavaExceptions(RescueNode.java:120)
        at org.jruby.ast.RescueNode.interpret(RescueNode.java:110)
        at org.jruby.evaluator.ASTInterpreter.INTERPRET_METHOD(ASTInterpreter.java:74)
        at org.jruby.internal.runtime.methods.InterpretedMethod.call(InterpretedMethod.java:169)
        at org.jruby.internal.runtime.methods.DefaultMethod.call(DefaultMethod.java:191)
        at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:302)
        at org.jruby.runtime.callsite.CachingCallSite.callBlock(CachingCallSite.java:144)
        at org.jruby.runtime.callsite.CachingCallSite.callIter(CachingCallSite.java:153)
        at org.jruby.ast.FCallSpecialArgBlockNode.interpret(FCallSpecialArgBlockNode.java:34)
        at org.jruby.ast.NewlineNode.interpret(NewlineNode.java:104)
        at org.jruby.ast.RescueNode.executeBody(RescueNode.java:216)
        at org.jruby.ast.RescueNode.interpretWithJavaExceptions(RescueNode.java:120)
        at org.jruby.ast.RescueNode.interpret(RescueNode.java:110)
        at org.jruby.evaluator.ASTInterpreter.INTERPRET_METHOD(ASTInterpreter.java:74)
        at org.jruby.internal.runtime.methods.InterpretedMethod.call(InterpretedMethod.java:233)
        at org.jruby.internal.runtime.methods.DefaultMethod.call(DefaultMethod.java:215)
        at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:332)
        at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:203)
        at org.jruby.ast.CallSpecialArgNode.interpret(CallSpecialArgNode.java:69)
        at org.jruby.ast.NewlineNode.interpret(NewlineNode.java:104)
        at org.jruby.evaluator.ASTInterpreter.INTERPRET_METHOD(ASTInterpreter.java:74)
        at org.jruby.internal.runtime.methods.InterpretedMethod.call(InterpretedMethod.java:233)
        at org.jruby.internal.runtime.methods.DefaultMethod.call(DefaultMethod.java:215)
        at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:332)
        at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:203)
        at org.jruby.ast.FCallSpecialArgNode.interpret(FCallSpecialArgNode.java:41)
        at org.jruby.ast.NewlineNode.interpret(NewlineNode.java:104)
        at org.jruby.evaluator.ASTInterpreter.INTERPRET_METHOD(ASTInterpreter.java:74)
        at org.jruby.internal.runtime.methods.InterpretedMethod.call(InterpretedMethod.java:190)
        at org.jruby.internal.runtime.methods.DefaultMethod.call(DefaultMethod.java:199)
        at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:312)
        at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:169)
        at org.jruby.ast.CallSpecialArgNode.interpret(CallSpecialArgNode.java:67)
        at org.jruby.ast.LocalAsgnNode.interpret(LocalAsgnNode.java:123)
        at org.jruby.ast.NewlineNode.interpret(NewlineNode.java:104)
        at org.jruby.ast.BlockNode.interpret(BlockNode.java:71)
        at org.jruby.evaluator.ASTInterpreter.INTERPRET_METHOD(ASTInterpreter.java:74)
        at org.jruby.internal.runtime.methods.InterpretedMethod.call(InterpretedMethod.java:147)
        at org.jruby.internal.runtime.methods.DefaultMethod.call(DefaultMethod.java:183)
        at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:292)
        at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:135)
        at org.jruby.ast.VCallNode.interpret(VCallNode.java:86)
        at org.jruby.ast.NewlineNode.interpret(NewlineNode.java:104)
        at org.jruby.ast.RootNode.interpret(RootNode.java:129)
        at org.jruby.evaluator.ASTInterpreter.INTERPRET_EVAL(ASTInterpreter.java:95)
        at org.jruby.evaluator.ASTInterpreter.evalWithBinding(ASTInterpreter.java:166)
        at org.jruby.RubyKernel.evalCommon(RubyKernel.java:1155)
        at org.jruby.RubyKernel.eval(RubyKernel.java:1112)
        at org.jruby.RubyKernel$s$0$3$eval.call(RubyKernel$s$0$3$eval.gen:65535)
        at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:181)
        at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:282)
        at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:71)
        at org.jruby.ast.FCallManyArgsNode.interpret(FCallManyArgsNode.java:60)
        at org.jruby.ast.NewlineNode.interpret(NewlineNode.java:104)
        at org.jruby.evaluator.ASTInterpreter.INTERPRET_METHOD(ASTInterpreter.java:74)
        at org.jruby.internal.runtime.methods.InterpretedMethod.call(InterpretedMethod.java:120)
        at org.jruby.internal.runtime.methods.InterpretedMethod.call(InterpretedMethod.java:134)
        at org.jruby.internal.runtime.methods.DefaultMethod.call(DefaultMethod.java:174)
        at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:282)
        at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:71)
        at org.jruby.ast.CallManyArgsNode.interpret(CallManyArgsNode.java:59)
        at org.jruby.ast.FCallOneArgNode.interpret(FCallOneArgNode.java:36)
        at org.jruby.ast.NewlineNode.interpret(NewlineNode.java:104)
        at org.jruby.ast.BlockNode.interpret(BlockNode.java:71)
        at org.jruby.evaluator.ASTInterpreter.INTERPRET_METHOD(ASTInterpreter.java:74)
        at org.jruby.internal.runtime.methods.InterpretedMethod.call(InterpretedMethod.java:233)
        at org.jruby.internal.runtime.methods.DefaultMethod.call(DefaultMethod.java:215)
        at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:332)
        at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:203)
        at org.jruby.ast.CallTwoArgNode.interpret(CallTwoArgNode.java:59)
        at org.jruby.ast.NewlineNode.interpret(NewlineNode.java:104)
        at org.jruby.ast.BlockNode.interpret(BlockNode.java:71)
        at org.jruby.ast.RescueNode.executeBody(RescueNode.java:216)
        at org.jruby.ast.RescueNode.interpretWithJavaExceptions(RescueNode.java:120)
        at org.jruby.ast.RescueNode.interpret(RescueNode.java:110)
        at org.jruby.ast.BeginNode.interpret(BeginNode.java:83)
        at org.jruby.ast.NewlineNode.interpret(NewlineNode.java:104)
        at org.jruby.ast.BlockNode.interpret(BlockNode.java:71)
        at org.jruby.evaluator.ASTInterpreter.INTERPRET_BLOCK(ASTInterpreter.java:111)
        at org.jruby.runtime.InterpretedBlock.evalBlockBody(InterpretedBlock.java:374)
        at org.jruby.runtime.InterpretedBlock.yield(InterpretedBlock.java:295)
        at org.jruby.runtime.InterpretedBlock.yieldSpecific(InterpretedBlock.java:229)
        at org.jruby.runtime.Block.yieldSpecific(Block.java:99)
        at org.jruby.ast.ZYieldNode.interpret(ZYieldNode.java:25)
        at org.jruby.ast.NewlineNode.interpret(NewlineNode.java:104)
        at org.jruby.ast.EnsureNode.interpret(EnsureNode.java:96)
        at org.jruby.ast.BeginNode.interpret(BeginNode.java:83)
        at org.jruby.ast.NewlineNode.interpret(NewlineNode.java:104)
        at org.jruby.ast.BlockNode.interpret(BlockNode.java:71)
        at org.jruby.evaluator.ASTInterpreter.INTERPRET_METHOD(ASTInterpreter.java:74)
        at org.jruby.internal.runtime.methods.InterpretedMethod.call(InterpretedMethod.java:212)
        at org.jruby.internal.runtime.methods.DefaultMethod.call(DefaultMethod.java:207)
        at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:322)
        at org.jruby.runtime.callsite.CachingCallSite.callBlock(CachingCallSite.java:178)
        at org.jruby.runtime.callsite.CachingCallSite.callIter(CachingCallSite.java:187)
        at org.jruby.ast.FCallOneArgBlockNode.interpret(FCallOneArgBlockNode.java:34)
        at org.jruby.ast.NewlineNode.interpret(NewlineNode.java:104)
        at org.jruby.evaluator.ASTInterpreter.INTERPRET_BLOCK(ASTInterpreter.java:111)
        at org.jruby.runtime.InterpretedBlock.evalBlockBody(InterpretedBlock.java:374)
        at org.jruby.runtime.InterpretedBlock.yieldSpecific(InterpretedBlock.java:260)
        at org.jruby.runtime.Block.yieldSpecific(Block.java:117)
        at org.jruby.ast.YieldTwoNode.interpret(YieldTwoNode.java:31)
        at org.jruby.ast.NewlineNode.interpret(NewlineNode.java:104)
        at org.jruby.ast.IfNode.interpret(IfNode.java:117)
        at org.jruby.ast.NewlineNode.interpret(NewlineNode.java:104)
        at org.jruby.ast.BlockNode.interpret(BlockNode.java:71)
        at org.jruby.ast.RescueNode.executeBody(RescueNode.java:216)
        at org.jruby.ast.RescueNode.interpretWithJavaExceptions(RescueNode.java:120)
        at org.jruby.ast.RescueNode.interpret(RescueNode.java:110)
        at org.jruby.ast.BeginNode.interpret(BeginNode.java:83)
        at org.jruby.ast.NewlineNode.interpret(NewlineNode.java:104)
        at org.jruby.evaluator.ASTInterpreter.INTERPRET_BLOCK(ASTInterpreter.java:111)
        at org.jruby.runtime.InterpretedBlock.evalBlockBody(InterpretedBlock.java:374)
        at org.jruby.runtime.InterpretedBlock.yield(InterpretedBlock.java:295)
        at org.jruby.runtime.InterpretedBlock.yieldSpecific(InterpretedBlock.java:229)
        at org.jruby.runtime.Block.yieldSpecific(Block.java:99)
        at org.jruby.RubyKernel.loop(RubyKernel.java:1439)
        at org.jruby.RubyKernel$s$0$0$loop.call(RubyKernel$s$0$0$loop.gen:65535)
        at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:302)
        at org.jruby.runtime.callsite.CachingCallSite.callBlock(CachingCallSite.java:144)
        at org.jruby.runtime.callsite.CachingCallSite.callIter(CachingCallSite.java:153)
        at org.jruby.ast.FCallNoArgBlockNode.interpret(FCallNoArgBlockNode.java:32)
        at org.jruby.ast.NewlineNode.interpret(NewlineNode.java:104)
        at org.jruby.evaluator.ASTInterpreter.INTERPRET_BLOCK(ASTInterpreter.java:111)
        at org.jruby.runtime.InterpretedBlock.evalBlockBody(InterpretedBlock.java:374)
        at org.jruby.runtime.InterpretedBlock.yield(InterpretedBlock.java:347)
        at org.jruby.runtime.InterpretedBlock.yield(InterpretedBlock.java:304)
        at org.jruby.runtime.Block.yield(Block.java:130)
        at org.jruby.RubyContinuation.enter(RubyContinuation.java:106)
        at org.jruby.RubyKernel.rbCatch(RubyKernel.java:1212)
        at org.jruby.RubyKernel$s$1$0$rbCatch.call(RubyKernel$s$1$0$rbCatch.gen:65535)
        at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:322)
        at org.jruby.runtime.callsite.CachingCallSite.callBlock(CachingCallSite.java:178)
        at org.jruby.runtime.callsite.CachingCallSite.callIter(CachingCallSite.java:187)
        at org.jruby.ast.FCallOneArgBlockNode.interpret(FCallOneArgBlockNode.java:34)
        at org.jruby.ast.NewlineNode.interpret(NewlineNode.java:104)
        at org.jruby.ast.BlockNode.interpret(BlockNode.java:71)
        at org.jruby.evaluator.ASTInterpreter.INTERPRET_METHOD(ASTInterpreter.java:74)
        at org.jruby.internal.runtime.methods.InterpretedMethod.call(InterpretedMethod.java:169)
        at org.jruby.internal.runtime.methods.DefaultMethod.call(DefaultMethod.java:191)
        at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:302)
        at org.jruby.runtime.callsite.CachingCallSite.callBlock(CachingCallSite.java:144)
        at org.jruby.runtime.callsite.CachingCallSite.callIter(CachingCallSite.java:153)
        at org.jruby.ast.CallNoArgBlockNode.interpret(CallNoArgBlockNode.java:64)
        at org.jruby.ast.NewlineNode.interpret(NewlineNode.java:104)
        at org.jruby.ast.BlockNode.interpret(BlockNode.java:71)
        at org.jruby.evaluator.ASTInterpreter.INTERPRET_METHOD(ASTInterpreter.java:74)
        at org.jruby.internal.runtime.methods.InterpretedMethod.call(InterpretedMethod.java:147)
        at org.jruby.internal.runtime.methods.DefaultMethod.call(DefaultMethod.java:183)
        at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:292)
        at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:135)
        at usr.lib.ams_minus_hbase.bin.$_dot_dot_.bin.hirb.block_2$RUBY$start(/usr/lib/ams-hbase/bin/../bin/hirb.rb:205)
        at usr$lib$ams_minus_hbase$bin$$_dot_dot_$bin$hirb$block_2$RUBY$start.call(usr$lib$ams_minus_hbase$bin$$_dot_dot_$bin$hirb$block_2$RUBY$start:65535)
        at org.jruby.runtime.CompiledBlock.yield(CompiledBlock.java:112)
        at org.jruby.runtime.CompiledBlock.yield(CompiledBlock.java:95)
        at org.jruby.runtime.Block.yield(Block.java:130)
        at org.jruby.RubyContinuation.enter(RubyContinuation.java:106)
        at org.jruby.RubyKernel.rbCatch(RubyKernel.java:1212)
        at org.jruby.RubyKernel$s$1$0$rbCatch.call(RubyKernel$s$1$0$rbCatch.gen:65535)
        at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:322)
        at org.jruby.runtime.callsite.CachingCallSite.callBlock(CachingCallSite.java:178)
        at org.jruby.runtime.callsite.CachingCallSite.callIter(CachingCallSite.java:187)
        at usr.lib.ams_minus_hbase.bin.$_dot_dot_.bin.hirb.method__5$RUBY$start(/usr/lib/ams-hbase/bin/../bin/hirb.rb:204)
        at usr$lib$ams_minus_hbase$bin$$_dot_dot_$bin$hirb$method__5$RUBY$start.call(usr$lib$ams_minus_hbase$bin$$_dot_dot_$bin$hirb$method__5$RUBY$start:65535)
        at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:203)
        at org.jruby.internal.runtime.methods.CompiledMethod.call(CompiledMethod.java:255)
        at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:292)
        at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:135)
        at usr.lib.ams_minus_hbase.bin.$_dot_dot_.bin.hirb.__file__(/usr/lib/ams-hbase/bin/../bin/hirb.rb:210)
        at usr.lib.ams_minus_hbase.bin.$_dot_dot_.bin.hirb.load(/usr/lib/ams-hbase/bin/../bin/hirb.rb)
        at org.jruby.Ruby.runScript(Ruby.java:697)
        at org.jruby.Ruby.runScript(Ruby.java:690)
        at org.jruby.Ruby.runNormally(Ruby.java:597)
        at org.jruby.Ruby.runFromMain(Ruby.java:446)
        at org.jruby.Main.doRunFromMain(Main.java:369)
        at org.jruby.Main.internalRun(Main.java:258)
        at org.jruby.Main.run(Main.java:224)
        at org.jruby.Main.run(Main.java:208)
        at org.jruby.Main.main(Main.java:188)
2016-07-14 08:50:00,588 ERROR [main] client.ConnectionManager$HConnectionImplementation: Can't get connection to ZooKeeper: KeeperErrorCode = ConnectionLoss for /ams-hbase-unsecure

That said, starting the application from within Ambari seems to have worked without an issue. I'm still not seeing metrics appear after this so I'm assuming that I'm just not seeing the error. Just for reference, ZooKeeper has no problems according to Ambari.

Expert Contributor

Hi Fraser,

Can you share the following :

ams-site.xml in /etc/ambari-metrics-collector/conf

hbase-site.xml, hbase-env.xml in /etc/ams-hbase/conf

New Contributor

Please find attached the requested configuration xml files. Just to give a further update, from my investigations it looks like the service is stopping at random intervals but also, when restarted, the Metrics are updating but only when I'm not on the Ambari server page. Oddly, when I'm browsing on other Firefox windows it works fine, but as soon as I tab back to the Ambari webpage the metrics stop - if you look at the attached screenshot, you should be able to see the drop in activity in the Memory, Network, CPU and Cluster Load metrics - they seemingly "go off a cliff" and all metrics drop to zero;

5786-hadoop1.png

Also, note that in the second screenshot that the time (10:43:56 - I've not changed the time to GMT, it is one hour ahead) is the moment I switched into the Ambari GUI and this is where the metrics have stopped reporting to the screen.

5787-hadoop2.png

Hopefully you can use the XML files to figure out what is going on here.

Expert Contributor

Hi Fraser,

What version of Ambari is this?

Did you happen to change the following ams-hbase-site config's value - hbase.zookeeper.property.clientPort ? In AMS embedded mode, it uses it's own local zookeeper running at port 61181. However, I see the value 2181 in your hbase-site.xml.

New Contributor

Hi @Aravindan Vijayan,

The version according to the UI is 2.2.1.0.

I haven't made any configuration changes as I am still learning about Hadoop - if that is the configuration that is in there, then that is the configuration that the VM came with. I downloaded the Hortonworks Sandbox v2.4 files (for VMWare Player) directly from the Hortonworks website so this is bizarre. Will changing the clientPort property address this issue or is this indicative of a larger mismatch in configuration?

Expert Contributor

Hi @Fraser Mills,

What is the size of your cluster?

Please shut down AMS collector, make the following changes in AMS configs and then start it up again.

  1. hbase.zookeeper.property.clientPort = {{zookeeper_clientPort}}
  2. hbase.zookeeper.quorum = {{zookeeper_quorum_hosts}}
  3. Delete the hbase-site : {hbase.tmp.dir}/hbase-tmp/zookeeper/* (In your case, it will be /tmp/hbase-${user.name}/hbase-tmp/zookeeper/*)

Also, can you share the value of hbase_master_xmn_size and metrics_collector_heapsize from AMS configs?

Expert Contributor

@Aravindan Vijayan, @Kuldeep Kulkarni, @Neeraj Sabharwal, @Fraser Mills - i also seem to be having issues with Ambari Metrics Collector - it is shutting down ..

Details in this link, pls let me know.