Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Error with ATSv2 after fresh HDP installation

Highlighted

Error with ATSv2 after fresh HDP installation

New Contributor

I have just installed new HDPv3.0.1 via Ambari-v2.7

I got an error in Timeline Servise:

ATSv2 HBase Application
The HBase application reported a 'STARTED' state. Check took 1.922s

I also saw "HBase application reported a 'FAILED' state" and "HBase application reported a 'ACCEPTED' state", but nothing changed.

Bellow is an error from /var/log/hadoop-yarn/yarn/hadoop-yarn-timelinereader-${hostname}.log file:

2018-10-21 12:05:20,881 WARN  client.ConnectionImplementation (ConnectionImplementation.java:retrieveClusterId(528)) - Retrieve cluster id failedjava.util.concurrent.ExecutionException: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /atsv2-hbase-unsecure/hbaseid
 at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
 at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1895)
 at org.apache.hadoop.hbase.client.ConnectionImplementation.retrieveClusterId(ConnectionImplementation.java:526)
 at org.apache.hadoop.hbase.client.ConnectionImplementation.<init>(ConnectionImplementation.java:286)
 at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
 at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
 at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
 at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
 at org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:219)
 at org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:114)
 at org.apache.hadoop.yarn.server.timelineservice.storage.HBaseTimelineReaderImpl.serviceInit(HBaseTimelineReaderImpl.java:88)
 at org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
 at org.apache.hadoop.yarn.server.timelineservice.reader.TimelineReaderServer.serviceInit(TimelineReaderServer.java:92)
 at org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
 at org.apache.hadoop.yarn.server.timelineservice.reader.TimelineReaderServer.startTimelineReaderServer(TimelineReaderServer.java:233)
 at org.apache.hadoop.yarn.server.timelineservice.reader.TimelineReaderServer.main(TimelineReaderServer.java:246)
Caused by: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /atsv2-hbase-unsecure/hbaseid
 at org.apache.zookeeper.KeeperException.create(KeeperException.java:111)
 at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
 at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:164)
 at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:321)
 at java.lang.Thread.run(Thread.java:745)
2018-10-21 12:05:21,044 INFO  common.HBaseTimelineStorageUtils (HBaseTimelineStorageUtils.java:getTimelineServiceHBaseConf(65)) - Using hbase configuratio at file:///usr/hdp/3.0.1.0-187/hadoop/conf/embedded-yarn-ats-hbase/hbase-site.xml
2018-10-21 12:05:21,046 INFO  zookeeper.ReadOnlyZKClient (ReadOnlyZKClient.java:<init>(130)) - Start read only zookeeper connection 0x306e95ec to hdp-mstr01-test.mac.org.il:2181,hdp-mstr03-test.mac.org.il:2181,hdp-mstr02-test.mac.org.il:2181, session timeout 90000 ms, retries 6, retry interval 1000 ms, keep alive 60000 ms
2018-10-21 12:05:21,046 INFO  zookeeper.ZooKeeper (ZooKeeper.java:<init>(438)) - Initiating client connection, connectString=hdp-mstr01-test.mac.org.il:2181,hdp-mstr03-test.mac.org.il:2181,hdp-mstr02-test.mac.org.il:2181 sessionTimeout=90000 <a href="mailto:watcher=org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$$Lambda$13/860115206@51d34977">watcher=org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$$Lambda$13/860115206@51d34977</a>
2018-10-21 12:05:21,048 INFO  zookeeper.ClientCnxn (ClientCnxn.java:logStartConnect(1019)) - Opening socket connection to server hdp-mstr01-test.mac.org.il/10.70.8.154:2181. Will not attempt to authenticate using SASL (unknown error)
2018-10-21 12:05:21,049 INFO  zookeeper.ClientCnxn (ClientCnxn.java:primeConnection(864)) - Socket connection established, initiating session, client: /10.70.8.185:57362, server: hdp-mstr01-test.mac.org.il/10.70.8.154:2181
2018-10-21 12:05:21,051 INFO  zookeeper.ClientCnxn (ClientCnxn.java:onConnected(1279)) - Session establishment complete on server hdp-mstr01-test.mac.org.il/10.70.8.154:2181, sessionid = 0x166827fcf3a0079, negotiated timeout = 60000
2018-10-21 12:05:21,052 WARN  client.ConnectionImplementation (ConnectionImplementation.java:retrieveClusterId(528)) - Retrieve cluster id failed
java.util.concurrent.ExecutionException: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /atsv2-hbase-unsecure/hbaseid
 at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
 at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1895)
 at org.apache.hadoop.hbase.client.ConnectionImplementation.retrieveClusterId(ConnectionImplementation.java:526)
 at org.apache.hadoop.hbase.client.ConnectionImplementation.<init>(ConnectionImplementation.java:286)
 at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
 at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
 at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
 at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
 at org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:219)
 at org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:114)
 at org.apache.hadoop.yarn.server.timelineservice.storage.HBaseTimelineReaderImpl.serviceInit(HBaseTimelineReaderImpl.java:93)
 at org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
 at org.apache.hadoop.yarn.server.timelineservice.reader.TimelineReaderServer.serviceInit(TimelineReaderServer.java:92)
 at org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
 at org.apache.hadoop.yarn.server.timelineservice.reader.TimelineReaderServer.startTimelineReaderServer(TimelineReaderServer.java:233)
 at org.apache.hadoop.yarn.server.timelineservice.reader.TimelineReaderServer.main(TimelineReaderServer.java:246)
Caused by: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /atsv2-hbase-unsecure/hbaseid
 at org.apache.zookeeper.KeeperException.create(KeeperException.java:111)
 at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
 at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:164)
 at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:321)
 at java.lang.Thread.run(Thread.java:745)
2018-10-21 12:05:21,057 INFO  storage.HBaseTimelineReaderImpl (HBaseTimelineReaderImpl.java:serviceStart(99)) - Scheduling HBase liveness monitor at interval 60000
2018-10-21 12:05:21,058 INFO  storage.HBaseTimelineReaderImpl (HBaseTimelineReaderImpl.java:run(170)) - Running HBase liveness monitor
2018-10-21 12:05:21,061 INFO  timeline.TimelineServerUtils (TimelineServerUtils.java:setTimelineFilters(66)) - Filter initializers set for timeline service: org.apache.hadoop.security.HttpCrossOriginFilterInitializer,org.apache.hadoop.yarn.server.timelineservice.reader.security.TimelineReaderAuthenticationFilterInitializer,org.apache.hadoop.yarn.server.timelineservice.reader.security.TimelineReaderWhitelistAuthorizationFilterInitializer
2018-10-21 12:05:21,061 INFO  reader.TimelineReaderServer (TimelineReaderServer.java:startTimelineReaderWebApp(187)) - Instantiating TimelineReaderWebApp at 0.0.0.0:8198
2018-10-21 12:05:21,087 INFO  util.log (Log.java:initialized(192)) - Logging initialized @1216ms
2018-10-21 12:05:21,150 WARN  storage.HBaseTimelineReaderImpl (HBaseTimelineReaderImpl.java:run(183)) - Got failure attempting to read from timeline storage, assuming HBase down
java.io.UncheckedIOException: org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0
 at org.apache.hadoop.hbase.client.ResultScanner$1.hasNext(ResultScanner.java:55)
 at org.apache.hadoop.yarn.server.timelineservice.storage.reader.TimelineEntityReader.readEntities(TimelineEntityReader.java:283)
 at org.apache.hadoop.yarn.server.timelineservice.storage.HBaseTimelineReaderImpl$HBaseMonitor.run(HBaseTimelineReaderImpl.java:174)
 at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
 at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
 at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
 at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
 at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0
 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:332)
 at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153)
 at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58)
 at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:192)
 at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269)
 at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437)
 at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312)
 at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597)
 at org.apache.hadoop.hbase.client.ResultScanner$1.hasNext(ResultScanner.java:53)
 ... 9 more
Caused by: java.io.IOException: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /atsv2-hbase-unsecure/meta-region-server
 at org.apache.hadoop.hbase.client.ConnectionImplementation.get(ConnectionImplementation.java:2002)
 at org.apache.hadoop.hbase.client.ConnectionImplementation.locateMeta(ConnectionImplementation.java:762)
 at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:729)
 at org.apache.hadoop.hbase.client.ConnectionImplementation.relocateRegion(ConnectionImplementation.java:707)
 at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:911)
 at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:732)
 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:325)
 ... 17 more
Caused by: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /atsv2-hbase-unsecure/meta-region-server
 at org.apache.zookeeper.KeeperException.create(KeeperException.java:111)
 at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
 at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:164)
 at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:321)
 ... 1 more

I have checked in zkCli.sh , and there is no "/atsv2-hbase-unsecure" node. Who should had been created it?

4 REPLIES 4

Re: Error with ATSv2 after fresh HDP installation

New Contributor

I also have the same problem.

Re: Error with ATSv2 after fresh HDP installation

New Contributor

@Zholaman Kubaliyev @Ilya Ulis

Was this ever resolved? I have the same issue?

Re: Error with ATSv2 after fresh HDP installation

New Contributor

You need to create a different queue for yarn-ats user

Re: Error with ATSv2 after fresh HDP installation

New Contributor

@Gaurav Sharma would you be able to help?