Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

HDP3.0: timeline service V2 reader cannot create zookeeper nodes.

avatar
Rising Star

Hi,

I am using HDP3.0 and ambari 2.7 blueprint to install my cluster. Yarn's timeline service V2 reader cannot create zookeeper node:

at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:168) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:323) at java.lang.Thread.run(Thread.java:745) 2018-08-14 17:59:55,827 INFO [main] client.RpcRetryingCallerImpl: Call exception, tries=6, retries=6, started=4142 ms ago, cancelled=false, msg=org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /atsv2-hbase-unsecure/meta-region-server, details=row 'prod.timelineservice.entity' on table 'hbase:meta' at null

I observed yarn is using timeline service 1.5 and timeline service reader v2. I am not sure if this is expected. My blueprint is using:

{
    "name": "APP_TIMELINE_SERVER"
},

Adding hbase client per this post did not help. Any idea? Thanks.

1 ACCEPTED SOLUTION

avatar
Master Mentor

@Lian Jiang

Also please find the below properties values from your Blueprint file and change the values to a path which has proper permission:

Specially inside "yarn-hbase-env"

hbase_java_io_tmpdir
yarn_hbase_java_io_tmpdir

.

View solution in original post

13 REPLIES 13

avatar
New Contributor

Not sure if this is still relevant or not, but I had a similar issue (NoNode for /atsv2-hbase-secure/master, details=row 'prod.timelineservice.entity' on table 'hbase:meta' at null) . I fixed it by changing the yarn hbase site's 'zookeeper.znode.parent' value to a non-existent zookeeper path and then restarting YARN via Ambari.

avatar
Contributor

This works, great workaround.

avatar
Contributor

This works, great workaround.

avatar
New Contributor

I've tried this and it doesn't seem to work. No matter what I change it to, I get errors in the log indicating it can't find znodes. ie:

2019-05-01 11:11:28,748 INFO zookeeper.ClientCnxn (ClientCnxn.java:logStartConnect(1019)) - Opening socket connection to server myserver.com/xx.xx.xxx.xxx:2181. Will not attempt to authenticate using SASL (unknown error)

2019-05-01 11:11:28,757 INFO zookeeper.ClientCnxn (ClientCnxn.java:primeConnection(864)) - Socket connection established, initiating session, client: /10.87.130.196:51436, server:myserver.com/xx.xx.xxx.xxx:2181

2019-05-01 11:11:28,772 INFO zookeeper.ClientCnxn (ClientCnxn.java:onConnected(1279)) - Session establishment complete on server myserver.com/xx.xx.xxx.xxx:2181, sessionid = 0x36a72f6d9090007, negotiated timeout = 60000

2019-05-01 11:11:28,792 WARN client.ConnectionImplementation (ConnectionImplementation.java:retrieveClusterId(528)) - Retrieve cluster id failed

java.util.concurrent.ExecutionException: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /atsv2-hbase-unsecure/hbaseid

at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)

at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1895)

at org.apache.hadoop.hbase.client.ConnectionImplementation.retrieveClusterId(ConnectionImplementation.java:526)

at org.apache.hadoop.hbase.client.ConnectionImplementation.<init>(ConnectionImplementation.java:286)

at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)

at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)

at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)

at java.lang.reflect.Constructor.newInstance(Constructor.java:423)

at org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:219)

at org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:114)

at org.apache.hadoop.yarn.server.timelineservice.storage.HBaseTimelineReaderImpl.serviceInit(HBaseTimelineReaderImpl.java:88)

at org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)

at org.apache.hadoop.yarn.server.timelineservice.reader.TimelineReaderServer.serviceInit(TimelineReaderServer.java:92)

at org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)

at org.apache.hadoop.yarn.server.timelineservice.reader.TimelineReaderServer.startTimelineReaderServer(TimelineReaderServer.java:233)

at org.apache.hadoop.yarn.server.timelineservice.reader.TimelineReaderServer.main(TimelineReaderServer.java:246)

Caused by: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /atsv2-hbase-unsecure/hbaseid

at org.apache.zookeeper.KeeperException.create(KeeperException.java:111)

at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)

at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:164)

at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:321)

at java.lang.Thread.run(Thread.java:745)

2019-05-01 11:11:28,982 INFO common.HBaseTimelineStorageUtils (HBaseTimelineStorageUtils.java:getTimelineServiceHBaseConf(65)) - Using hbase configuration at file:///usr/hdp/3.1.0.0-78/hadoop/conf/embedded-yarn-ats-hbase/hbase-site.xml

2019-05-01 11:11:28,984 INFO zookeeper.ReadOnlyZKClient (ReadOnlyZKClient.java:<init>(130)) - Start read only zookeeper connection 0x534a5a98 to myserver.com:2181,myserver2.com:2181,myserver3.com:2181, session timeout 90000 ms, retries 6, retry interval 1000 ms, keep alive 60000 ms