Timeline server V 2.0 reader not starting on HDP-3.0.1

[#1]

I am running on HDP-3.0.1 and Yarn 3.1.0 and ran into the same issue which says

resource_management.core.exceptions.ExecuteTimeoutException: Execution of 'ambari-sudo.sh su yarn-ats -l -s /bin/bash -c 'export PATH='"'"'/usr/sbin:/sbin:/usr/lib/ambari-server/*:/usr/sbin:/sbin:/usr/lib/ambari-server/*:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin:/var/lib/ambari-agent:/var/lib/ambari-agent'"'"' ; sleep 10;export HBASE_CLASSPATH_PREFIX=/usr/hdp/3.0.1.0-187/hadoop-yarn/timelineservice/*; /usr/hdp/3.0.1.0-187/hbase/bin/hbase --config /usr/hdp/3.0.1.0-187/hadoop/conf/embedded-yarn-ats-hbase org.apache.hadoop.yarn.server.timelineservice.storage.TimelineSchemaCreator -Dhbase.client.retries.number=35 -create -s'' was killed due timeout after 300 seconds

[#2]

Same here. Anyone who responsible for this please have a look. Thanks.

[#3]

@Siva Botcha

@Chandler Huang

Can you please check if the host where the TimelineServer V2.0 Reader is installed also has the "HBase Client" installed or not?

If not then please install the "HBase client" on that host and then try again.

[#4]

I have the same issue. I do have hbase client installed

2019-02-07 16:57:28,593 INFO  [ReadOnlyZKClient-ip-10-241-10-72.eu-west-1.compute.internal:2181,ip-10-241-10-7.eu-west-1.compute.internal:2181,ip-10-241-10-96.eu-west-1.compute.internal:2181@0x7770f470] zookeeper.ZooKeeper: Initiating client connection, connectString=ip-10-241-10-72.eu-west-1.compute.
internal:2181,ip-10-241-10-7.eu-west-1.compute.internal:2181,ip-10-241-10-96.eu-west-1.compute.internal:2181 sessionTimeout=90000 watcher=org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$$Lambda$13/122109374@59d29f3c

2019-02-07 16:57:28,619 INFO  [ReadOnlyZKClient-ip-10-241-10-72.eu-west-1.compute.internal:2181,ip-10-241-10-7.eu-west-1.compute.internal:2181,ip-10-241-10-96.eu-west-1.compute.internal:2181@0x7770f470-SendThread(ip-10-241-10-72.eu-west-1.compute.internal:2181)] zookeeper.ClientCnxn: Opening socket co
nnection to server ip-10-241-10-72.eu-west-1.compute.internal/10.241.10.72:2181. Will not attempt to authenticate using SASL (unknown error)

2019-02-07 16:57:28,628 INFO  [ReadOnlyZKClient-ip-10-241-10-72.eu-west-1.compute.internal:2181,ip-10-241-10-7.eu-west-1.compute.internal:2181,ip-10-241-10-96.eu-west-1.compute.internal:2181@0x7770f470-SendThread(ip-10-241-10-72.eu-west-1.compute.internal:2181)] zookeeper.ClientCnxn: Socket connection
 established, initiating session, client: /10.241.10.72:34382, server: ip-10-241-10-72.eu-west-1.compute.internal/10.241.10.72:2181

2019-02-07 16:57:28,642 INFO  [ReadOnlyZKClient-ip-10-241-10-72.eu-west-1.compute.internal:2181,ip-10-241-10-7.eu-west-1.compute.internal:2181,ip-10-241-10-96.eu-west-1.compute.internal:2181@0x7770f470-SendThread(ip-10-241-10-72.eu-west-1.compute.internal:2181)] zookeeper.ClientCnxn: Session establish
ment complete on server ip-10-241-10-72.eu-west-1.compute.internal/10.241.10.72:2181, sessionid = 0x268c8d1bb3600c1, negotiated timeout = 60000

2019-02-07 16:57:28,662 WARN  [main] client.ConnectionImplementation: Retrieve cluster id failed

java.util.concurrent.ExecutionException: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /atsv2-hbase-secure/hbaseid

        at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)

        at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1895)

        at org.apache.hadoop.hbase.client.ConnectionImplementation.retrieveClusterId(ConnectionImplementation.java:527)

        at org.apache.hadoop.hbase.client.ConnectionImplementation.<init>(ConnectionImplementation.java:287)

        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)

        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)

        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)

        at java.lang.reflect.Constructor.newInstance(Constructor.java:423)

        at org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:219)

        at org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:114)

        at org.apache.hadoop.yarn.server.timelineservice.storage.TimelineSchemaCreator.createAllTables(TimelineSchemaCreator.java:301)

        at org.apache.hadoop.yarn.server.timelineservice.storage.TimelineSchemaCreator.createAllSchemas(TimelineSchemaCreator.java:277)

        at org.apache.hadoop.yarn.server.timelineservice.storage.TimelineSchemaCreator.main(TimelineSchemaCreator.java:146)

Caused by: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /atsv2-hbase-secure/hbaseid

        at org.apache.zookeeper.KeeperException.create(KeeperException.java:111)

        at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)

        at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:168)

        at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:323)

        at java.lang.Thread.run(Thread.java:745)

2019-02-07 16:57:32,982 INFO  [main] client.RpcRetryingCallerImpl: Call exception, tries=6, retries=36, started=4124 ms ago, cancelled=false, msg=org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /atsv2-hbase-secure/meta-region-server, details=row 'prod.timelineservice
.entity' on table 'hbase:meta' at null

[#5]

any update on this issue, I still have the same issue

[#6]

Hello @Jay Kumar SenSharma I have the same issue. I installed hbase client ont the node but Timeline Service V2.0 Reader still do not start with the same message error. Any update please ?

[#7]

In my case, I had the following error in Ambari when trying to start the Timeline Reader:

KeeperErrorCode = NoNode for /atsv2-hbase-secure/hbaseid

It was caused by the embedded HBase service failing, which was not shown in Ambari. Without the embedded HBase service, Timeline Reader won't be able to start.

I had to look in the logs to find the root cause:

/var/log/hadoop-yarn/embedded-yarn-ats-hbase/hbase-yarn-ats-master-<host>.log

2019-02-26 09:46:50,589 ERROR [Thread-18] master.HMaster: Failed to become active master
org.apache.hadoop.security.AccessControlException: Permission denied: user=yarn-ats-hbase/<host>@<realm>, access=WRITE, path="/atsv2/hbase/data/.tmp"

I am using Isilon OneFS instead of HDFS, so this looks like a permission issue / missing user-mapping rule, which is specific to this setup.

[#8]

This is the output shown in "stderr" in Ambari when starting the service. It doesn't really give the exact error, just shows that it was timing out. Can you include the output shown in the "stdout" part - this should have more detail?

[#9]

I do have hbase client.

I've traced this to an issue with the move process. It turns out that the kerberos principals stayed pointing to the old host. Funny enough, I just clicked set recommended, restarted all services as set to require a restart and everything came back online

[#10]

I do have hbase client.

I've traced this to an issue with the move process. It turns out that the kerberos principals stayed pointing to the old host. Funny enough, I just clicked set recommended, restarted all services as set to require a restart and everything came back online