Created 10-01-2018 04:40 PM
I am running on HDP-3.0.1 and Yarn 3.1.0 and ran into the same issue which says
resource_management.core.exceptions.ExecuteTimeoutException: Execution of 'ambari-sudo.sh su yarn-ats -l -s /bin/bash -c 'export PATH='"'"'/usr/sbin:/sbin:/usr/lib/ambari-server/*:/usr/sbin:/sbin:/usr/lib/ambari-server/*:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin:/var/lib/ambari-agent:/var/lib/ambari-agent'"'"' ; sleep 10;export HBASE_CLASSPATH_PREFIX=/usr/hdp/3.0.1.0-187/hadoop-yarn/timelineservice/*; /usr/hdp/3.0.1.0-187/hbase/bin/hbase --config /usr/hdp/3.0.1.0-187/hadoop/conf/embedded-yarn-ats-hbase org.apache.hadoop.yarn.server.timelineservice.storage.TimelineSchemaCreator -Dhbase.client.retries.number=35 -create -s'' was killed due timeout after 300 seconds
Created 10-04-2018 09:22 AM
Same here. Anyone who responsible for this please have a look. Thanks.
Created 10-04-2018 09:25 AM
Can you please check if the host where the TimelineServer V2.0 Reader is installed also has the "HBase Client" installed or not?
If not then please install the "HBase client" on that host and then try again.
Created 02-07-2019 08:51 PM
I have the same issue. I do have hbase client installed
2019-02-07 16:57:28,593 INFO [ReadOnlyZKClient-ip-10-241-10-72.eu-west-1.compute.internal:2181,ip-10-241-10-7.eu-west-1.compute.internal:2181,ip-10-241-10-96.eu-west-1.compute.internal:2181@0x7770f470] zookeeper.ZooKeeper: Initiating client connection, connectString=ip-10-241-10-72.eu-west-1.compute. internal:2181,ip-10-241-10-7.eu-west-1.compute.internal:2181,ip-10-241-10-96.eu-west-1.compute.internal:2181 sessionTimeout=90000 watcher=org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$$Lambda$13/122109374@59d29f3c 2019-02-07 16:57:28,619 INFO [ReadOnlyZKClient-ip-10-241-10-72.eu-west-1.compute.internal:2181,ip-10-241-10-7.eu-west-1.compute.internal:2181,ip-10-241-10-96.eu-west-1.compute.internal:2181@0x7770f470-SendThread(ip-10-241-10-72.eu-west-1.compute.internal:2181)] zookeeper.ClientCnxn: Opening socket co nnection to server ip-10-241-10-72.eu-west-1.compute.internal/10.241.10.72:2181. Will not attempt to authenticate using SASL (unknown error) 2019-02-07 16:57:28,628 INFO [ReadOnlyZKClient-ip-10-241-10-72.eu-west-1.compute.internal:2181,ip-10-241-10-7.eu-west-1.compute.internal:2181,ip-10-241-10-96.eu-west-1.compute.internal:2181@0x7770f470-SendThread(ip-10-241-10-72.eu-west-1.compute.internal:2181)] zookeeper.ClientCnxn: Socket connection established, initiating session, client: /10.241.10.72:34382, server: ip-10-241-10-72.eu-west-1.compute.internal/10.241.10.72:2181 2019-02-07 16:57:28,642 INFO [ReadOnlyZKClient-ip-10-241-10-72.eu-west-1.compute.internal:2181,ip-10-241-10-7.eu-west-1.compute.internal:2181,ip-10-241-10-96.eu-west-1.compute.internal:2181@0x7770f470-SendThread(ip-10-241-10-72.eu-west-1.compute.internal:2181)] zookeeper.ClientCnxn: Session establish ment complete on server ip-10-241-10-72.eu-west-1.compute.internal/10.241.10.72:2181, sessionid = 0x268c8d1bb3600c1, negotiated timeout = 60000 2019-02-07 16:57:28,662 WARN [main] client.ConnectionImplementation: Retrieve cluster id failed java.util.concurrent.ExecutionException: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /atsv2-hbase-secure/hbaseid at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1895) at org.apache.hadoop.hbase.client.ConnectionImplementation.retrieveClusterId(ConnectionImplementation.java:527) at org.apache.hadoop.hbase.client.ConnectionImplementation.<init>(ConnectionImplementation.java:287) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:219) at org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:114) at org.apache.hadoop.yarn.server.timelineservice.storage.TimelineSchemaCreator.createAllTables(TimelineSchemaCreator.java:301) at org.apache.hadoop.yarn.server.timelineservice.storage.TimelineSchemaCreator.createAllSchemas(TimelineSchemaCreator.java:277) at org.apache.hadoop.yarn.server.timelineservice.storage.TimelineSchemaCreator.main(TimelineSchemaCreator.java:146) Caused by: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /atsv2-hbase-secure/hbaseid at org.apache.zookeeper.KeeperException.create(KeeperException.java:111) at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:168) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:323) at java.lang.Thread.run(Thread.java:745) 2019-02-07 16:57:32,982 INFO [main] client.RpcRetryingCallerImpl: Call exception, tries=6, retries=36, started=4124 ms ago, cancelled=false, msg=org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /atsv2-hbase-secure/meta-region-server, details=row 'prod.timelineservice .entity' on table 'hbase:meta' at null
Created 02-12-2019 12:00 AM
any update on this issue, I still have the same issue
Created 02-13-2019 07:09 PM
Hello @Jay Kumar SenSharma I have the same issue. I installed hbase client ont the node but Timeline Service V2.0 Reader still do not start with the same message error. Any update please ?
Created 02-26-2019 08:19 AM
In my case, I had the following error in Ambari when trying to start the Timeline Reader:
KeeperErrorCode = NoNode for /atsv2-hbase-secure/hbaseid
It was caused by the embedded HBase service failing, which was not shown in Ambari. Without the embedded HBase service, Timeline Reader won't be able to start.
I had to look in the logs to find the root cause:
/var/log/hadoop-yarn/embedded-yarn-ats-hbase/hbase-yarn-ats-master-<host>.log
2019-02-26 09:46:50,589 ERROR [Thread-18] master.HMaster: Failed to become active master org.apache.hadoop.security.AccessControlException: Permission denied: user=yarn-ats-hbase/<host>@<realm>, access=WRITE, path="/atsv2/hbase/data/.tmp"
I am using Isilon OneFS instead of HDFS, so this looks like a permission issue / missing user-mapping rule, which is specific to this setup.
Created 02-26-2019 08:22 AM
This is the output shown in "stderr" in Ambari when starting the service. It doesn't really give the exact error, just shows that it was timing out. Can you include the output shown in the "stdout" part - this should have more detail?
Created 02-26-2019 10:18 AM
I do have hbase client.
I've traced this to an issue with the move process. It turns out that the kerberos principals stayed pointing to the old host. Funny enough, I just clicked set recommended, restarted all services as set to require a restart and everything came back online
Created 02-26-2019 10:18 AM
I do have hbase client.
I've traced this to an issue with the move process. It turns out that the kerberos principals stayed pointing to the old host. Funny enough, I just clicked set recommended, restarted all services as set to require a restart and everything came back online